This week I read the story about how Iran hacked and captured one of our most sophisticated military drones. It struck me that this was an excellent example of the potentially disastrous ramifications of ignoring technical debt. It appears that the potential to spoof GPS was known and ignored for many years. Acknowledging and managing technical debt is an issue that separates the whiz kids from the graybeards in the tech industry. Let me start by saying that a military boof-a-rama of this magnitude is, by necessity, going to be followed up with multiple misinformation campaigns so I doubt we’ll really know what really happened for a few decades. Nevertheless it provides a teachable moment for us so let’s explore this topic.
Technical debt is analogous to financial debt so let’s start there. Imagine you are a fresh college graduate with a new job and a newly minted credit card. Thanks to the miracle of modern financial Svengalism, you can buy things you don’t have the money to pay for. You use the card to get set up in a new apartment and at the end of the month you have a choice, you can pay off the debt or you can pay the minimal possible. Debt is not a problem if no one makes you pay attention to it. And there’s the rub, at some point you reach your limit and the party comes to an abrupt halt. Not only do you have to start paying off your debt but you have accumulated a bunch of interest that needs to be paid off as well. At this point, you are screwed. You can’t buy new stuff and you have to spend all your effort trying to pay off a mountain of debt. You pay and pay and pay but for all your effort and money going out, you have no new stuff to show for it – you are managing your financial debt.
The same thing happens with technology. This is most clearly seen in startup companies but the same thing happens in most technology companies. At a tech startup, the most important thing to do is to get the right set of functions in the market as quickly as possible so you can start bringing money in. What this means is that you are often cutting corners – skipping documentation, ignoring supportability, deferring security work, doing quick and dirty designs vs. deep architectural work. This might be a reasonable strategy for a startup because once you get the product out, you might get feedback that you need to go in an entirely different direction. If that is the case, you can “declare bankruptcy” on your existing code, walk away from those debts and start on a new path. On the other hand, if you stick with that code, you have a technical debt which is going to have to be paid. In your next development cycle, you can either begin to pay down that technical debt or you can devote 100% of your resources on the next round of quick and dirty functions. Deciding when to start paying off your technical debt is what separates the whiz kids from the graybeards in the tech industry.
I’m sure most of us at some point have found ourselves on a project that was led by whiz kids that had completely ignored their technical debt and was now in shambles. I was hired at DEC to lead the turnaround of just such a project. The product was reasonably successful but the implementation “had some issues”. When I got onboard and did an assessment of where we were I was shocked – the product was a cobbled together hack using 18 (yes 18!) different languages. There was even some ADA code in there. I joked with one of the developers on the project saying, “How did that get in here? Did someone take a night course can decide to ship his homework in the product?” Imagine my horror when then answer came back, “Yes – that is exactly what happened”. When we explained where the code base stood and what was required to move forward, the management team cancelled the project and decided to just release it with a few new bug fixes and features. The interesting aspect of this story is that the whiz kids responsible for this mess were viewed very positively – they got things done and delivered on time and they got promoted and moved on to go screw up bigger and better projects. This is an example of where ignoring technical debt lead to the death of a project. Often the results are less dramatic, you’ll get a new version of the product with fewer new features. Sometimes the results of ignoring technical debt are much more dramatic. Like our drone.
Drone technology is the darling of the US military/industry/political parties. They provide a tireless presence on the field of battle, they can project force without risking the political ramifications of putting US personnel in mortal danger, and they generate new military contracts for the latest and greatest technology. The original drones were relatively simple and effective monitoring devices. They were great but had a lot of well understood problems. The data they streamed back was unencrypted so anyone could grab the signals and see what we saw as well as get detailed operational information about what we looked for and how we looked for it as well as how we operated the drones. The designers were able to anticipate some problems and attempted to address them. This provides us a great example of where you have to be just as vigilant about a mitigation as you are about the problem it tries to mitigate. The designers understood that there were various hostile and non-hostile situations where communications between the drone and the controllers would be severed. They addressed this by programming the drone to detect the situation and return home using GPS and a preprogrammed set of coordinates for the home base. And here is where the technical debt problem comes in.
Apparently the military has known about the potential for GPS to be spoofed for 20-30 years. This is a technical debt. The Christian Science Monitor reports that what Iran did was to jam communications with the drone so it engaged it’s “fly home” program and then spoofed the GPS signal to make it land in Iran. The Iranians now have some of our most sophisticated military technology because we failed to address our technical debt.
When we decided to go-big on drones and transform them from a marginalized science project to the sharp point of the US military spear we faced a choice: we could either address our technical debt or we could add bells and missiles. It appears that we decided not to address our technical debt (or not to address it in time to avoid this mess). After all, where is the excitement in making things work? I’m sure some argued that, “They do work, they just don’t work as securely we’d like and we don’t know that others can actually spoof it”. Given a fixed budget, any money spent on addressing technical debt is money not spend adding new bells and missiles. To ship is to choose. And that is exactly the point – it is what separates the whiz kids from the graybeards.
Whiz kids will ignore technical debts, putting 100% of the budget on new features. They build houses of cards on foundations of sand and then move on before the waves come. They come off looking like superstars and then move on to screw up something bigger. As graybeards (and architects) we need to be the guardians of the future. We need to understand our debts and be vigilant in addressing them even though there is no clear and immediate benefit for doing so. This is easier said than done. It is easy when we are in charge and just set the rules. It gets harder when we don’t own those decisions and need to advocate for them and fight for budget. Few people are interested in hearing it so it takes courage to stand up and be a butthead to ensure the right things happen. Some deal with it “off the books” by padding schedules with a fixed amount of overhead to address the issues. One way of the other, it needs to happen or you end up with a disaster. You can only avoid problems for so long and then you have to address them.
This article needs to be sent to any and every CEO, president… anyone who makes decisions. It’s not just about technical debt: it’s about making decisions that aren’t as sexy but make the durability of a project. It’s like car manufacturers making their cars safer in a crash – even though most people won’t notice what they’ve done even if a crash occurs.
Wonderfully thought provoking article. I don’t have a software/systems design background and had never heard of this term before. Though I recognize in my systems administration and management history that I most certainly have been subjected to this bad debt, “…skipping documentation, ignoring supportability, deferring security work…”.
This is getting a bookmark, and its time for me to learn a little more about “technical debt”!
Thanks,
Joel.
Love this concept. I haven’t heard the debt metaphor but it works pretty well. It implies that defects left in over long term become more expensive. I’m not sure I buy that, but I can see defects like this GPS Spoofing getting downgraded more with every release that they are not exploited. “it must not be a risk since it has never happened.”
This is so being linked inside my corporate network.
Wonderful article and great gotcha. This implies that why we need to give importance on R&D, continious improvement so that you can make enough progress.
I’m not sure this is technical debt, as much as poor military strategy. For all we know the drone could have been written in sparkling correct code with triple redundancy and correctness proofs. The problem is no-one set a requirement for it to be tough to subvert, and that would seem to be a failure of military thought. You put a weapon on a battlefield, the enemy will try to neutralize it. You underestimate your enemy at your peril, even on a hi-tech battlefield there is no free pass for any country. You would think the military would play capture the flag with equipment like this before sending it out – have some group tasked with trying to subvert equipment of all kinds long before it goes into the field. Heck, assume espionage gave them the basic plans, let them have enough information to make some sophisticated attempts.
I think the buck stops with the generals. They are not dummies, and they study strategy. Lets hope this rattles their cage and they wake up to the general principle of this, not treat it as a special case of embarassment to be swept under the carpet. How much of the rest of our equipment has never been challenged by cunning and motivated hackers?
Excellent article–a definite must-read every time anyone kicks off a new project. As Jeffrey wrote, it’s OK to go into debt, if you do so knowingly (and in some cases, you should go into debt), and if you have a game plan for how to get out of debt, how you’ll pay back over next releases.
Love it. As someone who had to play in the sandbox he helped make at Intel…amen brother. And unfortunately, our demand for the next shiny object (and therefore demand to produce them in whatever hack-job is necessary for speed) and increasingly shortened attention span ensures that this problem will only get worse, not better.
I sent the para on financial debt to my teenage son as a nice summary :).
Great Article!
I too read this article with interest. What I find amazing is how no one seems to understand how the drown got away and flown to a safe landing. Technically, yes, the details are there. But with the cost on one of these drones, why was there no secure back door for control. Or better yet no phone home control or ability to remotely destroy the drone(s) as soon as those controlling realized it got away from them. It’s apparent there is no redundancy. Now, the technology is up for auction by the Iranians to the Russians and Chinese.
Technical debt is the cost imparted for the losses experienced from lack of due diligence.
Is this really technical debt or is it akin to the security guard leaving the back door ajar? On purpose…
Pingback: “Technical Debt” | It all comes out in the wash…
Pingback: Technical Debt « Eric Kool-Brown's Blog
Pingback: Iranian Drone Hack and Software Debt | OnTechnicalDebt
You don’t see this just in IT, you see it in every organization that’s built its processes and procedures on the failure of previous processes and procedures. The quick fixes and patches can pose bigger problems than the originals.
This article uses the drone capture as an example of this snafu, but fails to entertain another likely scenario… that the spy plane was a decoy MEANT to be brought down by these means. This could be an interesting Trojan Horse situation.