Tech Tuesday: Auto Code

Front page Wikimedia Commons, Gnu Emacs running as an integrated development environment

This Tech Tuesday is a bit different, since it’s a collaboration. The genesis of this is an article that Michael Cain sent me about the struggles of embedded software in cars. It’s a multifaceted problem, and not just for cars, but for IOT¹ home appliances, airplanes, pipeline control systems, the electrical grid, etc.

Michael has contributed the first half of this. I will fill it out with my own comments at the end.

So, without further ado, here’s Michael!

Forty-some years ago, I was the poor schmuck in Bell Labs systems engineering who went around and told managers “It’s a software world. In the future your projects are much more likely to fail because of software than hardware.” That did not make me popular, as most project managers at Bell Labs at the time had hardware backgrounds. It didn’t make me wrong, though. A year or so after I’d been on my soapbox a senior vice president lost his job because his huge project’s software was a disaster. His replacement’s first action was to announce a major redesign and a one year slip in the schedule. This month’s issue of IEEE Spectrum has a fascinating article on the same basic idea titled “How Software Is Eating the Car.” The whole thing is worth reading. I’m going to pick out some of the more quotable parts of the article.

Electronics systems (including software) account for an increasing share of the total car cost. Like so many other devices, cars have become a collection of networked processors equipped with some specialized peripherals. In 2010 about 35% of the cost to produce a car was electronics and associated software. The article estimates that figure will reach 50% by 2030. Why? The ultrasonic sensors used to provide parallel parking assistance cost roughly $1,300 to replace. The radar sensors used to monitor blind spots and cross traffic about $2,000.

Automotive processors are specialized. The automotive environment is difficult to deal with. Large voltage spikes in the power system. Noisy in the radio-frequency sense. Vibration. Communication protocols not used anywhere else. The market for critical automotive processors is dominated by one company who makes its chips in Japan. From January through May 2021, the global automotive industry produced 4.1 million units fewer than they wanted to because that supplier could not meet demand. The problem is not that the chips themselves are difficult to make. The company has accumulated years of in-house solutions to the specific sub-problems in the automotive environment. Processor supply problems are not unique to the automotive world; personal computer manufacturers cringe any time they hear rumors that Intel or AMD are having production problems.

Even base model cars are approaching 100 separate processes running some subset of approximately 100 million lines of source code. Much of that software is written by subcontractors and companies like GM or Toyota are simply integrators. Billions of combinations are possible. It is no longer possible for the auto manufacturers to build a test vehicle for each combination. It may be impossible for them to even simulate each one within reasonable time and cost. There is a non-zero chance that you will purchase a new car in the near future and be the first person to test that particular software configuration. This used to be referred to as the “Facebook model” for testing: roll the new version(s) out to 10% of the user base and see how loudly they scream.

Senior management at the car companies are simply not prepared. Peter Mertens is the former head of Audi’s R&D effort and a member of the board of directors. In a recent interview he said, “Run a job assessment with all top managers at VW, Audi, Porsche, BMW, and Daimler tomorrow and ask them to code a small game or a simple but working virus. If they are not able to do so, fire them immediately, because they are not fit for the job.” I think that’s rather extreme, but it’s a problem companies need to worry about. Not just automotive companies, either. The F-35 fighter jet project has become infamous for its software difficulties. Boeing’s unmanned test flight of its Starliner capsule failed to reach the International Space Station because of a software bug that should have been identified years earlier (approximate launch cost, $400 million).

Now that Michael has given you the highlights, let me fill in some more of the nitty gritty. For those of us who have dipped our toes² into the software development world, I expect much of this will have you all nodding right along. For the rest of you, let just explain that software development is not at all like hardware development, and software consistently lags behind hardware, by a lot. And there are reasons for that, some of them ‘good’ (for certain values of good), and some of them are face-palmingly stupid, but they are the current state of things.

Let’s start with an analogy. It’s not perfect, but I think it will help. Think about a bolt with a right handed thread pitch. It’s a piece of hardware. If it had a software component, it would be basically two instructions:

if (tighten)
    turn(right);
else if (loosen)
    turn(left);
else
    exit();

Now, we could expand that with some error checking to make sure the diameter and thread pitch of the hole or nut is correct, but that’s it. Also, all that software is in your³ head. Back when the world was all simple machines and hardware, life was good, life was simple.

Then the electrical engineers had to go and screw it all up with transistors and Integrated Circuitry (IC).

We don’t really need to go into the nuts and bolts of how the hardware, the IC chips and controllers, execute the logic, or how the machine code interfaces with those transistors. We need to go higher level, up the stack, as it were, because the problem is how those instructions are written. And because those instructions are written by humans, there will be errors. And because the humans writing those instructions are managed by people who have competing incentives⁴ when it comes to controlling for those errors, we get more errors. Controlling for errors is a never-ending struggle, one complicated by the fact that high level development⁵ lacks a certain degree of rigor. Why? It’s a trade-off. The higher level languages allow for more abstract thinking⁶ and for a larger population to be proficient at programming, but in turn, you sacrifice the rigor of the languages that work more directly at the machine level.

So, let’s talk about how software gets developed, the making of the sausage, as it were. We’ll use to a simple embedded system, an ultrasonic parking sensor for a car.

1. Build the req. This is the high level plan for what the software is supposed to do. For our sensor, it needs to be able to turn the sensor on, turn it off, and do something with the information from the sensor. Here is the first major decision that can cause problems, what is done with the information the sensor produces. Does it output pure signal response and leave it to the user to figure out what it means, or does it interpret the signal internally and output something like distance information? Or does it do both, and the user chooses based upon a hardware or a software switch?
2. Figure out how much memory you have to work with, and if it can be flashed (over-written). If it can be flashed, you have to plan for that. Because now it’s a code base that must be maintained.
3. Develop an architecture – in the simplest terms, this is the flowchart for the software, although software architecture is more complex than a simple flow chart. What are the inputs and outputs, how does the information flow, how is it changed, etc.
4. Review all of this with the development team, and the hardware engineering team, and possibly the marketing team, and whatever other business team feels like it should have a say in any of this and can get themselves on the calendar. Do this multiple times until you are in real danger of the schedule slipping. Then watch the schedule slip as the customer, or marketing, or some regulatory body, decides to stack additional features into the req, necessitating a review of the hardware by the engineering team, and a rethink of the software architecture, and round and round you go.
5. If you are smart, start writing your tests. If you are not smart, just skip straight to 6.
6. Start laying down code. Go back to 4. Repeat. Repeat so, so many times…
7. *A miracle happens! Code is in a testable state!*
8. Testing!
   1. Load the code onto the hardware and test according to the conditions in the req.
   2. Then debug the code.
   3. Or the hardware.
   4. Repeat 8.1 – 8.3 until testing passes. Be prepared at any time to return to 4 (Review).
9. Ship the product! Happy Day!
10. Learn that the customer has found a new and exciting way to use that hardware and software that no one thought of; OR! Learn that the customer has found a way to use the hardware and software that everyone on the hardware team and the development team was fully aware of, and had warned management about, but no one allocated the time or resources to fix that, because management could not grok that the user is just as inventive and smart as everyone on your team and was convinced that the user would never stumble across that issue.
11. Write and deploy a patch. Go back to 8.

Now, do that for every single computer module the car has. Remember, it’s close to 100, and many of them are much, much more complex than a simple parking sensor. And because all embedded systems these days use flash memory, rather than having the instructions burned onto the chip, it’s entirely possible to have someone flash your device and have it do other things, like return crap data, or dangerous data, or something totally unexpected, because the hardware can probably do a whole lot more⁷ than just run a parking sensor. So, a clever person could examine a sensor, and realize that there are more features that can be added to the software to do all manner of interesting things. Things that may not be healthy for the car. Or its operating system, or passenger.

Oh, and every device has different inputs and outputs, because you are not coordinating your vendors.

And if the device is more complex, and capable of more than on/off/send data, you have to start thinking about security, which, frankly, no one seriously does for embedded systems, because security adds a whole lot of complexity at multiple levels, and it requires a great deal of coordination between development teams. And if all your development is farmed out to multiple vendors, and you are just an integrator, that coordination is a hell of a problem. And it’s one that if the people managing the coordination are not dialed in to the scope and severity of the problem…

Worried yet? No (What is wrong with you?!)? Yes? _{Just say yes, for the love of…}

Good, but we aren’t done yet.

Look back to item two in my list, notice the last sentence;

Because now it’s a code base that must be maintained.

Oh, code maintenance, how I loathe thee…

Not my code, oh no! My code is well structured, and clean, and fully documented with clear, descriptive comments everywhere.

Not like your code, which has maybe six lines of comments for every 1000 lines of code, and functions that are only called once, while other functions are 1000 lines of repetitive code that should be worked into a handful of function calls, and it is all full of clever bits of compact logic that take a half hour to untangle, and conditionals that go nowhere and can never actually be executed and serve as nothing more than a visual distraction while I am trying to figure out where your bug is and how to fix it⁸. Because the team didn’t enforce any kind of commenting or documentation standards, or management was lax with code reviews because the schedule was slipping, or it was from a development house that thought ‘Agile Development‘ meant being quick to market, or ‘Clean Code‘ meant the file server got dusted once a week with a can of air, etc.

So now we don’t just have the difficulty of developing the software for an embedded system, and making it secure, and coordinating the integration, but we also have to maintain that software, across versions, and recalls, and bug fixes, and outsourcing, and in-sourcing it back, and acquisitions and mergers. Don’t discount those last bits. Inheriting someone else’s code from the same team is bad enough, inheriting code from a completely different company, especially if that company was in a different country… Sometimes it’s better to just re-write the whole thing from scratch.

Seriously, there are days I sincerely think about just going back to being an aerospace engineer; I hear SpaceX is hiring…

So, let’s sum this up. We have:

• Machinery that is using more and more computers
• To manage more and more of the subsystems
• Requiring more and more software to operate the computers
• Where the companies who are designing and assembling this machinery
• Do not have, and do not want, any expertise with regards to software development and management
• And they are trusting that the general public will remain largely ignorant of the mess they are putting themselves in⁹.

Well, until Boeing exposed the problem with MCAS. No, not the problem with MCAS, the failure of MCAS exposed the problem of companies like Boeing who want to pretend they are simply systems integrators and don’t have to concern themselves with the software of the systems they are integrating. It boils down to this:

If you are a company, and you are installing into your product any electronic device with software that exists on a flash memory module, you have to starting acting like a software company, even if you don’t write a single line of code¹⁰. You have to understand how the software is developed, and tested, so you can competently work with suppliers and understand the limitations of their hardware and software. You have to understand how patches work, and how security plays into all this. Management, and executives, have to have a clue, they can’t just shove this off on the engineering team. Because when the fit hits the shan, no one is going to care that OCP is the one who made the defective module for GM¹¹, or that the avionics control system on a 737 was written by a subcontractor in India.

By the way, EVs will be even more dependent on computers and embedded software. Guess which company is actually doing better at this (if not perfectly)? Yep, Tesla. They write everything in-house.

Oh, and just to cue up the next problem, all the EV batteries and their source materials are made or mined in China.¹²

1. Internet of Things
2. Or our whole selves.
3. The person with the wrench.
4. Release schedules, bonuses, ego, etc.
5. Pretty much anything above working directly on the chip instruction set or memory registers.
6. Compilers take care of translating the higher level languages to machine code
7. Hardware always outpaces software.
8. Yes, I have a tool like this I am responsible for, why do you ask?
9. Wait, I’ve heard this before… Oh Yeah! – Structural Engineering is the Art of molding materials we do not wholly understand into shapes we cannot precisely analyze, so as to withstand forces we cannot really assess, in such a way that the community at large has no reason to suspect the extent of our ignorance.
10. And the more computers you have in your product, the sooner you will be writing lines of code for it, so get a jump on the mindset early.
11. This is what happens when you repurpose the ED-209 system for pedestrian collision avoidance; it was only a matter of time before a farmer’s market became a bloodbath
12. I do enjoy using footnotes, thank you for noticing!