#015 - Modernizing Embedded Systems: Step #1 - Overhauling Your Build System

Jacob Beningo (00:08.226)
Welcome to the Embedded Frontier Podcast, where we explore modern embedded systems development techniques so that you and your team can develop faster, smarter firmware. I'm your host, Beningo, founder of Beningo Embedded Group, the Embedded Software Academy, and co-founder of the Embedded Online Conference. All right, well, in the last couple of episodes, we have started to look at a seven-step framework that you can use to develop faster, smarter firmware.
And as you may recall, that framework involved, first of all, overhauling your build system. Second, modernizing your architecture. Third, implementing DevOps practices. Fourth, adopting test-driven development. Fifth, incorporating simulation. Sixth, leveraging AI and ML technologies. And then the final step was establishing a metrics scoreboard. Now in today's episode, what we are going to do is dive deeper into discussing
how you can overhaul your build system. Now, build systems in general are something that many of us are familiar with, right? You might, when I say build system, you're probably thinking about that compiler that you use, right? The thing that's taking your code, whether it's written in C, C++, Rust, or dare I even say assembly, and it's converting it into an executable binary that we can run on an embedded system, whether that is an embedded Linux machine or my favorite, a real-time microcontroller based.
system. Now overhauling our build system is really important because I have found from talking with engineers, working with clients on many different projects, that the way that people build their software today is still very antiquated. All right. And what happens here is that a lot of us are still building software the way that we did 10, 15 years ago. But the fact of the matter is, is the way that we even build our software today,
has dramatically changed for embedded systems. It's no longer just simply convert, you know, whatever code I wrote into something that I can then deploy out of my system and calling it good. A build system is much more than converting your code. It actually involves things like the development environment that we work in. It involves our compiler. It involves scalability. What we use actually debug and test our system. We could even argue potentially that
Jacob Beningo (02:31.949)
our build system moves all the way into our DevOps solutions in our CI-CD platform. And so just building code the way that we always used to really isn't going to help in today's fast-paced smart development activities. Now, many of you might say, well, it might seem like the way that I build my software is just fine, right? But the fact of the matter is the way you build your system is probably actually holding you back.
Well, many times we find something that works and we stick with it. There are times when we need to start to pivot and adjust how we are building our software. Now, what you might find as an example, right now we have seen over the last five years periods of time where microcontrollers, you just couldn't buy them. If you remember back and think back to COVID years, the supply chain was very pressed and some lead times for microcontrollers, even the most popular and widely available ones,
The lead times actually turned out to periods of like six and 12 months. It got to a point where people were de-soldering chips off of existing products so that they could have enough chips to launch a product or ship product, right? And that's not the position you want to be in. You want a build system that's flexible and not tightly coupled to your microcontroller vendor. Okay. And so supply chain changes can be a big issue where you have to make sure your build system is flexible enough. You want to make sure that
the workflow that you follow to actually develop software is more modern. What you'll find today is that every developer actually has their own preferences on how they want to develop software as part of their workflow. It might be that, you know, when I sit down to write code, I want to use Visual Studio Code. I might have plugins I want to use. I might have extra tool sets that I find help me write code extremely quickly and test it as well. But you might find that
you know, another engineer, a colleague of mine, or maybe some team that I'm working with, they might have totally different preferences. Maybe, you know, some developer on the team is, would rather use the vendor supplied IDE, right? Because of all the way that all the components and everything are already pre-integrated, there's a lot of tool sets that come with that. And so if you try to force any individual developer to adhere to, you know, an overall company workflow, that can be quite challenging sometimes. And the challenge is that one developer could be working very fast because it's
Jacob Beningo (04:54.656)
part of the workflow that's natural to them, while somebody else may be actually getting less done and be less efficient and less productive because it's a workflow that they're trying to adopt to that just is kind of foreign and doesn't really fit well with the way that they think and develop things. And so a build system that is flexible enough for people to adapt based on their own needs and preferences is really important. That's one of reasons why I like Visual Studio Code. A lot of the projects I work on, I'll set my build system so I can build directly from a command line.
I'll have it so can build and develop my code within Visual Studio Code, but I'll also have it so that I can use the vendor supply tools if need be. Because again, the tools provided by the vendors are often quite good for debugging and working with their chips, but a lot of times they don't necessarily have adopted kind of a modern workflow. And so you might find that working with CI-CD or CMake or some of these other tools and capabilities that are out there today is very tough to do from within a vendor supplied IDE.
And so for that matter, that's why I like Visual Studio Code. But I'm kind of digressing here a little bit, right? You know, the IDE that you use is part, an important part of your build system. So don't forget about that. Choose something that's definitely flexible and allows each person on your team to customize what their workflow actually looks like. Okay. So workflows are really important. A workflow is nothing more than something that you can sit down and say, hey, this is the way, these are the steps that I take when I develop software.
And you should find that your workflow is pretty consistent, at least on a developer to developer, their workflow should remain pretty consistent. And then working with that workflow for building your software, you can have a team-oriented workflow, but make sure that you leave the flexibility in place for individual preferences and so that you don't handcuff any of your development team. Sometimes you'll find that there are team members that will be deemed to be not as productive as others.
And you know, when you try to compare people, right? And part of the problem that I have often found is that it's not that the other person can't do the work or can't be productive. It's often their handcuffed by being forced into a workflow or a process structure that just doesn't meet their natural way of developing software. So having a build system that has that flexibility, I think, is really important.
Jacob Beningo (07:13.58)
I've also seen with some teams where builds can actually be unreliable. This can be a pretty big problem where maybe I build code on my system and it works perfectly fine. And then on another team, you know, another team member, they build code and it doesn't work. I can even remember a situation one time where I was working with a customer and I was, I was working with, you know, I won't get to say the compilers and IDs and stuff like that, but we were, we were working on a project and I was trying to build a code and it just was not working.
Okay. I could not get this thing to build on my machine. And, you know, my customer was like, no, this is, it's building and it's working perfectly fine. And as I try to drive in and try to figure out, like, I'm getting hard faults. I'm getting really weird things happening here. Like this doesn't make sense why we're getting two different results, compiling the exact same version of firmware. And as I dug into it, what I ended up discovering was that I was using the latest and greatest version of the compiler from the Silicon vendor. And it turned out that there was a bug in it that was causing the hard faults that.
when they compiled the code that was causing a hard fault that was then creating the issue. Okay. And so because we were on totally different tool chains, using different computers, all that fun stuff, we were working with totally different results or getting totally different results from the exact same code. And so, you know, that can be a real big issue, right? And modernizing your build system is a good way to get around that. All right. And I'm going to share some techniques here with you in a little bit of what you can do to ensure that you don't have those types of issues.
Okay. Right now I'm trying to share with you some of these issues with the way that I've seen people building software, embedded software today. And then we'll talk about some of the solutions that I found that really go along the lines of helping you to modernize your build systems. Okay. So reproducible builds is a big problem. I've seen, you know, painful experiences trying to adopt things like DevOps. I've worked with teams where we're like, all right, let's go and we're going to adopt DevOps. We're going to get a nice CIA.
continuous integration pipeline running where we can build our software and analyze it. And lo and behold, the modern processes that we were trying to adopt, the silicon vendor with their SDKs and the tool sets that they were providing to the company, it just didn't fit well. And so what we found was that the build system we were trying to use from the vendor just was impossible to get working with a good modern CI, CD workflow.
Jacob Beningo (09:36.78)
All right, and that's, can, again, it can hold you back if you can't leverage that. You want to be able today to be able to automate as much of your CI, CI, and even parts of your CD, the continuous deployment as possible. If I can commit code and have it analyzed and tested and get reports back on, on whether we're meeting the metrics that we need to and whether everything is good or, you know, maybe I made a mistake and there's some warning in the compilation that could result in a bug. I want to know those things as soon as I commit my code. I don't want to have to wait three months for a QA department.
to come back and tell me that they tested my software and there's a problem with it. I want that instant feedback or near instant feedback. And so a build system that is still antiquated or isn't modern, you may have issues adopting other modern workflows like CI, CD and DevOps. Okay. And you may even find that your system with the way you're building software today, it doesn't scale very well. A lot of teams that I work with today, they're not creating a single product that they're going to ship to one customer.
A lot of them are thinking much bigger than that. They're saying, hey, software is expensive to develop. Embedded systems are expensive and they're complex. We want to build a platform that we can leverage for the next five to 10 years. And maybe we want to build a half dozen or dozen products off of that one core code base. And maybe we're not even going to have just a single customer. Maybe there's multiple customers with multiple SKUs or maybe even nearly very different needs and requirements for those different systems, but still a common core code.
And when you have that type of requirement, the way that we build software today just isn't going to cut it. All right. You're going to find that you're going to be spending more time fighting your build system than anything else. And so part of this idea of overhauling your build system, I think this is a great place to start. Okay. I would even argue that the next step, which is all about your software architecture, modernizing your architecture to some degree could actually, you know,
work into how you modernize your build system, because the architecture of your software, the way that you build it may be dependent on the architecture. But for the most part, you'll find a lot of modernization we can do to our build system without the need to make those changes to the architecture. OK, so I've shared with you some of the common problems that I've seen so far with the way that we build embedded software today. What are some of the potential solutions? If you're going to overall overhaul your build system,
Jacob Beningo (12:04.244)
What should you be looking at doing? Well, one of the first things I always recommend, and I've seen a lot of teams actually already taking this step, which I think is great, and that is that you containerize your build system. Now, containerization has a lot of benefits to it. Now, I personally use Docker. I know some IT teams don't like Docker, but there are alternative open source container technologies out there. I've seen lots of commercial companies also trying to bring containerization to embedded systems as well, like actually running on the microcontroller.
But we're not talking about those types of that type of containerization. We're talking about on your, in your build system, on your build environment, even within your DevOps platform. We're talking about containerizing all the tools and scripts that you use to actually build, analyze and develop your software. And Docker is really cool because you can write a single script that installs all the tools that you need. It can go out and grab the right version of Python. It can grab, you know, auto config. It can grab.
The compiler that you're using, can set up your analysis tools. It could set up testing harnesses like CPP U-Test or Unity or Google's test harness or whatever your preferred set of tools actually are. And what that does is it provides you with a consistent build environment. And so I told you a little bit earlier about when I was working with a team and for me, I went out and just installed the latest tools and I couldn't build their code. And it turned out they were on an older version of compiler.
Well, if they had containerized their build system and given me their Docker file, and then I just built the image, I would have had no problems compiling their code. And what is even better is I would have saved several hours of digging in to try to figure out why is this code not running the way that it's supposed to? Because let's be honest, what was the first thing I did when I got hard faults? I didn't stop and say, what was my tool chain? I said, okay, what's causing the hard fault? Right. And I started going down the rabbit hole to try to investigate.
why the code running on the system is giving me hard faults. When in fact, you know, and I'm not going to, I waste days. mean, after looking at things for, know, an hour or hour and a half, I kind of said, well, these, really isn't making a whole lot of sense. Let me step back and, you know, figure out what the tool chains are and make sure everything is actually, environment is set up properly, but that could have saved me, you know, a couple of hours of development and pain, right? And so containerizing your build, that is one great benefit. You end up
Jacob Beningo (14:29.12)
All developers are working from the exact same tool sets. Okay. No inconsistencies there. Secondly, we all know when you bring a new member onto the team, what happens? You spend a day installing tools on a machine or IT gets a computer and they go through and they install all these, these tools and stuff on a machine. Right? Well, that's so much easier if you just have a container that you build the, build the container and then the person has everything they need to build the software. Right? So that right there, a couple of really big,
benefits. Now you might be saying those are pretty trivial benefits, right? You were talking about saving maybe a day over the course of a year or something, or, know, we only bring maybe a team member on once a year, every couple of years or something, if you're small and you don't have a lot of turnover. And so maybe those aren't big benefits. The biggest benefit though, I've saved it for last, is that containerizing your build systems sets you up to be able to adopt DevOps. If you have your build system containerized, we can easily
set up a continuous integration pipeline that uses that container and automatically builds your software, leverages the tools that are in that container to test your software. So we get test automation built in there. We can analyze it using tools like CPP Ucheck or other commercial tools that are available. I'd like to use tools like, well, I won't name specifics. Okay, I'll keep things very general for us so that I...
You know, we all have our favorite tools that we use, but that gives you the idea, you know, whether it's a commercial tool or whether it's a open source tool, we can have these things so that they easily integrate into a CI-CD platform. Now, instead of fighting our CI-CD workflow or processes, it seamlessly integrates using Docker and we can get a very repeatable build happening on our system. Okay. So containerizing our build system, that is one of the first things that you can do to help overhaul your build system.
The second thing that you can do is you can adopt CMake. Now I have found that a lot of embedded teams are starting to slowly adopt CMake. There's still a lot of make file scripts out there that people are manually maintaining. There are still a lot of make files from the Silicon vendors that are automatically provided by their IDEs that people are leveraging. There's nothing wrong with make or the make files are generated by the Silicon vendors. But what I have found is that they...
Jacob Beningo (16:52.53)
oftentimes don't provide you the flexibility you need if you're building a platform based system. If you want code that scales as flexible and reusable across multiple product lines or customers, you really need a system that builds your software in a modular componentized manner. And the solution for that is to adopt CMake. CMake allows us to standardize our outputs, standardize the way that we build components and even the entire project.
Okay. It provides us with a nice clean separation between configuration of our system and the source code. And the cool thing about that is that each of my components that I create in a folder, I can have my own individual CMake file that tells how to build this little individual component that later on is used as part of the larger system. All right. I can have a high level CMake file that then builds all of, know, CMake is basically a configurator and I can, you know, use it with Ninja.
And I can use it with other tools to generate the make files that need and get a little bit more performance out of even my builds. Now I've seen personally along the lines of like 10, 15 % improvement, sometimes more in the speed at which I can build my software using CMake with Ninja. I've also talked with other teams that haven't seen any performance improvements. So, you know, CMake paired with Ninja, you know, in theory, it's supposed to speed up your builds, but you know, mileage may vary essentially, right? But.
I have found that that pairing those two together provides you with a very flexible build system that allows you to adapt your existing code, which let's be honest, is probably one of your company's greatest assets, right? Is the software that you're writing. And so if you can reuse that asset over more projects with more customers without having to rewrite it, and you can minimize your technical debt, then guess what? Your team is going to be in a much better shape in order to be successful as time goes on.
Okay. And the cool thing about using a system like CMake, you know, I talked about the flexibility. It kind of sets us up as well to be able to adjust our builds as we have to switch, maybe microcontrollers. Now I know we don't like that idea of having to change microcontrollers or hardware that we're working with, but in the global environment that we are currently in, supply chains are often disrupted. All right. And you don't know when suddenly the microcontroller you designed in is going to go up dramatically in price.
Jacob Beningo (19:17.29)
And that may force you to redesign your system. And if you have a build system that has been tightly coupled to your code and the specific microcontroller, you're going to have a lot of expense and time to go back and have to change the microcontroller, your build system, and even the software that interacts at the lowest level with the microcontroller. Okay. And we'll talk about architectures and decoupling in the near future when we get to step two, which is all about modernizing your architecture. But for us today, you know, if you leverage
CMake with Ninja and Docker, you can have an overhauled build system that will allow you to be flexible for when there are supply chain disruptions or when you have customers that come up with new requirements or behaviors that you need. And that can really dramatically change, I think, the way that you develop software and ensure that you are set up to be flexible and scalable. Now, you've probably heard of these two tools, right?
Docker and CMake, they've been around for quite a while. Another tool that I think a lot of teams should start to adopt is the open source tool, West. Now West has traditionally been, it was created and it is used to maintain the Zephyr RTOS build system. Now the interesting thing with West is that it is a scalable multi-repo project manager. All right.
Now, what on earth do I mean by that? Well, a lot of embedded software teams will create a repo for their project and they end up with one master repo that tries to govern everything. And while that can work, and I've seen lots of teams be successful with it, and I've done that even on my own projects, what I have found is that a modern build system is going to actually leverage a tool like West so that you can create multiple repos
create a high level manifest file that then specifies which repos are going to be included in your project. Now, you might be saying right there, well, Jacob, I could just use, you know, sub modules and I'll be perfectly fine. And the truth is you could use sub modules. That is another method to go about doing that. But what I have found is that by using West, West provides that high level manifest file that you can customize that specifies the exact version of
Jacob Beningo (21:44.634)
the repo and the branches and everything that you want to use so that you can ensure that at the project level that you're building for, you actually have the right repos, you have the right versions that you're working with, and that can ensure that you don't have any mistakes with your builds. It can provide scalability. It can also provide a lot of high level flexibility. So suddenly now I can create separate repos for my low level drivers, maybe for different microcontrollers. And if I follow a good architecture, I can abstract out then.
You know, maybe the RTOS is that I'm using and put a nice abstraction layer there. And then maybe I have different application components that are stored in a, you know, a couple of repos. Now suddenly I can create a high level West manifest that goes out and pulls, you know, this specific board or this electronic repo. And this one for my drivers and this one for just this application pieces that I need. And suddenly you have something that's far more scalable and flexible. so West allows us to have that modularity.
when it comes to our source control. And it allows us to manage our overall and overarching project much easier and conveniently. Okay. Now, as it turns out, you're going to see here that because of that, you you're actually going to find that a lot of modern build systems, there's a lot that has come out of the open source movement and out of open source tools and open source projects on what you really need to do to overhaul and modernize your build system. And there's certainly other things that we could do. For example, you could adopt
tools like Kconfig. This is basically a declarative config system that comes out of Linux that allows you to easily configure individual components, maybe security libraries, real time operating systems, graphical user interface code, things like that. I think that is another great tool that teams can leverage. But I'll be honest, I think for a lot of teams using something like Kconfig is probably overkill. At a minimum, I see an overhauled modern build system.
to include some type of containerization technology, leveraging CMake and Ninja for builds, and then using West to manage the different repos for the projects that are being used. Okay? And so I think when you go through and you take those major pieces, what you will find is that it creates a far more flexible build system. One that's scalable, one that's adaptable, one that is...
Jacob Beningo (24:12.262)
platform based so that you can create a platform that you launch multiple products off of or that you support multiple customers with or maybe both well into the future, five, 10 years from now. While these individual technologies may change and certainly others could be introduced, I think at least for the next five or 10 years, we're going to find that Containers, CMake and Zephyr, not Zephyr, Zephyr is the RTOS, West are going to be
the major tools that find their way into a modern build system. And for some of the, know, at least for West, we have Zephyr RTOS, I think for that, and the open source movement. But I also think that if you are very much looking to have a build system that is modern, that is flexible, you know, looking at something like Zephyr RTOS can give you really good ideas on how you can modernize your build system. And as I mentioned, you know, this is step one. If you have a modern build system,
It sets up your ability to scale, to be flexible, and to be able to deliver faster and on time. And certainly by doing this, this is making the way that you develop your workflows far smarter. Okay. And certainly we can leverage AI and machine learning technologies to really make our system smart. But that's not for us to talk about here at the build system level necessarily. And even as I say that, I realize that
Another technique that you can use to help overhaul your build system is to leverage artificial intelligence. If you have a legacy build system, you very easily could leverage tools like ChatGPT, for example, or Claude, and you could go through and say, hey, here are my make files. How can I modernize this and to create a modern CMake Ninja build system? And you could, you know, it may not be able to do it all in one shoot, right?
But you can leverage that in order to try to progress and improve your build system over time. You don't necessarily have to do it all in one shot, but try to use some of these new technologies to improve the way that you are building your system. I remember when I first started working with AI tools, I had a SCM32 project that I was working on where I was working within their development environment. They generate a really nice make file. And yes, their make file was great. And I could go and create my own custom CMake file.
Jacob Beningo (26:36.4)
But wanted to make sure that because I was using their tool chain, that I created an equivalent file and it was going to take me probably a little, you know, several hours, half a day to a day to go through and really make sure everything worked the way that it was supposed to. Well, what I did was I took the existing make script and I used AI to generate the CMake files and I gave it my project directory structure and, and, you know, kind of work with it there. And over the course of maybe 30 minutes, I was able to migrate my
make file based system over into a CMake based system with Ninja with an overarching project script that allowed me to build either with the ST tools or with this new CMake system that was in place. And of course I had to tweak things in order to really make sure that it was the way that it should be and the way I wanted it to be. Right. So I'm not saying blindly rely on AI. You have to look at the output and tailor it. But within a very short period of time, I was able to build
with CMake and Ninja and adapt that over to work within Visual Studio Code. And so what I'm getting at here is that these technologies, if you know how to use them, can dramatically help you overhaul your build system, provide that flexibility that you need, and ensure that you are working with a modern build system that will help you not just today, but even years from now so that you and your team can be very efficient, productive. And let's be honest, it can be removed of stress.
all the extra long hours, working weekends, all that kind of stuff can go away, you know, if you're using the right build system, along with many other things that go into a development cycle. So it's not just one thing that that results in that. But as you can imagine, you can become a lot faster and smarter about the way that you develop your modern embedded systems today. All right. So with that in mind, this gives you hopefully a little bit of an idea of
what overhauling your build system actually means, and hopefully gives you some ideas of different tools and technologies that you can use in order to actually do that. And in order to kind of help you go a little bit further, in our next podcast, we are actually going to be talking with Kyle Dando. He is an NXP FAE, and he is going to be kind of sharing what NXP is doing with their SDKs. They have a new MCU Expresso SDK that has adopted a lot of these
Jacob Beningo (28:58.736)
build modern build system capabilities and technologies. And so we're going to sit down and talk about what NXP is doing and how they've overhauled their build system so that it is adaptable and modern. And we'll learn a little bit more about that. So I think that'll be a really good way for you to see some practical applications of these different pieces into a modern build system. And certainly if you find that you're stuck and you are, you you and your team are trying to figure out how you overhaul your build system.
You can always reach out to me and we can certainly discuss how I can help you to overhaul your build system as well so that you have a nice modern build system that can then help you with the additional steps that we're going to be talking about over the next several months on how you modernize your embedded software development. So with that in mind, thank you for tuning in. Certainly feel free to provide me with feedback in the comments below. Reach out to me if there's any questions or ways that I can be of
further assistance. with that in mind, happy coding and I look forward talking with you next time.
Jacob Beningo (30:07.142)
you