Linux Profiling with Mohammed Billoo

We sit down with Mohammed Billoo, founder of Mab Labs and author of the Embedded Linux Essentials Handbook, to explore the world of embedded Linux profiling and optimization. Mohammed shares hard-won lessons from the field, including debugging a scientific instrument that mysteriously crashed after 60-minute runs and optimizing a sophisticated MANET platform that took a 20% throughput hit.

The conversation reveals a fundamental truth: in embedded Linux, the CPU is rarely the bottleneck. Mohammed walks us through his systematic approach to performance problems, starting with simple tools like HTOP before diving into specialized instrumentation. We discuss the critical difference between VM size and VM RSS for memory analysis, why dumping console output can kill boot times, and how to leverage kernel configurations for maximum diagnostic bang-for-buck. Mohammed emphasizes the importance of building instrumentation into systems from day one—not for premature optimization, but to give your future self the data needed when problems inevitably surface. The discussion also touches on how LLMs can accelerate the learning curve for complex tools like Valgrind and perf, while stressing that physical reality remains the ultimate arbiter of system performance.

• [03:15] The surface area problem: why embedded Linux profiling requires a tool chest, not just a toolbox
• [06:30] Case study: debugging a scientific instrument that crashed after 60-minute runs
• [08:45] VM size vs. VM RSS: understanding the critical difference in memory analysis
• [14:20] Why the CPU is rarely the bottleneck: coprocessors, DMA, and crypto engines
• [18:50] Essential kernel configurations: function tracer, perf, and config kallsyms
• [24:10] File system bottlenecks: moving from CSV files to SQLite for data integrity
• [28:40] Boot time optimization: why console output is one of the biggest time sinks
• [32:15] Premature optimization vs. smart instrumentation: building in diagnostic capability from day one
• [38:25] Leveraging LLMs for visualization and analysis of perf data and Valgrind output
• [43:50] The first five commands: starting with HTOP and working down to specialized tools

"When you first get started, you have generally this arrogance that like, oh, it works fine. I've tested it. It's good to go. But then as you get more experience, as you become a more senior-level engineer, that arrogance, you start to kind of strip away a lot of that arrogance. You get humbled pretty quickly." — Mohammed Billoo

"The CPU is very rarely the bottleneck because it's meant to, and the drivers are implemented in Linux in such a way that they're intelligent enough that they can hand off a lot of the things of the CPU to coprocessors so that the CPU is really idle." — Mohammed Billoo

"I don't convince myself of a claim that I'm making until I have data to back it up. So I don't say, oh, you know, this is working fine. Like, well, again, what does fine mean? Or, you know, what does well mean? And what is the data to prove that?" — Mohammed Billoo

• HTOP - Interactive process viewer for Linux - Mohammed's first tool for getting a high-level view of system performance
• perf - Linux profiling tool with performance counters - requires kernel configuration to enable
• LTTng - Linux Trace Toolkit Next Generation - provides visibility across both user space and kernel space
• Valgrind - Memory debugging and profiling tool for detecting memory leaks
• iperf - Network throughput measurement tool with server and client components
• GStreamer - Multimedia framework with built-in tools for per-frame timestamp analysis
• Tracealyzer - Visualization tool for LTTng and other performance data
• SQLite - Embedded database recommended for data integrity over CSV files in embedded systems
• Embedded Linux Essentials Handbook - Mohammed Billoo's book published by Packt
• Mab Labs - Mohammed Billoo's embedded solutions consultancy with blog on embedded Linux topics

You can find Jeff at https://jeffgable.com.
You can find Luca at https://luca.engineer.

Want to join the agile Embedded Slack? Click here

Are you looking for embedded-focused trainings? Head to https://agileembedded.academy/
Ryan Torvik and Luca have started the Embedded AI podcast, check it out at https://embeddedaipodcast.com/