The Embedded Buddy System

WEBVTT
NOTE
This file was generated by Descript
00:00:13.702 --> 00:00:15.442
This
is Self-Directed Research!
00:00:15.547 --> 00:00:18.884
James and Amos get hyped about different
topics and take turns each week,
00:00:18.914 --> 00:00:20.414
presenting their ideas to each other.
00:00:20.655 --> 00:00:22.485
This episode wraps up season one.
00:00:22.608 --> 00:00:23.898
We hope you enjoyed it so far.
00:00:24.204 --> 00:00:27.084
We'll be taking a little break over the
holidays, but we'll be back next year.
00:00:27.330 --> 00:00:30.780
Make sure to like follow and subscribe
wherever you find us to stay up to date.
00:00:31.050 --> 00:00:35.339
And as usual visit
sdr-podcast.com/episodes for all
00:00:35.339 --> 00:00:38.129
of the presentations, videos,
show notes and transcripts.
00:00:38.374 --> 00:00:41.974
This week, James makes his pitch
for "The Embedded Buddy System."
00:00:42.359 --> 00:00:44.989
This episode is sponsored
by Poststation, a new tool for
00:00:44.989 --> 00:00:46.579
embedded systems from OneVariable.
00:00:47.069 --> 00:00:50.294
Check out the link in our show notes or
listen at the end for more information.
00:00:55.544 --> 00:00:59.734
This week I invented a whole new term for
something that I've been doing for years.
00:00:59.809 --> 00:01:00.939
It seemed like a good name.
00:01:00.939 --> 00:01:03.639
So, this week we're going to talk
about "The Embedded Buddy System."
00:01:03.939 --> 00:01:04.269
Ooh...
00:01:04.419 --> 00:01:07.049
The alternative
title here is actually
00:01:07.359 --> 00:01:11.369
"James' Cheat Codes for Low to Mid Volume
and Rapid Embedded Development" because
00:01:11.369 --> 00:01:16.739
it's basically how I've done almost every
customer project for certainly the last
00:01:16.989 --> 00:01:20.869
three years but probably more likely
the last five or six years because I
00:01:20.869 --> 00:01:24.089
end up helping a lot of companies that
are either startups so they're making
00:01:24.089 --> 00:01:26.887
their first batch of hardware where
they're shipping the first thousand
00:01:26.887 --> 00:01:29.939
units or they're shipping the first
10 units so they can get feedback
00:01:29.939 --> 00:01:31.599
from people and things like that.
00:01:31.599 --> 00:01:35.705
So they're not necessarily at the point
where they're making a million items yet.
00:01:35.705 --> 00:01:39.525
And a lot of what they're doing is either
trying to figure out what their device
00:01:39.525 --> 00:01:43.045
should do, or for more specialized people,
they're never going to ship a million
00:01:43.045 --> 00:01:46.867
units because it's some huge piece of
scientific or industrial equipment.
00:01:46.867 --> 00:01:49.737
And if they sell 20 a year, they're
going to be pretty pumped because
00:01:49.757 --> 00:01:51.667
they cost six or seven digits a piece.
00:01:51.987 --> 00:01:55.357
I help out a lot with
those kinds of problems.
00:01:55.781 --> 00:01:58.991
If you're building something
embedded today and you're
00:01:58.991 --> 00:02:01.161
designing it from scratch, you're
not iterating on something.
00:02:01.161 --> 00:02:05.551
You don't have some really ridiculous
constraint on size or anything like that,
00:02:05.841 --> 00:02:09.621
you should probably consider this
buddy system because I know you're
00:02:09.621 --> 00:02:12.241
going to immediately have some- I'm
talking to the embedded developers
00:02:12.241 --> 00:02:15.261
out there- you're going to immediately
have some pushback like, "Ah, this
00:02:15.261 --> 00:02:16.601
certainly doesn't address my needs."
00:02:16.601 --> 00:02:20.361
And for a very few of you,
you will be very, very right.
00:02:20.631 --> 00:02:22.531
But more of you than maybe you'd think.
00:02:22.531 --> 00:02:26.541
So keep an open mind on this one, embedded
developers and stick with me on this.
00:02:27.539 --> 00:02:32.989
What I mean by the buddy system is we have
some more powerful embedded Linux SoC.
00:02:33.019 --> 00:02:37.449
This is like Raspberry Pi class,
something, you know, ARM Cortex-A that
00:02:37.459 --> 00:02:39.589
has a fair amount of processing power.
00:02:39.854 --> 00:02:42.578
These days they're getting faster
and they're getting more powerful
00:02:42.578 --> 00:02:44.678
and get them in fairly tiny sizes.
00:02:44.908 --> 00:02:45.698
But they're running Linux.
00:02:45.718 --> 00:02:48.278
They are more like a computer
than you would think.
00:02:48.798 --> 00:02:51.628
And on the other side, we've got
our bare metal microcontroller.
00:02:51.638 --> 00:02:55.859
This is more Raspberry
Pi Pico or RP2040 class.
00:02:55.869 --> 00:02:59.389
You know, we're talking ARM Cortex-M
or other 32 bit microcontrollers
00:02:59.399 --> 00:03:03.089
these days and having them
linked up by some interface.
00:03:03.119 --> 00:03:07.089
I have USB in this one, but it could be
whatever you happen to have available,
00:03:07.129 --> 00:03:10.239
but have them working together as buddies.
00:03:10.999 --> 00:03:13.616
There's a real push to try and pick one.
00:03:13.616 --> 00:03:17.526
You go: well, is this a microcontroller
sized project, or is this an
00:03:17.526 --> 00:03:19.596
embedded Linux SoC sized problem?
00:03:19.596 --> 00:03:21.636
You go: well, obviously
I want it to be cheap.
00:03:21.636 --> 00:03:23.466
I don't wanna have a
bunch of things going on.
00:03:23.676 --> 00:03:27.846
I wanna try and do it all in one
chip, and I think that is a foot gun.
00:03:27.906 --> 00:03:31.846
Maybe not on day one, but certainly
as the project grows, it can
00:03:31.846 --> 00:03:33.436
very quickly become a foot gun.
00:03:33.616 --> 00:03:35.376
James, I think this
is going to be a very short episode
00:03:35.406 --> 00:03:37.106
because I wholeheartedly agree with you.
00:03:37.106 --> 00:03:39.786
I mean, I've never done this
professionally, unlike you, but
00:03:39.786 --> 00:03:41.106
it does seem to make sense to me.
00:03:41.116 --> 00:03:45.126
Like, don't try to do everything on a
system that is only suited for half of it.
00:03:45.126 --> 00:03:46.076
Just have both.
00:03:46.106 --> 00:03:47.840
It's actually affordable nowadays.
00:03:48.070 --> 00:03:50.868
You talk about Raspberry
Pi, the computer like ones?
00:03:50.868 --> 00:03:53.238
The, like, Raspberry Pi 4, 5.
00:03:53.788 --> 00:03:55.098
Aren't they always out of stock though?
00:03:55.098 --> 00:03:55.878
Isn't that an issue?
00:03:56.198 --> 00:03:58.148
Well, if you pick the
newest model, yeah, but this is
00:03:58.148 --> 00:04:01.918
sort of my placeholder for, it's not
just Raspberry Pi that makes SoCs.
00:04:01.918 --> 00:04:04.838
And in fact, they're usually the more
high end, low volume kind of thing.
00:04:05.088 --> 00:04:09.088
There's a lot of very cheap SoCs,
especially now that RISC-V is a thing.
00:04:09.288 --> 00:04:14.948
You can get like, surprisingly powerful
quad core 64 bit RISC-V processors
00:04:15.108 --> 00:04:16.838
for basically nothing at this point.
00:04:17.048 --> 00:04:19.758
So RISC-V is actually
getting adoption right now...
00:04:20.058 --> 00:04:22.833
They're selling a lot of
chips, so I think, especially coming out
00:04:22.833 --> 00:04:28.068
of China, you see a ton of Allwinner parts
that are getting used for set-top boxes.
00:04:28.147 --> 00:04:30.797
Depending on when you will listen to
this, it will either be different or not.
00:04:30.807 --> 00:04:34.527
But especially for things that are not
super cutting edge, like high end phones,
00:04:34.537 --> 00:04:39.697
things like set-top boxes or signage or
appliance type stuff where, yeah, it's
00:04:39.697 --> 00:04:44.703
like 40 percent slower than the equivalent
ARM device or something like that.
00:04:44.703 --> 00:04:48.583
But if it's a set-top box and it has the
DSP that you need, and you know, you've
00:04:48.583 --> 00:04:52.153
hit the minimum threshold, you might be
able to hit that minimum threshold for way
00:04:52.153 --> 00:04:54.403
cheaper, but it could be an ARM Cortex-A-
00:04:54.423 --> 00:04:58.673
So that is why
so many lower end TVs have UI
00:04:58.683 --> 00:05:01.543
that freaking lags so much.
00:05:02.108 --> 00:05:04.488
Minimum viable
product for the price point.
00:05:04.606 --> 00:05:07.206
They have
hardware H.264 decoding.
00:05:07.476 --> 00:05:11.076
So that works, but
everything else is so slow.
00:05:11.076 --> 00:05:13.223
It takes seconds to get to
where you want to watch.
00:05:13.223 --> 00:05:14.243
It's incredible.
00:05:14.285 --> 00:05:16.045
You can't just blame
that on RISC-V because that's
00:05:16.045 --> 00:05:17.205
been the case for 20 years.
00:05:17.940 --> 00:05:18.292
That's true...
00:05:18.292 --> 00:05:20.038
That's just one of
those, like we're hitting a price
00:05:20.038 --> 00:05:22.898
point with the compute that we
can afford to put in all of these
00:05:22.898 --> 00:05:24.398
set-top boxes and stuff like that.
00:05:24.533 --> 00:05:26.243
I mean, meanwhile,
people are complaining about Apple
00:05:26.243 --> 00:05:29.193
TVs and saying you should just
use the newer Mac Mini M4 instead.
00:05:30.473 --> 00:05:31.513
Because reasons?
00:05:31.573 --> 00:05:32.173
I don't know.
00:05:32.563 --> 00:05:34.797
I follow too many weird people on BlueSky
00:05:34.797 --> 00:05:34.857
yeah.
00:05:35.113 --> 00:05:37.623
But the thing is, embedded Linux
is great for this big stuff.
00:05:37.653 --> 00:05:40.873
If we're talking about video encoding
or decoding, if we're talking about
00:05:40.927 --> 00:05:44.445
crunching numbers, Linux is just easier
for doing these things, especially when
00:05:44.445 --> 00:05:46.355
you want to have an operating system.
00:05:46.575 --> 00:05:49.765
You want networking, you want file
systems, you want databases, you
00:05:49.765 --> 00:05:52.455
want to be able to do over the air
updates without having to really
00:05:52.455 --> 00:05:54.245
dangerously re flash the whole device.
00:05:54.415 --> 00:05:58.545
You just want, like apt-get update or
whatever BitBake tool you're using.
00:05:59.115 --> 00:06:01.705
And the other thing that's not
even a technical thing is hiring.
00:06:02.145 --> 00:06:05.375
You would probably be comfortable,
Amos, sitting down on an embedded
00:06:05.375 --> 00:06:08.385
Linux device because it's still
Linux and it feels very comfortable.
00:06:08.385 --> 00:06:11.605
So even if you don't necessarily
fully consider yourself an embedded
00:06:11.615 --> 00:06:15.705
developer, it's still a pool you're
much more comfortable in, especially
00:06:15.705 --> 00:06:18.205
if someone else has done the setup
and you're like, "Okay, now I can SSH
00:06:18.205 --> 00:06:20.501
in," and it's not markedly different.
00:06:20.501 --> 00:06:24.171
It might even be faster than a
super cheap VPS instance, which is
00:06:24.181 --> 00:06:27.935
also, you know, itty bitty in terms
of performance and capabilities.
00:06:28.208 --> 00:06:30.468
Yeah, I was thinking
if you have some sort of web UI,
00:06:30.488 --> 00:06:34.138
which a lot of connected gadgets
do: it would definitely be served
00:06:34.138 --> 00:06:35.638
from like the big, the big buddy.
00:06:35.922 --> 00:06:38.648
So these kinds of things
are all things that are super off
00:06:38.658 --> 00:06:41.828
the shelf and easy, both in terms
of like actually doing it and what
00:06:41.838 --> 00:06:43.228
exists out there for doing it.
00:06:43.228 --> 00:06:47.288
So these are all things that are
very easy to do on Linux, even if
00:06:47.288 --> 00:06:49.138
it's a relatively small Linux system.
00:06:49.353 --> 00:06:51.093
Actually,
isn't that what Tesla does?
00:06:51.233 --> 00:06:52.893
Like they run Ubuntu in their cars?
00:06:52.923 --> 00:06:55.108
A lot of gauge
clusters these days are running
00:06:55.138 --> 00:06:56.408
embedded Linux and things like that.
00:06:56.438 --> 00:07:00.318
They have some ways of like hyper visoring
away the safety critical stuff and the
00:07:00.318 --> 00:07:03.764
non safety critical stuff and you have-
exactly- that kind of stuff of 'not
00:07:03.784 --> 00:07:07.954
on the critical path' things because
it just makes iteration way easier.
00:07:07.954 --> 00:07:11.059
And if you can segment your product
in a way where a little bit of lag
00:07:11.059 --> 00:07:14.349
on the user interface isn't going to
cause problems with your brakes, then
00:07:14.349 --> 00:07:17.779
it's way easier to not do everything
in the super safety critical world.
00:07:17.779 --> 00:07:19.049
So exactly this kind of thing.
00:07:19.252 --> 00:07:21.409
And this is why I
like that you're my co host James!
00:07:21.409 --> 00:07:25.039
Because- because people get scared, Oh,
they hear about Kubernetes and the cars
00:07:25.039 --> 00:07:27.589
or whatever, and they go like, "No, I
don't want my brakes to stop working!"
00:07:27.589 --> 00:07:28.159
Because- whatever.
00:07:28.169 --> 00:07:31.714
Because they have their own experience of
like, someone tried to roll out Kubernetes
00:07:31.724 --> 00:07:35.554
in their internet startup, and it took
eight months and everyone was lost.
00:07:35.588 --> 00:07:37.798
But actually, yeah, it
is just for some stuff.
00:07:37.838 --> 00:07:39.828
And the rest of the stuff
can still keep working.
00:07:39.828 --> 00:07:42.368
It's very important, like isolating-
reducing the blast radius is
00:07:42.368 --> 00:07:44.413
also a thing in web services.
00:07:45.498 --> 00:07:47.168
Then you can do bare metal.
00:07:47.188 --> 00:07:49.128
And when I say bare metal, I
mean, like this is stuff that
00:07:49.128 --> 00:07:50.078
runs on microcontrollers.
00:07:50.118 --> 00:07:52.568
It could be on a real-time
operating system.
00:07:52.758 --> 00:07:55.716
But they're still fairly limited,
it's not the full capabilities
00:07:56.036 --> 00:07:56.886
of something like Linux.
00:07:56.896 --> 00:08:01.066
Or it could be a totally bare metal
program, or just an async executor,
00:08:01.076 --> 00:08:03.666
which is very common in like
embassy and things like that today.
00:08:04.006 --> 00:08:06.309
But we're using bare metal for
the little stuff, because it's
00:08:06.369 --> 00:08:07.649
good at a lot of things too.
00:08:08.009 --> 00:08:12.104
If you're building up custom hardware,
a microcontroller is usually pretty well
00:08:12.104 --> 00:08:16.114
self contained, they're cheap to put on
a circuit board, you don't have to put
00:08:16.184 --> 00:08:22.234
DRAM and an SSD and all of cooling and
all of these kind of things- usually you
00:08:22.234 --> 00:08:27.299
can drop a single chip, or a chip and its
storage on a circuit board, which means
00:08:27.299 --> 00:08:31.461
that board is very easy to design and
it also means it's very cheap to make.
00:08:31.461 --> 00:08:35.160
So if you go, "I just want something
that's a little custom," you don't
00:08:35.170 --> 00:08:37.922
have to build a motherboard, basically.
00:08:37.922 --> 00:08:41.076
You can just build like a little
accessory card fairly quickly.
00:08:41.163 --> 00:08:44.633
It lowers the threshold of how
skilled of an electrical designer
00:08:44.733 --> 00:08:46.983
you either need to hire or pay for.
00:08:47.033 --> 00:08:51.453
Right, just for the rest
of us: so DRAM would be actually chips
00:08:51.463 --> 00:08:53.343
that you would solder onto the board?
00:08:53.403 --> 00:08:54.033
It can be.
00:08:54.203 --> 00:08:57.803
the same way, like Apple solders,
their memory onto the board,
00:08:57.803 --> 00:09:02.263
you'll have like DDR2 or DDR3 is
fairly common for embedded devices.
00:09:02.263 --> 00:09:05.303
They're behind what your top
tier Android phone would be.
00:09:05.843 --> 00:09:09.318
But they're still more complicated
to have external memory.
00:09:09.578 --> 00:09:12.998
There are some chips that have
built in like, on the same chip as
00:09:12.998 --> 00:09:17.242
the CPU they have their memory, but
usually DRAM takes a lot of space.
00:09:17.362 --> 00:09:20.696
So that's usually why it's outside of
the main CPU even though you want it
00:09:20.696 --> 00:09:24.736
to be as close as possible but just
physically the transistors and everything
00:09:24.736 --> 00:09:26.686
that you need, you need space for that.
00:09:26.807 --> 00:09:27.077
Right.
00:09:27.157 --> 00:09:29.847
And the D stands for dynamic
because it's refreshed...
00:09:29.847 --> 00:09:30.227
Exactly.
00:09:30.252 --> 00:09:30.462
Yeah.
00:09:30.462 --> 00:09:33.622
So SRAM is usually what you
have in microcontrollers
00:09:33.772 --> 00:09:35.332
and it's also fairly large.
00:09:35.552 --> 00:09:39.853
But it's comparatively like electrically
and space-wise, expensive compared to
00:09:39.853 --> 00:09:46.303
DRAM, but main CPUs use SRAM as cache
because it's the fastest thing and
00:09:46.303 --> 00:09:47.983
you put it right next to your CPU.
00:09:48.263 --> 00:09:52.571
But usually when you want a big bulk
DRAM, which is your DDR memory- like
00:09:52.571 --> 00:09:56.141
you have on your desktop or your laptop
or whatever- that's a separate chip.
00:09:56.191 --> 00:10:00.001
On desktops it might be on a card that
you plug in just for convenience, but on
00:10:00.001 --> 00:10:03.741
your laptop or your phone it's going to
be soldered on right next to your chip.
00:10:03.973 --> 00:10:06.505
I remember upgrading
old laptops, they still had
00:10:06.505 --> 00:10:07.875
the, whatever it's called...
00:10:07.949 --> 00:10:08.699
you slot them in.
00:10:08.729 --> 00:10:10.913
I think ThinkPads might
still have it like the bigger ones,
00:10:10.913 --> 00:10:13.283
but not the like ultrabook style ones.
00:10:13.368 --> 00:10:15.778
I don't know, because
ThinkPads is no longer IBM, so
00:10:15.778 --> 00:10:16.908
I just assume they're crap now.
00:10:16.938 --> 00:10:17.278
I don't know.
00:10:17.983 --> 00:10:19.353
So the other
stuff that microcontrollers
00:10:19.353 --> 00:10:20.703
are great for is real-time.
00:10:20.723 --> 00:10:23.493
When you don't have an operating system
or you have a really purpose built
00:10:23.523 --> 00:10:26.123
operating system, you can make guarantees.
00:10:26.163 --> 00:10:30.943
Like: if this signal comes in, I will
respond to it and finish responding
00:10:30.943 --> 00:10:32.823
to it within 10 microseconds.
00:10:32.823 --> 00:10:36.777
Where that might be something that you
can start to get real-time or you can
00:10:36.807 --> 00:10:40.127
overcome the problem with power because
if your CPU is running fast enough,
00:10:40.157 --> 00:10:44.077
you know, probably you'll get to it
quickly, but it can be challenging
00:10:44.077 --> 00:10:46.237
to really like objectively guarantee.
00:10:46.237 --> 00:10:46.337
And-
00:10:46.337 --> 00:10:49.047
It's this kind of
thinking that leads to audio glitches
00:10:49.047 --> 00:10:51.850
in consumer hardware all the time
because they're designed with like
00:10:51.850 --> 00:10:55.100
it's probably gonna be fine and then
sometimes like *glitching sound*.
00:10:55.262 --> 00:10:58.472
So audio is usually called
soft real-time because no one's day's
00:10:58.472 --> 00:11:00.292
really messed up if it goes wrong.
00:11:00.312 --> 00:11:00.962
Except mine.
00:11:01.015 --> 00:11:03.127
Hard real-time is
usually, like, you stick your arm
00:11:03.127 --> 00:11:07.359
into a machine and the sensor detects
that and kills the motor within 10
00:11:07.359 --> 00:11:10.439
microseconds Like the SawStop, for
example, where there's that table saw.
00:11:10.479 --> 00:11:12.799
It's basically a cap touch
sensor, like your phone.
00:11:13.069 --> 00:11:16.089
And if you touch it within a
certain amount of time, it fires an
00:11:16.089 --> 00:11:18.059
explosive, which drops the blade down.
00:11:18.069 --> 00:11:20.959
Like you don't want that to lag
by like 30 milliseconds, because
00:11:20.959 --> 00:11:23.769
that might be the difference
between having your finger and not.
00:11:23.819 --> 00:11:25.089
So that one's pretty important.
00:11:25.539 --> 00:11:25.799
True.
00:11:26.219 --> 00:11:30.854
I wanted to mention that the
real-time patch set for the Linux
00:11:30.884 --> 00:11:33.344
kernel has been merged recently.
00:11:33.794 --> 00:11:36.364
They had a whole celebration
of that and I didn't know.
00:11:36.414 --> 00:11:39.404
So to give you an idea, right now
I'm using all Apple hardware because
00:11:39.454 --> 00:11:42.064
I unified everything and it makes
my life so much simpler and I'm
00:11:42.064 --> 00:11:43.804
able to get so much more work done.
00:11:44.029 --> 00:11:46.669
But back in the days, I
was an early Linux adopter.
00:11:46.730 --> 00:11:50.732
I had like Mandrake, I used Red Hat
before it was renamed to Fedora, I
00:11:50.732 --> 00:11:53.812
installed Slackware from floppies, I
don't know, I'm older than I look, people
00:11:53.852 --> 00:11:57.775
assume I'm a zoomer when they look at my
YouTube videos, but I'm 34, and I grew
00:11:57.775 --> 00:11:59.838
up with 10 years of lag on technology.
00:11:59.856 --> 00:12:00.746
Why was I saying all that?
00:12:00.956 --> 00:12:06.323
Because I remember building my own
version of the Ubuntu kernel with
00:12:06.343 --> 00:12:10.903
the real-time patch set to be able
to run a digital audio workstation
00:12:10.903 --> 00:12:13.537
and do audio work on Linux because
that was the only way to do it.
00:12:13.618 --> 00:12:17.388
It's crazy to me that it took,
I don't know, two decades?
00:12:17.798 --> 00:12:20.361
I don't know exactly how long to get
this merged into the main kernel.
00:12:20.361 --> 00:12:20.921
I have no idea.
00:12:20.921 --> 00:12:22.181
This is probably a very good reason.
00:12:22.766 --> 00:12:27.026
So my question to you James is, is it
still as relevant now that real-time
00:12:27.046 --> 00:12:30.166
Linux is in the main line kernel?
00:12:30.291 --> 00:12:34.347
Yes, because there's
still limitations on how granular
00:12:34.347 --> 00:12:36.637
you want your real-time guarantees
to be, because that's one of those
00:12:36.637 --> 00:12:39.517
things where real-time doesn't mean
faster, it means deterministic.
00:12:39.807 --> 00:12:42.867
So the scheduler might be way
less efficient, but you have the
00:12:42.867 --> 00:12:44.947
chance to swap tasks often enough.
00:12:44.997 --> 00:12:48.107
But there's still some limit
where a microcontroller is likely
00:12:48.107 --> 00:12:49.937
to be able to respond faster.
00:12:50.207 --> 00:12:53.027
Sometimes that's down
to the CPU architecture.
00:12:53.297 --> 00:12:56.917
For example, on Cortex-M there's a
bounded guaranteed amount of time when
00:12:56.917 --> 00:13:01.197
an interrupt occurs, how many cycles
it takes to when you're starting to do
00:13:01.197 --> 00:13:03.467
that versus on Linux, it might be...
00:13:03.719 --> 00:13:06.809
because Cortex-A processors as an
architecture don't have the same...
00:13:07.219 --> 00:13:08.992
it might not even be at
the operating system.
00:13:09.022 --> 00:13:11.672
It might be down to the, like,
architectural level of the CPU,
00:13:11.672 --> 00:13:15.562
because these CPUs are designed to
be real-time and if we're talking
00:13:15.562 --> 00:13:18.512
about like a millisecond, both of
them will be able to do that fine.
00:13:18.702 --> 00:13:21.212
A hundred microseconds, probably fine?
00:13:21.512 --> 00:13:25.222
But when we're talking like one
microsecond or 10 microseconds, you're
00:13:25.262 --> 00:13:29.202
really counting CPU cycles and you
want the actual electrical hardware
00:13:29.462 --> 00:13:32.782
to guarantee how long it's going
to take before you start executing
00:13:32.782 --> 00:13:34.842
those cycles that turn the motor off.
00:13:35.117 --> 00:13:37.959
Yeah, love the way
you phrased it, like 'real-time
00:13:37.979 --> 00:13:41.224
doesn't mean faster' because I'm
thinking about tokio as well.
00:13:41.324 --> 00:13:45.184
A bunch of people, you know, benchmark
reading a large file from synchronous
00:13:45.184 --> 00:13:47.996
Rust and then they do it with
tokio and like, "tokio is slower."
00:13:48.116 --> 00:13:51.076
Well, first of all, yes, for file system
IO, yes, because it's actually using
00:13:51.076 --> 00:13:52.676
blocking sys calls in a thread pool.
00:13:52.826 --> 00:13:56.496
But even if it wasn't, it would still
probably be slower because there's
00:13:56.516 --> 00:14:00.586
overhead involved in making sure you can
actually multiplex that on a single thread
00:14:00.596 --> 00:14:05.046
and that costs CPU cycles, but then you
can service multiple requests at once.
00:14:05.046 --> 00:14:07.832
It's better to have like all the
customers get their response in
00:14:07.882 --> 00:14:11.407
some time then be blocked on only
handling one customer, one response.
00:14:11.426 --> 00:14:11.656
Yeah.
00:14:11.706 --> 00:14:14.456
I know I've talked about this with Eliza,
but in the same way that you care about
00:14:14.506 --> 00:14:18.806
tail latency on a server where you say:
well, yeah, okay, 99 percent are handled.
00:14:18.856 --> 00:14:22.141
But what about the next nine or the
next nine, if that becomes ridiculous?
00:14:22.332 --> 00:14:25.922
In hard real-time, you want to say it
will never literally never, no matter
00:14:25.922 --> 00:14:29.672
what's going on- I don't care if I'm in
the middle of a database transaction- it
00:14:29.672 --> 00:14:35.030
will never take more than two and a half
microseconds from 'we noticed a touch'
00:14:35.180 --> 00:14:40.240
to 'we're firing the explosive' and the
explosive takes 20 microseconds to fire.
00:14:40.370 --> 00:14:44.500
Which means we know that from when your
finger touches that, it will never be
00:14:44.500 --> 00:14:48.217
more than like whatever I said, 12 and
a half microseconds before that blade is
00:14:48.217 --> 00:14:50.137
dropping and no longer touching your hand.
00:14:50.357 --> 00:14:52.485
It's one of those things where
engineering is all about 'good
00:14:52.485 --> 00:14:55.215
enough for what I'm doing.' But
sometimes you really care about that.
00:14:55.245 --> 00:14:59.875
And honestly- and this is what I'll get
into later- sometimes it could be possible
00:14:59.875 --> 00:15:02.735
with Linux, but it's not necessarily easy.
00:15:02.745 --> 00:15:03.765
We're on a microcontroller.
00:15:03.765 --> 00:15:05.855
It's like, "Okay, yeah, I analyzed it.
00:15:05.865 --> 00:15:06.595
It's fine."
00:15:06.825 --> 00:15:07.545
Versus on Linux.
00:15:07.545 --> 00:15:08.855
You're like, "I'm...
00:15:08.935 --> 00:15:11.335
I'm pretty sure it's
probably fine-" you know...
00:15:12.195 --> 00:15:14.075
And all of this stuff that I'm
recommending is not saying that
00:15:14.085 --> 00:15:15.525
you couldn't do it another way.
00:15:15.970 --> 00:15:19.690
But if you use the tools for
what they're good for, you're
00:15:19.690 --> 00:15:20.960
going to have a better time.
00:15:20.971 --> 00:15:24.277
So that's why you're saying
it's not for very large productions.
00:15:24.677 --> 00:15:25.527
You don't say productions...
00:15:25.557 --> 00:15:26.147
runs?
00:15:26.281 --> 00:15:26.622
Yeah.
00:15:26.622 --> 00:15:27.852
We'll get into that a little bit later.
00:15:28.092 --> 00:15:30.462
But then there's other things
like the amount of IO you have.
00:15:30.642 --> 00:15:35.198
And this can be just like, "Hey, I need
to talk to 600 switches cause I'm a big
00:15:35.228 --> 00:15:40.291
panel," Or I need to talk to 27 different
weird serial ports and things like that.
00:15:40.341 --> 00:15:43.711
These embedded Linux devices, especially
like the Raspberry Pi has popularized it.
00:15:43.761 --> 00:15:47.141
You see people plugging them straight into
sensors and buttons and stuff like that.
00:15:47.141 --> 00:15:47.971
And they do work.
00:15:48.221 --> 00:15:52.021
They just usually have a fairly limited
palette of things that they can do, or
00:15:52.021 --> 00:15:55.451
a limited number of pin counts where
it's challenging to get, you know,
00:15:55.451 --> 00:15:58.298
super high pin counts on one chip.
00:15:58.298 --> 00:16:00.858
And so you end up having
to have like expanders or
00:16:00.858 --> 00:16:02.188
adapters and things like that.
00:16:02.358 --> 00:16:04.118
And it ends up complicating your design.
00:16:04.128 --> 00:16:08.108
And you end up having to like, "Oh, well,
I can only scan these things so fast
00:16:08.138 --> 00:16:11.188
because I have to talk to 10 of them,"
and you can get into real trouble when
00:16:11.188 --> 00:16:13.758
it comes to doing what you'd like to do.
00:16:14.018 --> 00:16:16.198
Well, and then if you use
an adapter, aren't you using the buddy
00:16:16.198 --> 00:16:19.523
system anyway, because you have big
Raspberry Pi and then something over USB
00:16:19.533 --> 00:16:21.463
that has all the IO pins that you need?
00:16:21.633 --> 00:16:24.737
There's something
to be said about paying for less
00:16:24.737 --> 00:16:26.467
software, if that makes sense.
00:16:26.477 --> 00:16:28.915
In that hardware it acts
the way it acts and you don't
00:16:28.915 --> 00:16:29.845
have to think about software.
00:16:29.845 --> 00:16:31.305
So there is something
to be said about that.
00:16:31.322 --> 00:16:34.087
By the time you've added a couple
adapters and things like that,
00:16:34.097 --> 00:16:37.257
it ends up costing more than a
microcontroller in a lot of cases.
00:16:38.107 --> 00:16:39.517
And the other thing is low power.
00:16:39.627 --> 00:16:42.987
Your phone is relatively low power,
but you are charging it every day.
00:16:43.351 --> 00:16:46.891
A lot of these, especially the
low cost embedded Linux SoCs,
00:16:46.921 --> 00:16:50.761
don't have nearly as good power
management as a nice phone would.
00:16:51.111 --> 00:16:53.901
And if you're trying to deploy
something that's going to last
00:16:53.911 --> 00:16:57.781
for a year with no solar or no
landline or anything like that.
00:16:57.974 --> 00:17:01.709
If you have like a Raspberry Pi, unless
you have a car battery it's going
00:17:01.709 --> 00:17:06.759
to drain your battery in a day or a
week or maybe a month, but you want
00:17:06.759 --> 00:17:08.759
something that's going to last years.
00:17:09.039 --> 00:17:11.559
What you really need to be able
to do is not just sleep, but
00:17:11.559 --> 00:17:13.519
shut down the power hungry...
00:17:13.959 --> 00:17:15.752
it's one of those things where
it's not even the CPU like...
00:17:15.802 --> 00:17:19.762
One of the differences between
SRAM- static RAM- and DRAM, which is
00:17:19.762 --> 00:17:23.292
dynamic RAM is you have to refresh
dynamic RAM constantly which takes
00:17:23.292 --> 00:17:27.282
electricity to basically recharge
the capacitors in the memory so that
00:17:27.292 --> 00:17:28.872
your memory doesn't get corrupted.
00:17:29.132 --> 00:17:32.492
And that takes energy, and so if you
can't shut off your memory, if you
00:17:32.492 --> 00:17:35.792
can't shut off your CPU, you just
have this sort of baseline power
00:17:35.792 --> 00:17:39.153
usage that most, not all of them,
most microcontrollers will be able to
00:17:39.163 --> 00:17:41.733
go down to like micro or nano amps.
00:17:42.173 --> 00:17:45.743
Whereas a lot of these embedded Linux
SoCs are way, like, orders of magnitude
00:17:45.743 --> 00:17:49.743
higher in milliamps, even when they're
in sort of like low power mode.
00:17:50.540 --> 00:17:53.520
And I'm going to tell you to use Rust
for both because I am who I am, but
00:17:53.660 --> 00:17:56.010
it allows you to do a lot of very
cool things because we don't have
00:17:56.010 --> 00:17:59.672
to pretend that our microcontrollers
are archaic little systems anymore.
00:17:59.982 --> 00:18:03.242
They are 32 bit processors, and
that's not the same as the 64 bit
00:18:03.242 --> 00:18:06.782
processor you're using, but it's
much closer than the little 8 bit
00:18:06.782 --> 00:18:08.672
microcontrollers with weird architectures,
00:18:08.912 --> 00:18:12.912
which means you can share code,
tools, workflows, and even developers.
00:18:12.932 --> 00:18:16.712
It gets to the point where if I sat you
down and showed you an embassy project
00:18:16.712 --> 00:18:20.332
that's still using Rust async, there
might be some learning curve of like, "Oh,
00:18:20.332 --> 00:18:21.672
I need to learn some different habits.
00:18:21.672 --> 00:18:23.132
Cause not everything's great here."
00:18:23.537 --> 00:18:25.047
But the big picture would make sense.
00:18:25.581 --> 00:18:27.137
You'd go, "Yeah, I'm going
to put a timeout on it.
00:18:27.137 --> 00:18:27.777
I'm going to await."
00:18:28.017 --> 00:18:30.137
It's all going to make
sense fairly quickly.
00:18:30.177 --> 00:18:33.292
And that's super valuable because
a lot of embedded teams that I work
00:18:33.292 --> 00:18:37.343
with, there's usually like five or
10 backend and front end people.
00:18:37.603 --> 00:18:41.473
There's like one embedded developer
and one hardware person, and that's
00:18:41.473 --> 00:18:44.513
like the whole team, which means if
you ever need peer review or getting
00:18:44.513 --> 00:18:47.393
people to work together, if you're
not sharing tools and language, it
00:18:47.393 --> 00:18:51.963
becomes very challenging to ever get
good code review or just a good second
00:18:51.963 --> 00:18:53.753
set of eyes on what you're building.
00:18:54.223 --> 00:18:59.253
Yeah, and I mean, this is
context for the entire first season of
00:18:59.263 --> 00:19:03.673
SDR, because it's like: why do you care
about serialization/deserialization, why
00:19:03.673 --> 00:19:05.293
do you care about it running on embedded?
00:19:05.513 --> 00:19:05.833
That's true!
00:19:06.263 --> 00:19:08.513
The nice thing of like, if
it runs on embedded, it will definitely
00:19:08.513 --> 00:19:14.083
for sure work on desktop or like a
Raspberry Pi 4 or 5 Cortex-A, I guess.
00:19:14.159 --> 00:19:14.789
Yeah, exactly.
00:19:14.789 --> 00:19:17.679
And that's what I'm going to recommend is
you tie it together with a library like
00:19:17.679 --> 00:19:22.049
Postcard RPC or a tool that I'm working
on called Poststation, which is sort of
00:19:22.049 --> 00:19:24.259
like a reverse proxy for embedded devices.
00:19:24.259 --> 00:19:27.309
But the thing is it's a way of
communicating where we treat
00:19:27.309 --> 00:19:31.054
them like a real set of peers
that communicate with each other.
00:19:31.054 --> 00:19:34.094
In the same way that we treat a
browser and a server as peers,
00:19:34.284 --> 00:19:37.482
like the client and the server,
they are equivalent to each other.
00:19:37.482 --> 00:19:40.242
We're not just treating them
like totally different machines.
00:19:40.242 --> 00:19:41.272
They are both computers.
00:19:41.602 --> 00:19:44.922
Using something like Postcard RPC also
lets you use whatever communication
00:19:44.922 --> 00:19:46.742
link you have between these devices.
00:19:46.822 --> 00:19:51.697
Like the Raspberry Pi, it's got a
USB port, it's got UART or a serial
00:19:51.697 --> 00:19:55.627
port, it's got SPI or I2C, all these
different kinds of serial ports that
00:19:55.627 --> 00:19:59.571
are better or worse at certain things,
but you can use whatever links you have,
00:19:59.631 --> 00:20:03.581
because Postcard RPC is very simple,
and Poststation is very flexible in
00:20:03.581 --> 00:20:07.391
what it will proxy to, instead of just
over the network, you can throw it over
00:20:07.676 --> 00:20:12.136
essentially any interface that you can
make frames on, which is every serial
00:20:12.136 --> 00:20:13.986
interface, every one that I have here.
00:20:14.256 --> 00:20:18.516
So it becomes very easy to proxy
over whatever link you have, even
00:20:18.516 --> 00:20:19.926
if it doesn't look like ethernet.
00:20:20.199 --> 00:20:22.719
And really my goal here is to
have the best of both worlds.
00:20:22.739 --> 00:20:25.699
You don't have to sort of pick and
choose and be awkward on some things and
00:20:25.699 --> 00:20:29.889
just lean into it, design your system
in a way where you go: look, all the
00:20:29.889 --> 00:20:32.859
real-time stuff- the microcontroller
is going to be in charge of that.
00:20:33.089 --> 00:20:36.979
And I'm going to make it sort of the
authoritative one on anything safety
00:20:36.979 --> 00:20:40.979
or real-time or signal processing or
IO expansion and things like that.
00:20:41.349 --> 00:20:45.924
And for all the big number crunching
and networking and deciding user
00:20:45.924 --> 00:20:48.654
interface things, I'm going to put
that on an embedded Linux and it's
00:20:48.654 --> 00:20:49.934
going to be in charge of that.
00:20:50.204 --> 00:20:52.854
And when you think of your system, you're
going to go, who's in charge of what?
00:20:52.916 --> 00:20:56.254
In a network cluster, you're like: that's
my database server and that's my front
00:20:56.254 --> 00:20:57.794
end server and that's my caching server.
00:20:57.974 --> 00:21:01.094
They're specialized to different
tasks, but I still treat them as
00:21:01.104 --> 00:21:04.894
peers that are working together
on like the bigger common goal.
00:21:04.952 --> 00:21:07.729
I keep thinking of how
you said you can use Rust on the big
00:21:07.729 --> 00:21:12.699
one and on the little one, and the
equivalent in my head I have is using
00:21:12.699 --> 00:21:15.089
JavaScript in the browser and on server.
00:21:15.579 --> 00:21:17.708
And it's much more awkward to do that.
00:21:17.723 --> 00:21:18.433
What's the word for that?
00:21:18.463 --> 00:21:19.313
Isomorphic.
00:21:19.378 --> 00:21:20.468
Yeah, I think so.
00:21:20.648 --> 00:21:23.793
But with Rust, it's so interesting because
I've been mostly in touch with people
00:21:23.793 --> 00:21:28.313
who write Rust not on embedded and they
look at async Rust and they're like, "Ah,
00:21:28.313 --> 00:21:31.303
it's, it's too complicated," and they
don't understand the design constraints.
00:21:31.303 --> 00:21:36.348
But then if you actually are working
on both sides, you see why it was
00:21:36.348 --> 00:21:40.338
designed that way, because then you
just get to write async Rust code on
00:21:40.368 --> 00:21:43.228
those tiny, well, tiny ish devices.
00:21:43.543 --> 00:21:44.343
Yeah, for sure.
00:21:44.438 --> 00:21:45.223
And it's the same language.
00:21:45.227 --> 00:21:47.967
It's something I really appreciate with
Rust, is that you have so much range.
00:21:48.027 --> 00:21:52.597
You can build abstractions, you're
not stuck to the ground, like with
00:21:52.597 --> 00:21:53.637
C, where it's like, Nowhere safe.
00:21:53.742 --> 00:21:55.722
Truck is all you get and that's it.
00:21:55.992 --> 00:21:57.312
You don't get anything else.
00:21:57.312 --> 00:22:00.052
Or C++ where I'm not sure
what's happening with rustc.
00:22:00.072 --> 00:22:03.162
You get like this whole range and
you can go down into the library.
00:22:03.282 --> 00:22:04.842
And, yeah, there's some unsafe code.
00:22:04.842 --> 00:22:05.142
Okay.
00:22:05.142 --> 00:22:08.652
Unsafe is- it's- it needs to be
there, but it's all the same language.
00:22:08.652 --> 00:22:12.132
You can dig, you can understand
there's no VM boundary, like high
00:22:12.132 --> 00:22:13.662
level code versus low level code.
00:22:13.842 --> 00:22:14.922
It's all the same language.
00:22:15.252 --> 00:22:15.482
Yeah.
00:22:15.482 --> 00:22:18.452
And I think the isomorphic JavaScript
speaks to that of like, there's
00:22:18.452 --> 00:22:23.272
a tangible benefit when you have
multiple parts in your system, having
00:22:23.282 --> 00:22:25.712
common tooling and common language
and things like that, because
00:22:25.712 --> 00:22:27.772
there's developer benefits to that.
00:22:27.827 --> 00:22:30.107
Because you're able to jump
between the two of them.
00:22:30.117 --> 00:22:31.957
This is something that I recommend
to people where they're like,
00:22:31.957 --> 00:22:33.317
"Rust is really good at FFI!"
00:22:33.627 --> 00:22:37.497
and I go, "Yes, it is, if you have
to," but if you can, sometimes like,
00:22:37.527 --> 00:22:41.457
there's a real benefit to only having
one language because anytime you jump
00:22:41.487 --> 00:22:45.057
that gap, you need to be an expert
in the semantics of both of them.
00:22:45.367 --> 00:22:49.637
That's possible, but it's way easier to
get someone to be really understanding
00:22:49.637 --> 00:22:53.177
of Rust and where things start and stop
there, or even just someone to be familiar
00:22:53.177 --> 00:22:56.907
and C where things start and stop there,
whereas whenever you transfer over
00:22:56.907 --> 00:22:58.897
that boundary, you have to think both.
00:22:59.002 --> 00:23:03.215
And that's true over the network as well,
because if you have some Go server that
00:23:03.215 --> 00:23:07.095
does verification of data in one way, but
JavaScript or Node does it in a different
00:23:07.095 --> 00:23:11.435
way, you get something that mostly works
until that one day when they don't agree.
00:23:11.725 --> 00:23:15.225
And then you need someone who knows
enough about both of them to really
00:23:15.225 --> 00:23:18.365
figure out where that just like
tiny little slip in the seam is.
00:23:18.695 --> 00:23:19.055
Yeah.
00:23:19.108 --> 00:23:22.928
Again, that's also a place where Rust
shines, is that you- specifically you
00:23:22.928 --> 00:23:26.048
in Postcard RPC, are leveraging the
type system a lot, so there's just
00:23:26.048 --> 00:23:27.658
a lot of mistakes you cannot make.
00:23:27.738 --> 00:23:31.518
When I try to teach Rust, I don't really
try to advocate for it, I just try to
00:23:31.518 --> 00:23:34.468
teach it, because I learned it, I thought
it was neat, and so I've been teaching
00:23:34.468 --> 00:23:36.398
it, and people are advocating for it.
00:23:36.478 --> 00:23:39.988
What you're taking away from them is
the pride of being in very dangerous
00:23:40.028 --> 00:23:43.578
waters and not dying, like having
all those mistakes you can make,
00:23:43.728 --> 00:23:47.310
but being smart and clever enough
that you don't actually make them.
00:23:47.740 --> 00:23:50.300
And then switching to something like
Rust is like: okay, occasionally
00:23:50.310 --> 00:23:52.650
you're going to have the compiler
be mad at you and it's going to
00:23:52.660 --> 00:23:53.680
take some time to understand why.
00:23:53.710 --> 00:23:56.325
But most of the time, there's
so many mistakes you can never
00:23:56.325 --> 00:23:58.215
make and it's so relaxing.
00:23:58.225 --> 00:24:01.651
You just kind of: if it compiles it
probably works, if you design your
00:24:01.651 --> 00:24:03.141
types correctly which is a big if.
00:24:03.161 --> 00:24:05.091
But you know you get a
sense for it after a while.
00:24:05.261 --> 00:24:08.291
That in the embedded world, where
you don't have all the niceties of
00:24:08.291 --> 00:24:10.953
the operating system which is going
to have your back and like just it's
00:24:10.953 --> 00:24:13.679
no big deal to blow up the stacks no
big deal to do a lot of things, you
00:24:13.689 --> 00:24:14.839
don't even have to free your memory.
00:24:15.159 --> 00:24:17.299
If you exit the process you're
just going to reclaim everything.
00:24:17.299 --> 00:24:18.509
Virtual memory is really nice.
00:24:18.539 --> 00:24:19.782
I don't know if you get that on MCUs.
00:24:19.942 --> 00:24:20.552
No, nope.
00:24:20.576 --> 00:24:22.446
We have no MMU, so
there's no virtual memory.
00:24:22.866 --> 00:24:23.266
Exactly.
00:24:23.316 --> 00:24:26.546
But my threshold for this
is usually around one to 10,000 units.
00:24:26.639 --> 00:24:30.199
This number seems higher than most
people, where I'm like: below that,
00:24:30.226 --> 00:24:34.226
what you do in terms of the hardware
you design surprisingly doesn't matter.
00:24:34.446 --> 00:24:37.286
That's probably more like one
to ten thousand units per year.
00:24:37.536 --> 00:24:38.996
If you're below that threshold...
00:24:39.176 --> 00:24:43.472
you mentioned Kubernetes- embedded
sort of has the same problem as some
00:24:43.472 --> 00:24:47.162
people complain about Kubernetes- you're
designing for a scale that you don't have.
00:24:47.362 --> 00:24:49.712
It's not that it's a bad
solution, but it's a solution
00:24:49.712 --> 00:24:51.672
to a problem you don't have yet.
00:24:51.702 --> 00:24:56.402
And so when I see people like really
optimizing the cost of per unit If
00:24:56.402 --> 00:25:00.222
you're not making 10,000 units, yeah,
don't do something super egregious,
00:25:00.232 --> 00:25:01.852
but I'll get to this in a little bit.
00:25:01.852 --> 00:25:02.122
Like...
00:25:02.462 --> 00:25:06.667
the buddy system is probably
going to be cheaper in the big
00:25:06.667 --> 00:25:08.083
picture for what you're doing.
00:25:08.083 --> 00:25:10.443
I see what you're doing
with the Kubernetes comparison and
00:25:10.483 --> 00:25:15.503
I agree with your point, but I think
Kubernetes makes sense for a lot more
00:25:15.503 --> 00:25:17.033
people just because it's standardized.
00:25:17.033 --> 00:25:18.803
I think we'll have
to save that chat for another day.
00:25:18.928 --> 00:25:20.498
It breaks down at this point!
00:25:21.084 --> 00:25:23.890
Because you brought up Kubernetes:
there is a nice article that I
00:25:23.890 --> 00:25:28.060
read recently called, "Dear friend,
I'm afraid you've just written a
00:25:28.060 --> 00:25:29.210
Kubernetes," or something like that?
00:25:29.230 --> 00:25:32.214
We'll have it in the show notes, so that
we don't have to discuss it right now.
00:25:32.284 --> 00:25:32.454
Sure.
00:25:33.292 --> 00:25:37.622
And so my pitch is if you're under this
sort of like one to 10,000 unit threshold,
00:25:37.882 --> 00:25:39.752
the buddy system is probably cheaper.
00:25:39.752 --> 00:25:42.342
Even though I'm telling you to use
two chips, you know, a slightly
00:25:42.342 --> 00:25:45.702
more complex design, because you've
got two systems that you're writing
00:25:45.702 --> 00:25:47.142
software for and things like that.
00:25:47.372 --> 00:25:49.592
It will likely end up
being cheaper because
00:25:49.782 --> 00:25:56.227
development time is expensive, and
being able to do two easy things is
00:25:56.237 --> 00:26:00.401
often much faster as a developer for
trying to pack both of those easy
00:26:00.401 --> 00:26:04.161
things into one thing, which makes it
end up being a very complex problem.
00:26:04.171 --> 00:26:07.721
It's not that you can't, it's that: you
might have to write kernel patches, you
00:26:07.721 --> 00:26:10.311
might have to figure out a lot of external
hardware that you need as adapters,
00:26:10.311 --> 00:26:13.811
you might have to really pack things
in and optimize because you're like, "I
00:26:13.811 --> 00:26:15.891
really have to hit that timing thing."
00:26:16.151 --> 00:26:20.191
If you just use two systems, then
you can let them both play on their
00:26:20.191 --> 00:26:24.241
strengths and you can even pick
versions of each of those two buddies
00:26:24.431 --> 00:26:26.111
that are really specialized for that.
00:26:26.111 --> 00:26:31.341
And usually, things that are specialized
for one feature is much cheaper than if
00:26:31.341 --> 00:26:35.491
you need one chip that is specialized
on three different things because
00:26:35.491 --> 00:26:36.921
you need to do that all in one chip.
00:26:37.091 --> 00:26:40.051
So either the per unit cost or
just the amount of time you spend
00:26:40.051 --> 00:26:43.701
on development because developers
are very expensive per hour.
00:26:43.747 --> 00:26:44.547
I agree.
00:26:44.615 --> 00:26:47.405
And yeah, like I said, doing
microcontroller things on Linux sucks.
00:26:47.445 --> 00:26:48.155
You can do it.
00:26:48.155 --> 00:26:49.335
There's the real-time patches.
00:26:49.565 --> 00:26:51.205
Down to some threshold it will be fine.
00:26:51.225 --> 00:26:53.315
If you're building something
that needs one serial port,
00:26:53.565 --> 00:26:54.765
ignore me, you'll be fine.
00:26:54.965 --> 00:26:58.365
As soon as that starts chafing, as
soon as it starts being uncomfortable,
00:26:58.565 --> 00:27:01.925
that should be your sign of like,
instead of trying harder and committing
00:27:02.028 --> 00:27:03.558
harder and harder to optimize things.
00:27:03.608 --> 00:27:06.278
Just give up, use the buddy
system because you know, for
00:27:06.278 --> 00:27:07.488
one or two things, it's fine.
00:27:07.668 --> 00:27:09.068
When you get to five, it's uncomfortable.
00:27:09.068 --> 00:27:10.518
When you get to 10, it's egregious.
00:27:10.568 --> 00:27:12.998
It's one of those things where you don't
have to necessarily do it on day one,
00:27:13.058 --> 00:27:14.278
unless you think you might need it.
00:27:14.518 --> 00:27:18.565
But just trying to make the big system
do little system things is painful.
00:27:18.995 --> 00:27:20.915
I'm imagining the Floppotron?
00:27:21.885 --> 00:27:22.095
Okay.
00:27:22.095 --> 00:27:24.865
The thing that plays music
with like a hundred floppy drives or
00:27:24.865 --> 00:27:27.145
whatever, and it's taking MIDI files?
00:27:27.295 --> 00:27:29.305
That's something you would
need an MCU for, right?
00:27:29.333 --> 00:27:30.429
To drive all of those.
00:27:30.480 --> 00:27:31.665
Or like a lot of expanders.
00:27:31.777 --> 00:27:34.987
You need stepper motor controller
drivers times a hundred.
00:27:35.045 --> 00:27:36.785
The timing
has to be very precise.
00:27:36.785 --> 00:27:37.175
Yeah.
00:27:37.175 --> 00:27:37.176
Yeah.
00:27:37.220 --> 00:27:37.480
Yeah.
00:27:37.480 --> 00:27:40.199
You won't find any embedded Linux
system that can run a hundred of those.
00:27:40.199 --> 00:27:43.979
So it needs to be a special driver IC,
or you need some way of managing that.
00:27:43.979 --> 00:27:46.435
And the same way doing Linux
things on a microcontroller sucks.
00:27:46.455 --> 00:27:49.825
Like there are libraries that are
nice these days where you can have a
00:27:49.825 --> 00:27:52.145
TCP/IP stack on an embedded device.
00:27:52.145 --> 00:27:56.315
You can have HTTPS on an embedded
device and it's async and it's
00:27:56.315 --> 00:27:59.455
nice, but it still means you need
a really big microcontroller.
00:27:59.675 --> 00:28:01.815
It still means that you're
running a network stack.
00:28:01.865 --> 00:28:04.205
What happens when you want to
update your certificate chain?
00:28:04.265 --> 00:28:06.510
On Linux it's just an apt get update.
00:28:06.550 --> 00:28:09.490
And on the microcontroller, you go:
we need to be really careful because
00:28:09.490 --> 00:28:11.130
we want to do our update over SSL.
00:28:11.150 --> 00:28:14.410
If we ever send a corrupted
set of SSL certificates or TLS
00:28:14.410 --> 00:28:17.420
certificates- oops, now our
firmware updates don't work anymore!
00:28:17.590 --> 00:28:20.220
I was thinking Linux
people get uppity when you do static
00:28:20.220 --> 00:28:21.770
linking, like bring your own libraries.
00:28:21.770 --> 00:28:25.470
Cause they're like, "No, I want you to
use my distribution's version of LibSSL!"
00:28:25.470 --> 00:28:27.080
So that they can patch vulnerabilities.
00:28:27.080 --> 00:28:31.000
But now imagine your entire
TCP stack is baked into-
00:28:31.010 --> 00:28:32.160
Baked into the firmware.
00:28:32.160 --> 00:28:33.140
Yep, exactly.
00:28:33.220 --> 00:28:33.490
Yeah.
00:28:33.490 --> 00:28:34.990
A firmware is just one application.
00:28:35.040 --> 00:28:37.270
Like it is an application
that does operating system
00:28:37.270 --> 00:28:38.960
jobs is all a firmware is.
00:28:39.140 --> 00:28:41.760
And the thing is, off the shelf
boards are cheap, like I said.
00:28:41.850 --> 00:28:48.700
It's way easier to buy a high end off
the shelf board than to design a custom
00:28:48.819 --> 00:28:53.599
optimized low cost board, because just
economies of scale are what they are, and
00:28:53.599 --> 00:28:57.789
it's cheaper, in a lot of cases, to get a
Raspberry Pi than a correctly sized thing.
00:28:57.789 --> 00:28:59.299
And this is one of those
things that people on the
00:28:59.299 --> 00:29:00.489
internet lose their mind over.
00:29:00.669 --> 00:29:01.249
They're like, "Ah!
00:29:01.249 --> 00:29:03.519
Why are you using a whole
Raspberry Pi to do that?!"
00:29:03.519 --> 00:29:07.849
And it's like, because my time is
expensive and that costs 40 bucks.
00:29:07.879 --> 00:29:13.049
And by the time that I have to buy three
cheap things and wire them together, even
00:29:13.049 --> 00:29:14.699
just in parts, I've probably paid more.
00:29:14.699 --> 00:29:18.779
But then when I factored my time,
we're way, way over these things.
00:29:18.839 --> 00:29:21.629
Do your hobbies as you like, if
you like retrocomputing, yes.
00:29:21.819 --> 00:29:26.012
But you should not approach a company
project as this, unless you have to.
00:29:26.694 --> 00:29:29.884
That's the other thing: when you do want
to design something of your own, the
00:29:29.884 --> 00:29:31.744
simpler it can be, the cheaper it is.
00:29:31.774 --> 00:29:35.616
If you can have someone throw together
a two layer circuit board, or use off
00:29:35.616 --> 00:29:39.066
the shelf parts that have fairly big
pins, and you can throw them in on a
00:29:39.066 --> 00:29:42.562
board, I can send that design over to
JLC, and in two weeks I'll have the
00:29:42.562 --> 00:29:44.312
board for, like, five bucks a board.
00:29:44.522 --> 00:29:47.072
Or I can get it made locally for
a bit more, but I'll have it in,
00:29:47.072 --> 00:29:50.937
like, two days, because I don't
need an eight layer graphics card
00:29:50.947 --> 00:29:52.927
design sort of thing going on.
00:29:52.997 --> 00:29:55.467
So even when you do design
something custom, cause you want
00:29:55.527 --> 00:29:58.977
it to be smaller or fit whatever
you'd like, that's cheaper too.
00:29:59.007 --> 00:30:02.877
As long as you can have two simple things
instead of one really complex thing.
00:30:02.877 --> 00:30:07.607
Cause if you need the 800 pin count
CPU to get all of your IO, that's
00:30:07.607 --> 00:30:08.817
going to be an expensive board.
00:30:08.847 --> 00:30:10.257
And every time you get it wrong...
00:30:10.717 --> 00:30:14.659
Circuit board design is like when you
hit compile, it takes two weeks to figure
00:30:14.659 --> 00:30:18.749
out if the compile worked and also like
a couple thousand dollars every time
00:30:18.749 --> 00:30:21.399
you compile because you have to get
these board made, you have to bring
00:30:21.399 --> 00:30:24.849
them in, someone either has to do the
assembly or you have to do the assembly
00:30:25.049 --> 00:30:27.259
and then you might find you shorted
two pins together and the whole power
00:30:27.259 --> 00:30:30.279
supply fries and you go, "Well, I guess
we're going to wait two more weeks."
00:30:30.689 --> 00:30:33.439
Versus a simple design, you go:
yeah, we'll just slap it on.
00:30:33.439 --> 00:30:35.532
It's probably going to work
more likely in the first thing.
00:30:35.532 --> 00:30:39.659
And even if it does not work, it's
faster to cycle and cheaper to cycle.
00:30:39.743 --> 00:30:43.573
If you go even deeper in
designing your own hardware and you're
00:30:43.573 --> 00:30:48.703
writing VHDL or whatever the other
one is, you can simulate those kind
00:30:48.703 --> 00:30:50.813
of, but synthesizing takes forever.
00:30:50.823 --> 00:30:53.493
The whole tooling around that is horrible.
00:30:53.773 --> 00:30:56.273
I've done a little bit of it at
university, so it dates back.
00:30:56.313 --> 00:30:58.910
I hope things are improved,
but I'm not holding my breath.
00:30:59.360 --> 00:31:02.142
FPGAs are a whole
conversation for another day.
00:31:02.463 --> 00:31:05.283
And that's the other thing about Rust
is you get a lot of stuff off the shelf.
00:31:05.313 --> 00:31:08.683
Like I said, I talked about postcard,
but we also get libraries for stuff.
00:31:08.723 --> 00:31:12.233
And this is both on the embedded Linux
side and the microcontroller side.
00:31:12.473 --> 00:31:16.623
Sometimes the same library, like
in Postcard, but if not, when
00:31:16.623 --> 00:31:18.723
you're doing your microcontroller
stuff, we have embassy.
00:31:18.883 --> 00:31:21.303
You don't have to build your
own OS or scheduler or whatever.
00:31:21.573 --> 00:31:23.333
You know, there's drivers
for all of these things.
00:31:23.333 --> 00:31:24.463
They're fairly portable.
00:31:24.633 --> 00:31:27.713
We have the whole trait system, which
allows you to do portable drivers and
00:31:27.713 --> 00:31:31.973
stuff, even to the point where you
could port it from the Raspberry Pi to
00:31:31.973 --> 00:31:36.343
the Raspberry Pi Pico and use the exact
same display driver on both of them.
00:31:36.643 --> 00:31:41.198
So like being able to use pieces off
the shelf, either public ones or ones
00:31:41.198 --> 00:31:43.998
that you've built at your company where
you go: well, sometimes we just have
00:31:43.998 --> 00:31:47.468
the Raspberry Pi, but sometimes we have
the Raspberry Pi with a buddy system.
00:31:47.898 --> 00:31:51.328
You just write the driver once and
use it in house and reuse things.
00:31:51.648 --> 00:31:53.388
Yeah, I guess a
buddy system with a language
00:31:53.388 --> 00:31:55.798
like C, an ecosystem like C-
00:31:55.867 --> 00:31:56.378
It's hard.
00:31:56.418 --> 00:31:58.298
-without a package
manager or a standardized thing,
00:31:58.298 --> 00:31:59.438
yeah, it would be a lot more work.
00:31:59.569 --> 00:32:00.528
Yeah, it's possible.
00:32:00.528 --> 00:32:01.228
And people have done it.
00:32:01.228 --> 00:32:02.328
It's just, it's challenging.
00:32:02.368 --> 00:32:04.738
Like the language doesn't fight
you, but it doesn't help you.
00:32:04.988 --> 00:32:05.768
It can be done.
00:32:05.989 --> 00:32:09.536
Yeah, you have to enforce
convention inside of the project...
00:32:09.688 --> 00:32:11.056
And be really careful, yeah.
00:32:11.916 --> 00:32:13.796
So Postcard RPC is an
off the shelf comm stack.
00:32:13.806 --> 00:32:18.076
You just plug stuff together, whether
it's over USB or serial or whatever.
00:32:18.296 --> 00:32:20.936
And you just have client and
server libraries for both of them.
00:32:21.436 --> 00:32:24.076
And Poststation, the tool that I'm
building now also has a bunch of
00:32:24.456 --> 00:32:26.446
tools and APIs and SDKs for it.
00:32:26.466 --> 00:32:29.980
So when you plug your device in,
you get a GUI for it, or you get
00:32:29.980 --> 00:32:31.541
history data and things like that.
00:32:31.591 --> 00:32:35.281
Whether you're plugging it into your
laptop or the actual embedded SoC, you
00:32:35.281 --> 00:32:38.751
can just mess around, because there's
different stages to designing hardware.
00:32:39.031 --> 00:32:42.041
First, you design the circuit boards
and usually you're sitting there making
00:32:42.041 --> 00:32:44.841
sure all the motors spin and you can
read raw sensor data where you're not
00:32:44.851 --> 00:32:47.291
really doing business logic stuff yet.
00:32:47.301 --> 00:32:50.961
You're just poking it slowly
to make sure it does all work.
00:32:51.371 --> 00:32:53.431
And then you sort of iterate from
there where you go: okay, now I'm
00:32:53.431 --> 00:32:57.451
going to make it automatically do
this at 100 hertz or whatever, and
00:32:57.461 --> 00:32:59.081
then aggregate or filter the data.
00:32:59.091 --> 00:33:01.321
And then I get to sort
of the shape that I want.
00:33:01.904 --> 00:33:06.247
Being able to not have to build custom
stuff, not having to build jigs yourself
00:33:06.267 --> 00:33:08.677
where you can just go: okay, I'm
just gonna, can I poke, poke, poke?
00:33:08.757 --> 00:33:09.067
Cool.
00:33:09.107 --> 00:33:09.447
Done.
00:33:09.527 --> 00:33:10.007
Move on.
00:33:10.177 --> 00:33:13.454
And then I can hand that tool to the
factory people to make sure that whenever
00:33:13.454 --> 00:33:18.145
we assemble one, that it wasn't miswired
or not calibrated or whatever you need.
00:33:18.889 --> 00:33:23.140
We've seen you do that a bit
on your YouTube live streaming channel?
00:33:23.150 --> 00:33:23.430
You've been
00:33:23.665 --> 00:33:24.115
-
Yeah.
00:33:24.410 --> 00:33:26.923
-iterating
on the real life version...
00:33:27.003 --> 00:33:27.593
Of our logo?
00:33:27.593 --> 00:33:27.963
Yeah.
00:33:27.985 --> 00:33:28.965
of the SDR logo.
00:33:29.120 --> 00:33:29.360
Yeah.
00:33:29.360 --> 00:33:32.120
That was a fun one because a lot of
that is like the first stage of any
00:33:32.120 --> 00:33:35.500
project is just wiring it up and
then poking it to make sure that you
00:33:35.510 --> 00:33:38.690
wired it up right before you start
spending a lot of time writing a lot
00:33:38.690 --> 00:33:41.114
of software that adds more confusion.
00:33:41.114 --> 00:33:41.724
to the pile.
00:33:41.879 --> 00:33:44.575
And the other thing it lets you do is
scale horizontally This is another one
00:33:44.575 --> 00:33:47.845
of those server terms: scaling vertically
means you just buy a bigger server.
00:33:47.875 --> 00:33:52.689
You get the 64 with 96 gigs of ram
instead of a bunch of little machines,
00:33:52.689 --> 00:33:55.753
because for some things that's
just easier because then you're not
00:33:55.783 --> 00:33:57.303
worrying about a bunch of machines, but
00:33:57.573 --> 00:34:00.148
James, they
make so much bigger servers.
00:34:00.193 --> 00:34:00.894
Yeah, they do.
00:34:00.933 --> 00:34:01.503
Yeah, they do.
00:34:01.503 --> 00:34:02.603
You can
have terabytes of RAM.
00:34:02.873 --> 00:34:03.313
Hell yeah.
00:34:03.493 --> 00:34:05.363
But scaling horizontally
lets you say: look...
00:34:05.573 --> 00:34:09.823
if I'm doing something really weird and
I need a hundred input pins, instead of
00:34:09.823 --> 00:34:13.403
trying to find a chip that has a hundred
input pins, you go: I'm just going to buy
00:34:13.823 --> 00:34:16.763
10 chips that have a hundred input pins.
00:34:16.773 --> 00:34:20.803
So then I get my thousand input
pins and that's 10 USB connections.
00:34:20.803 --> 00:34:22.703
Linux is like, whatever, that's a USB hub.
00:34:22.743 --> 00:34:23.643
It's not a big deal.
00:34:24.363 --> 00:34:27.763
So we can go from our embedded
Linux SoC with one microcontroller.
00:34:28.078 --> 00:34:29.848
We stick a little USB hub in front of it.
00:34:29.848 --> 00:34:32.988
And all of a sudden we now have triple the
capacity for the thing that we're doing.
00:34:32.988 --> 00:34:35.458
We can have triple the motors,
triple the sensors, whatever.
00:34:35.808 --> 00:34:37.598
And this was cheap to do.
00:34:37.608 --> 00:34:43.338
You can buy a single chip USB hub
from WCH for like 60 cents and
00:34:43.338 --> 00:34:45.168
it's four port USB high speed.
00:34:45.178 --> 00:34:48.568
And all of a sudden you can talk to
four devices now for an extra buck.
00:34:48.638 --> 00:34:53.055
What is the maximum number
of devices you can hook onto a USB hub?
00:34:53.221 --> 00:34:55.655
There's, uh- I don't remember
either limit off the top of my head.
00:34:55.665 --> 00:34:58.265
The answer is you'll run into some
limit in your operating system and
00:34:58.265 --> 00:35:02.225
you'll run into some limit in the
version of USB that you are using.
00:35:02.225 --> 00:35:04.995
So like full speed and high
speed, but it's in the hundreds.
00:35:05.085 --> 00:35:08.777
Up to like a hundred or so everything
will probably work if you have enough
00:35:08.777 --> 00:35:13.099
power and things like that, that you
might have some degradation of how
00:35:13.099 --> 00:35:14.659
fast you can talk to all of them.
00:35:14.659 --> 00:35:17.859
You'll hit some other thresholds
first, but like tens: no problem.
00:35:18.109 --> 00:35:20.699
Fifties: probably fine to clear.
00:35:20.849 --> 00:35:21.479
Hundreds...
00:35:21.489 --> 00:35:22.919
well, you'll probably
start having problems.
00:35:22.919 --> 00:35:25.609
James, I need to pitch
you a video idea to your company.
00:35:25.629 --> 00:35:29.559
It's how many USB devices can we
hook up to a single computer before
00:35:29.679 --> 00:35:31.049
things start going very wrong?
00:35:31.336 --> 00:35:31.636
What was it?
00:35:31.726 --> 00:35:32.666
LTT did that.
00:35:33.077 --> 00:35:35.551
Yeah, they did it with like keyboards
or, flash drives or something.
00:35:35.581 --> 00:35:37.491
We can put that in the show
notes, but they did exactly that.
00:35:37.681 --> 00:35:41.111
They even talk about like, there's
the theoretical USB protocol limit.
00:35:41.461 --> 00:35:43.261
And then there's when
Windows just falls over.
00:35:43.321 --> 00:35:44.341
They all have power.
00:35:44.341 --> 00:35:45.171
It should be fine.
00:35:45.171 --> 00:35:48.241
But just at some point, the operating
system's like, "I just can't anymore."
00:35:48.241 --> 00:35:50.681
And it drops like 60 of them immediately.
00:35:50.681 --> 00:35:52.261
Like things just timeout probably.
00:35:52.291 --> 00:35:55.151
Then the next obvious step
is to write a driver so that you can have
00:35:55.161 --> 00:35:59.611
virtual USB devices that are actually
proxied over ethernet to a mega hub.
00:35:59.611 --> 00:35:59.701
Yeah.
00:35:59.891 --> 00:36:01.981
And I'm not sure
Linus did that one, James.
00:36:02.021 --> 00:36:02.431
Did he?
00:36:03.071 --> 00:36:05.351
By the time that you just
have that- probably Ethernet's the answer.
00:36:05.386 --> 00:36:05.906
Yeah.
00:36:06.001 --> 00:36:08.471
But that's
again, another conversation...
00:36:08.506 --> 00:36:10.096
Mac Minis are going
the other way, you're replacing
00:36:10.116 --> 00:36:12.796
Ethernet with USB4, then you get,
00:36:12.981 --> 00:36:14.121
or Thunderbolt specifically.
00:36:14.121 --> 00:36:14.471
yeah.
00:36:15.456 --> 00:36:17.732
Isn't USB4 and Thunderbolt
basically the same thing now?
00:36:17.799 --> 00:36:18.786
Uh, kinda.
00:36:18.786 --> 00:36:19.996
They're on the same connector.
00:36:19.996 --> 00:36:21.006
They have similar capabilities.
00:36:21.026 --> 00:36:23.097
Thunderbolt is slightly different.
00:36:23.153 --> 00:36:24.207
I don't know off the top of my head.
00:36:24.442 --> 00:36:25.432
That's very confusing.
00:36:26.197 --> 00:36:29.457
My opinion is more
buddies, more better, because if you
00:36:29.457 --> 00:36:32.827
can make a firmware that does one
thing and it's a hundred lines of
00:36:32.827 --> 00:36:36.857
code and it is obviously correct,
you will never need to touch that.
00:36:37.077 --> 00:36:40.857
If you write a firmware that does
10 things and it's 10,000 lines of
00:36:40.857 --> 00:36:43.947
code and you have to worry about
interactions between the different
00:36:43.947 --> 00:36:48.527
ones, that's not 10 times harder, it's
a hundred times harder versus having
00:36:48.527 --> 00:36:52.527
10 really simple things where you go:
yeah, I can see it all on one page.
00:36:52.737 --> 00:36:56.597
It's obviously just going to receive
a message, set the motor value.
00:36:56.867 --> 00:36:58.497
I don't have to worry
about synchronization.
00:36:58.507 --> 00:36:59.227
It's all fine.
00:36:59.237 --> 00:37:01.407
You don't want to talk
about Kubernetes, cause you're trying
00:37:01.407 --> 00:37:05.127
to keep it tight, but you're just
selling us a microservice architecture.
00:37:05.527 --> 00:37:05.797
Yeah.
00:37:05.877 --> 00:37:07.117
Oh, I absolutely am.
00:37:07.187 --> 00:37:08.407
I absolutely am.
00:37:08.427 --> 00:37:09.627
It's the same kind of vibe.
00:37:09.677 --> 00:37:11.587
It's just learning
lessons from other people.
00:37:11.697 --> 00:37:15.217
Sometimes, not all the time, but
sometimes it's easier to have one
00:37:15.217 --> 00:37:18.977
thought in your brain at a time, if
you can design it in a way where it's
00:37:18.977 --> 00:37:22.047
clear how you separate those and how
they interact with each other and
00:37:22.047 --> 00:37:23.467
you don't run into any other limits.
00:37:23.577 --> 00:37:27.423
Much like in real life,
it is all about choosing boundaries.
00:37:27.728 --> 00:37:28.808
Yeah, it's engineering!
00:37:29.318 --> 00:37:30.968
And the other thing is
you can iterate faster.
00:37:31.018 --> 00:37:34.338
When you have these big complex circuit
board designs, iteration takes a while.
00:37:34.338 --> 00:37:37.028
Like I said, designing the board,
updating the board, making any fixes...
00:37:37.508 --> 00:37:40.168
different people have to
synchronize on it because you know,
00:37:40.168 --> 00:37:41.368
different people have opinions.
00:37:41.578 --> 00:37:44.728
Versus if you have like relatively
siloed functionality, it's like,
00:37:44.978 --> 00:37:48.038
"Okay, we switched to a higher power
motor driver," or "Hey, we switched
00:37:48.038 --> 00:37:50.718
to three sensors instead of one
sensor," or something like that.
00:37:51.148 --> 00:37:53.638
Because, in the beginning, when
you're bringing up, like I was
00:37:53.638 --> 00:37:57.208
doing on my stream, you don't even
need embedded Linux to be working.
00:37:57.208 --> 00:37:58.748
You don't need to worry
about the image for that.
00:37:58.748 --> 00:38:01.324
You don't worry, you have flashing
that, updating it, provisioning it,
00:38:01.344 --> 00:38:03.254
whatever your laptop is the SoC.
00:38:03.504 --> 00:38:06.264
And at least with Rust, you can run
Rust on Windows, Mac, and Linux.
00:38:06.694 --> 00:38:08.214
If it's over USB, who cares?
00:38:08.244 --> 00:38:12.914
And if you don't have USB, you buy a
little USB to serial adapter or whatever.
00:38:12.914 --> 00:38:15.671
And all of a sudden you're
testing not that far from things.
00:38:15.721 --> 00:38:17.971
The first prototypes might be
thousands of dollars because you're
00:38:17.971 --> 00:38:19.241
making them in small batches.
00:38:19.521 --> 00:38:21.661
If you have five devs on your
team, you go, "I don't really
00:38:21.661 --> 00:38:23.071
want to make 10 boards..."
00:38:23.291 --> 00:38:25.671
and then someone fried one of them
because they plugged it in backwards.
00:38:25.721 --> 00:38:29.611
It's way easier to just have this
collection of cheap things and everyone
00:38:29.611 --> 00:38:31.021
gets a version that they can plug in.
00:38:31.241 --> 00:38:34.073
You can hack on that quickly because
you can go from your embedded
00:38:34.073 --> 00:38:36.793
Linux SoC to a laptop: whatever!
00:38:36.913 --> 00:38:37.263
No big deal.
00:38:37.263 --> 00:38:38.539
I was thinking
about the other way around.
00:38:38.539 --> 00:38:40.809
You can also mock the microcontroller.
00:38:40.839 --> 00:38:42.569
Like, it's just a bunch of messages.
00:38:42.609 --> 00:38:42.919
Yeah, yeah.
00:38:44.728 --> 00:38:45.413
We're going to get there.
00:38:46.453 --> 00:38:47.023
We're going to get there.
00:38:47.144 --> 00:38:49.174
You can replace either
half as the design evolves.
00:38:49.264 --> 00:38:51.794
You don't necessarily have to
worry about breaking your motor
00:38:51.794 --> 00:38:55.024
control circuit because you
updated your embedded Linux SoC.
00:38:55.054 --> 00:38:58.174
You can sort of tick tock
between the two pieces.
00:38:58.564 --> 00:39:00.964
And like you were saying, you can
test either half in isolation.
00:39:01.264 --> 00:39:05.804
If we have our normal system, which is our
SoC talking to the microcontroller, if we
00:39:05.804 --> 00:39:07.124
want to test the microcontroller system,
00:39:07.134 --> 00:39:10.234
we can have a hardware test rack
that is just exercising this
00:39:10.244 --> 00:39:11.394
with a bunch of sensors on it.
00:39:11.614 --> 00:39:12.944
And there's no embedded Linux.
00:39:12.944 --> 00:39:15.175
It's just CI server that's
just chugging on that.
00:39:15.585 --> 00:39:17.655
And if we want to test
just our application.
00:39:17.885 --> 00:39:18.935
Like you said, we can mock that.
00:39:18.935 --> 00:39:23.138
Postcard RPC even has the ability to
say like, it has a generic interface and
00:39:23.138 --> 00:39:27.028
I have a version that instead of using
the USB port, it uses tokio channels.
00:39:27.068 --> 00:39:31.188
So you can just have another task that's
pretending to be something like that.
00:39:31.438 --> 00:39:34.798
Poststation actually does that built in
where it will simulate devices for you.
00:39:35.028 --> 00:39:36.538
That just like ping and show data.
00:39:36.538 --> 00:39:38.526
So you can test that you can poke them.
00:39:38.819 --> 00:39:40.039
This is the same in software.
00:39:40.199 --> 00:39:43.369
You can buy yourself time
with things that work enough.
00:39:43.419 --> 00:39:47.009
Even if this design is too big to be
shipping to your customers, you can
00:39:47.009 --> 00:39:49.549
make a hundred units for internal usage.
00:39:49.719 --> 00:39:51.429
You can take them to demos.
00:39:51.439 --> 00:39:55.859
You can take them to user testing
where you're buying yourself time where
00:39:55.859 --> 00:39:59.249
you don't have to do the waterfally
kind of design, where you have to
00:39:59.249 --> 00:40:03.089
wait until everything's in its final
plastics before people can touch it.
00:40:03.519 --> 00:40:06.789
You can start getting that feedback
early and iterating on that design.
00:40:07.199 --> 00:40:11.029
And you can buy yourself a lot of
development time and unsticking that
00:40:11.039 --> 00:40:15.936
critical path in the design and worry
about optimizing for cost or these
00:40:15.936 --> 00:40:17.946
kinds of things after shipping V1,
00:40:18.246 --> 00:40:22.176
because a lot of the customers I
work with don't need V2 or they
00:40:22.176 --> 00:40:24.906
have the V1 that they shipped,
which is based on this buddy system.
00:40:25.106 --> 00:40:26.436
And they went, "You know, it's fine.
00:40:26.436 --> 00:40:27.026
It's okay.
00:40:27.026 --> 00:40:28.426
It's like 10 cents more a unit.
00:40:28.826 --> 00:40:29.956
A dollar more a unit..."
00:40:29.996 --> 00:40:33.996
usually the way you translate from
hardware to retail is: every dollar it
00:40:33.996 --> 00:40:37.706
costs you more to make something, you
should be charging three more at retail.
00:40:37.716 --> 00:40:40.106
So if it costs three bucks more,
it's going to increase your
00:40:40.106 --> 00:40:42.059
product probably by like 10 bucks.
00:40:42.517 --> 00:40:45.618
But in a lot of these low volume
cases or initial prototypes and
00:40:45.618 --> 00:40:49.498
things like that, just having
something that works well today is
00:40:49.508 --> 00:40:51.638
worth charging an extra 10 bucks for.
00:40:51.875 --> 00:40:53.315
And I'm going to use a couple terms here.
00:40:53.315 --> 00:40:54.605
There's BOM and NRE.
00:40:54.615 --> 00:40:56.615
So BOM is our bill of material.
00:40:56.835 --> 00:40:58.000
Byte order mark...
00:40:58.160 --> 00:40:58.770
Byte order mark.
00:40:58.790 --> 00:40:59.380
Exactly.
00:40:59.780 --> 00:41:02.460
Our bill of material, which is what
it costs us to make everything.
00:41:02.520 --> 00:41:05.580
It's the checklist of every bit that
gets assembled into our product.
00:41:05.860 --> 00:41:09.060
So our bill of materials, it's
a whole list and what it costs
00:41:09.060 --> 00:41:10.520
us per unit to make that.
00:41:10.930 --> 00:41:14.620
And the NREs, which are non recurring
expenses, which is you and me.
00:41:14.690 --> 00:41:18.030
That's engineering, design
time, that stuff you pay once.
00:41:18.370 --> 00:41:22.210
You have to include that cost in what
you're selling, but it's not a cost you
00:41:22.210 --> 00:41:25.923
have to keep paying when you're going
from a thousand units to a million units.
00:41:26.313 --> 00:41:29.403
But it's one of those things where if
you're only making 10 units and you have
00:41:29.403 --> 00:41:32.713
to pay 10 developers for a year, each
of those units needs to have a whole
00:41:32.713 --> 00:41:35.683
developer salary in the cost of the unit.
00:41:35.733 --> 00:41:40.443
Which is why this industrial stuff or
the experimental stuff: you buy one
00:41:40.443 --> 00:41:43.069
unit where you go, it's like a Windows
computer in a couple of things.
00:41:43.069 --> 00:41:44.609
Why are you charging me a million for it?
00:41:44.869 --> 00:41:47.639
And it's like, because I only make 10
a year and you need this and I'm the
00:41:47.639 --> 00:41:48.979
only one that makes this kind of thing.
00:41:49.542 --> 00:41:52.937
Yeah, so this is our per unit cost
versus our design and dev time, that's
00:41:52.937 --> 00:41:54.197
where that threshold comes from.
00:41:54.477 --> 00:41:58.567
When you have more than 10,000, more than
a hundred thousand units, that fraction
00:41:58.577 --> 00:42:00.807
gets divided really significantly.
00:42:01.217 --> 00:42:04.047
And it only adds pennies,
dollars to your bill of material.
00:42:04.247 --> 00:42:07.897
When you're making a thousand, if you
have one dev for one year and they're
00:42:07.897 --> 00:42:12.027
making 50, a hundred thousand dollars
a year, like not even San Francisco
00:42:12.027 --> 00:42:14.027
salaries, you divided that by a thousand.
00:42:14.027 --> 00:42:19.127
That is a non trivial amount of what you
have to charge per unit for one developer,
00:42:19.127 --> 00:42:20.821
and you probably have more than a couple.
00:42:20.932 --> 00:42:24.302
The whole goal here is you should
be treating your buddy as a partner
00:42:24.672 --> 00:42:26.132
instead of as a black box.
00:42:26.142 --> 00:42:28.522
Cause that's the history of embedded
is you go: well, I make my little
00:42:28.522 --> 00:42:31.442
embedded system and then we pretend
that it's not running firmware.
00:42:31.452 --> 00:42:34.372
It is just a thing that
works and it's a card.
00:42:34.402 --> 00:42:36.622
Whoever writes the driver has
to think about it, but then
00:42:36.622 --> 00:42:37.862
we never think about it again.
00:42:38.282 --> 00:42:42.186
My pitch is to treat it more like a
partner in the same way on servers where
00:42:42.186 --> 00:42:45.436
you go: look, I'm going to consciously
acknowledge these are all computers
00:42:45.446 --> 00:42:46.466
and they can talk to each other.
00:42:46.466 --> 00:42:49.606
And we can treat them not as
equals, but as like reasonable
00:42:49.606 --> 00:42:50.856
partners in this whole thing.
00:42:56.066 --> 00:42:59.456
This episode is sponsored by Postation,
a tool from OneVariable that makes
00:42:59.456 --> 00:43:02.806
it easy to set up communication
between your desktop, laptop, or
00:43:02.876 --> 00:43:06.276
an embedded Linux system to as many
connected microcontrollers as you need.
00:43:06.708 --> 00:43:09.458
If you're a company building a product
around multiple devices, and would
00:43:09.458 --> 00:43:12.538
like to have all of the plumbing,
tooling, and device management handled
00:43:12.538 --> 00:43:17.038
out of the box, send us an email to
contact@onevariable.com for early access.
00:43:17.405 --> 00:43:21.385
Check out onevariable.com/poststation
for more information.
00:43:21.775 --> 00:43:25.205
That's onevariable.com/poststation.