Rust for Linux with Alice Ryhl and Greg Kroah-Hartman

WEBVTT
00:00:01.450 --> 00:00:05.830
It's Rust in Production, a podcast about companies who use Rust to shape the
00:00:05.830 --> 00:00:06.750
future of infrastructure.
00:00:07.190 --> 00:00:11.450
My name is Matthias Endler from corrode, and in this very special live episode
00:00:11.450 --> 00:00:16.510
at RustWeek, we welcome Alice Ryhl and Greg Kroah-Hartman from the Rust for
00:00:16.510 --> 00:00:20.530
Linux project to talk about saving the Linux kernel with Rust.
00:00:23.330 --> 00:00:29.990
Welcome everyone to a very special live recording here at RustWeek in Utrecht.
00:00:30.970 --> 00:00:33.850
This one has been a long time in the making at
00:00:33.850 --> 00:00:36.950
least for me two of my personal heroes
00:00:36.950 --> 00:00:40.230
are here today and we will talk about the
00:00:40.230 --> 00:00:43.130
rust for linux project rust in the linux kernel
00:00:43.130 --> 00:00:46.250
now if some of you have
00:00:46.250 --> 00:00:49.530
listened to the podcast before we had danilo krummrich
00:00:49.530 --> 00:00:52.750
from red hat on the podcast already
00:00:52.750 --> 00:00:56.770
and we talked about the nouveau driver and
00:00:56.770 --> 00:01:03.090
yeah we are back with one more episode for rust in production my name is matthias
00:01:03.090 --> 00:01:10.790
and i have to thank Alice and Greg for coming over the first question is always
00:01:10.790 --> 00:01:15.650
can you introduce yourself and perhaps the organization you're working for Alice
00:01:15.650 --> 00:01:16.430
maybe if you want to start.
00:01:16.430 --> 00:01:23.090
So i'm Alice i work at google on the android rust team where I work on Rust,
00:01:23.330 --> 00:01:26.810
you know, in the Linux kernel, primarily.
00:01:27.470 --> 00:01:33.550
Rust on Android more generally. And I also maintain Tokio, the asynchronous runtime for Rust.
00:01:34.090 --> 00:01:38.510
And you are known as @darksonn on the internet, if some people remember that.
00:01:38.850 --> 00:01:39.310
Exactly, yeah.
00:01:40.960 --> 00:01:44.520
I'm Greg Kroah-Hartman linux kernel developer and maintainer i work for the links
00:01:44.520 --> 00:01:50.380
foundation for the past many years i maintain parts of the links kernel like the driver core usb,
00:01:51.240 --> 00:01:55.300
tty and lots of little things and i also maintain the stable kernels so we do
00:01:55.300 --> 00:01:59.160
the stable kernel releases once or twice a week and i'm part of the security
00:01:59.160 --> 00:02:05.260
team and part of the cna team for the CVEs and i've been getting involved in the rust stuff as well.
00:02:05.260 --> 00:02:11.200
Now the kernel is getting close to i think 30 years of age nowadays which is
00:02:11.200 --> 00:02:16.100
incredible and of course it has started as a c project and it has been a c project
00:02:16.100 --> 00:02:21.280
for a very long time when was the time when you got first involved in linux in general.
00:02:21.280 --> 00:02:31.660
Me i'm late in the late 1990s so only 25 years almost a newbie yeah a lot of
00:02:31.660 --> 00:02:36.320
the so no i got involved late 90s i had been using it in work for other things
00:02:36.320 --> 00:02:40.000
but i wrote firmware for usb devices and I worked on the USB specification.
00:02:40.400 --> 00:02:43.880
And I made devices like barcode scanners and I had to hook my,
00:02:44.040 --> 00:02:48.180
write my firmware to work on all other operating systems. So I tested on different ones.
00:02:48.380 --> 00:02:50.440
Linux did not have good USB support at the time.
00:02:50.800 --> 00:02:53.900
My device was supposed to show up as a keyboard, but if I plugged into Linux,
00:02:54.040 --> 00:02:55.760
it showed up as a 102 key mouse.
00:02:57.580 --> 00:03:01.160
That was my firmware's bug, but funnily enough, Windows worked fine.
00:03:01.440 --> 00:03:05.400
So I fixed the firmware and Linux worked fine. but then I started contributing
00:03:05.400 --> 00:03:09.120
to Linux kernel that way because I saw little bugs I could fix and things like
00:03:09.120 --> 00:03:12.760
that and got involved and I realized I could get a job doing Linux full time
00:03:12.760 --> 00:03:14.940
in the late 90s and haven't stopped since.
00:03:15.970 --> 00:03:20.810
Can you say you started contributing to linux because you got curious and you
00:03:20.810 --> 00:03:25.230
wanted to tinker with your own hardware and and you started to take on more
00:03:25.230 --> 00:03:26.850
and more responsibility in the project.
00:03:26.850 --> 00:03:30.570
Yeah i wanted my drivers and it was fun it was a fun feedback loop i wrote my
00:03:30.570 --> 00:03:33.830
first driver my wife went away for the weekend said work on that driver that
00:03:33.830 --> 00:03:37.850
you want to do i was like okay and i submitted it and i it was for a usb to
00:03:37.850 --> 00:03:41.190
serial converter and it was like instantly people wrote back this is wrong this
00:03:41.190 --> 00:03:44.930
is wrong this is wrong and oh have you heard of smp multi-processors i'm like
00:03:44.930 --> 00:03:46.670
what's that two processors at once?
00:03:47.410 --> 00:03:49.950
Locking? I never heard of that. But it was great. It was this feedback loop.
00:03:50.030 --> 00:03:53.450
And I learned that the engineering culture is, from an engineer's point of view,
00:03:53.750 --> 00:03:56.190
you learn so much from people who are much, much smarter than you.
00:03:56.310 --> 00:03:57.830
And I could become a much better programmer.
00:03:58.010 --> 00:04:02.850
I'd never learned about multiprocessors and four and 16-wave machines and things like that.
00:04:02.950 --> 00:04:06.930
And it teaches you to be a better engineer over time. And I just love that feedback loop.
00:04:07.250 --> 00:04:11.150
And it got really, really good after time. Funnily enough, one of my first messages
00:04:11.150 --> 00:04:12.970
to the mailing list was like, how do I make a patch?
00:04:13.190 --> 00:04:15.990
Cause it didn't, the documentation wasn't there. And the person who responded
00:04:15.990 --> 00:04:17.570
to me was very helpful and nice and whatnot.
00:04:18.110 --> 00:04:20.210
Like 10 years later, he ended up being my boss at SUSE.
00:04:21.310 --> 00:04:22.750
So it's a small world at times.
00:04:23.190 --> 00:04:23.510
Wow.
00:04:24.330 --> 00:04:28.450
It's pretty incredible. Did you get into any segfaults? Did you have any segfaults?
00:04:28.470 --> 00:04:31.150
Oh, machines crash all the time. But that's, I mean, you can crash your own
00:04:31.150 --> 00:04:33.830
machine and keep on going and whatnot. So it's not a big deal.
00:04:34.550 --> 00:04:38.750
And Alice, when was the first time you got involved in the Linux project or
00:04:38.750 --> 00:04:40.010
in the Rust for Linux project?
00:04:40.410 --> 00:04:43.030
Oh, I've been there for a long time, like three, maybe four years.
00:04:47.810 --> 00:04:54.050
But you worked on Rust as well, way prior to that. So when was the time you started with Tokio?
00:04:54.610 --> 00:04:58.830
That's right. So I've been playing around with Rust for maybe 10 years,
00:04:59.010 --> 00:05:04.630
but for Tokio, the Async space, I got involved in around 2020, I would say.
00:05:05.710 --> 00:05:10.350
And when was the time when you took over the project? Was that pretty much immediately
00:05:10.350 --> 00:05:12.490
or was there a transition period?
00:05:13.210 --> 00:05:17.710
Well, so, I mean, the way it all started was I showed up in the chat room.
00:05:18.210 --> 00:05:21.590
People were asking questions. How do I do this with Tokio?
00:05:22.090 --> 00:05:26.610
At some point I was like, I've seen this question a few times, let me go fix the docs.
00:05:27.110 --> 00:05:31.010
So that was my first pull request to Tokio that was a docs fix.
00:05:31.290 --> 00:05:35.870
And it kind of just kept going and after I don't actually know how long time
00:05:35.870 --> 00:05:37.190
it took, but after a while I,
00:05:38.080 --> 00:05:42.220
started you know becoming part of the maintainer team and you know it all went from there.
00:05:43.200 --> 00:05:46.640
What i find interesting in both of your stories is that
00:05:46.640 --> 00:05:53.140
you started taking on responsibility for extremely important technical projects
00:05:53.140 --> 00:05:58.980
and both of you came from a position of curiosity you started because you wanted
00:05:58.980 --> 00:06:04.260
to learn about that ecosystem and it wasn't a big bang moment or so,
00:06:04.380 --> 00:06:08.520
or you didn't graduate from university to become maintainers of these projects,
00:06:08.700 --> 00:06:12.960
but it was rather a very gradual process. And I really liked it.
00:06:13.880 --> 00:06:20.240
Over 50% of the Rust crates depend on Tokio, and I think everyone in this room has used Linux.
00:06:21.140 --> 00:06:23.380
Raise your hand who has never used Linux before.
00:06:24.380 --> 00:06:25.780
Because that would be a lie.
00:06:28.140 --> 00:06:32.280
Tons of Linux devices in this room alone. The projector. Sorry?
00:06:32.440 --> 00:06:34.660
The projector. The projector, even.
00:06:36.480 --> 00:06:39.620
And probably the technology for microphones and everything else.
00:06:41.400 --> 00:06:47.940
Now, when we talk about Rust for Linux specifically, was that a social challenge
00:06:47.940 --> 00:06:51.340
for the most part, or was it a technical challenge to introduce Rust?
00:06:52.880 --> 00:06:53.920
I mean...
00:06:53.920 --> 00:06:54.640
It's a trick question.
00:06:57.100 --> 00:06:59.220
Let's go with... I mean, that was both.
00:06:59.580 --> 00:07:01.000
It is both. It is both.
00:07:01.000 --> 00:07:04.860
But probably the biggest part was social.
00:07:05.580 --> 00:07:05.900
Why is that?
00:07:06.620 --> 00:07:13.400
Well, I mean, if you are going to introduce a new language to a project, it's a big investment.
00:07:14.460 --> 00:07:20.820
You have to do a lot of work to, you know, you start to use it in one place
00:07:20.820 --> 00:07:23.780
and then, you know, and then people have to...
00:07:25.300 --> 00:07:30.540
People who have been coding for C in C code all of their lives now have to suddenly
00:07:30.540 --> 00:07:34.980
care about this Rust code that's becoming part of their code base.
00:07:35.420 --> 00:07:41.700
And so there's a lot of stuff that you have to get buy-in from a lot of people.
00:07:42.020 --> 00:07:47.460
It's almost a bit like a leap of faith of sorts. You have to build trust first.
00:07:48.340 --> 00:07:54.500
Totally, yeah. Trust is the key here. I mean, technically it was proven five,
00:07:54.920 --> 00:07:58.100
eight years ago that Rust could work in the kernel, right? They had examples
00:07:58.100 --> 00:07:59.120
out of tree, it could work.
00:07:59.320 --> 00:08:05.140
But the kernel works on trust of individuals and we trust people to maintain
00:08:05.140 --> 00:08:07.560
the code that they're going to contribute and maintain it.
00:08:07.640 --> 00:08:10.400
And we don't necessarily trust that you're going to get it right because we all get it wrong.
00:08:10.540 --> 00:08:13.000
We trust that you're going to be there to fix it when you get it wrong,
00:08:13.200 --> 00:08:15.840
because we all get it wrong. And that's the trust model.
00:08:15.980 --> 00:08:19.660
We need to have the trust built up in the community that we have a big enough
00:08:19.660 --> 00:08:23.700
body of people both that can deal with this infrastructure and deal with this
00:08:23.700 --> 00:08:27.680
language and that benefits are there. We all agree that the benefits are there finally.
00:08:28.160 --> 00:08:32.120
And that we trust the people who are maintaining and working on this that they
00:08:32.120 --> 00:08:33.620
will maintain and work on it continue.
00:08:34.300 --> 00:08:38.280
What made Rust unique in that sense? Because there were other languages which
00:08:38.280 --> 00:08:43.440
were attempting to be adopted in the Linux kernel, which were not adopted before.
00:08:44.900 --> 00:08:50.180
I think if you're going to go to the big investment of introducing a new language,
00:08:50.780 --> 00:08:52.340
I think it has to be memory safe.
00:08:53.180 --> 00:08:58.600
There's no way it makes sense to spend that much time and effort if you're not
00:08:58.600 --> 00:09:01.920
going to go to a memory safe language these days.
00:09:03.900 --> 00:09:06.620
And it also has to be a language that you can use in the kernel.
00:09:06.920 --> 00:09:09.200
I mean, I'm sure Java would be great.
00:09:10.500 --> 00:09:13.320
But, you know, there's a lot of languages out there that
00:09:13.320 --> 00:09:16.140
are memory safe with a garbage collector or stuff
00:09:16.140 --> 00:09:21.760
like that but i don't think any of those would work in the kernel and on the
00:09:21.760 --> 00:09:27.140
other side you have all the memory and safe languages and i think rust is almost
00:09:27.140 --> 00:09:33.280
the only language that fits it's the combination of things right it has all
00:09:33.280 --> 00:09:34.400
the things you need i think.
00:09:34.400 --> 00:09:36.620
It was the right idea at the right time.
00:09:37.540 --> 00:09:41.980
I think people proved that it could work. I mean, before we always had these
00:09:41.980 --> 00:09:45.800
ideas, wouldn't it be great if we could use C++? And then all of us were like, no, please no.
00:09:46.440 --> 00:09:49.240
And for specific reasons. So nothing came along.
00:09:50.020 --> 00:09:54.000
Nobody spent the time and effort and energy to prove, hey, here is a solution
00:09:54.000 --> 00:09:57.660
of some way we can evolve forward. And the Rust developers did that.
00:09:58.060 --> 00:10:01.120
And I have to hand it to them for doing that real work. It took them many,
00:10:01.220 --> 00:10:04.080
many years, but they were the ones that did the work and proved that it could
00:10:04.080 --> 00:10:06.560
be done. And previously, nobody had ever done that.
00:10:07.400 --> 00:10:09.080
It's as simple as that. They did the work.
00:10:10.400 --> 00:10:12.960
But you could also do the work with another language.
00:10:13.180 --> 00:10:16.940
Nobody did, though. Yes, I'm not disagreeing with you, but see,
00:10:17.080 --> 00:10:20.160
these people did the work and proved to us that this is a good idea,
00:10:20.320 --> 00:10:23.120
and here are the benefits, and they're going to stick around.
00:10:23.300 --> 00:10:26.920
This is going to save—I mean, I'll talk in my talk tomorrow about why I think
00:10:26.920 --> 00:10:29.020
Rust is going to save Linux in the end.
00:10:29.460 --> 00:10:35.460
And I think that's—we now understand and believe that. and they did the work to prove that to us.
00:10:35.460 --> 00:10:38.040
It's a big statement to say rust is gonna safe Linux
00:10:38.040 --> 00:10:38.360
Yeah,
00:10:38.360 --> 00:10:47.480
I agree yeah i gotta talk tomorrow all about that.
00:10:49.860 --> 00:10:52.800
Now of course when when you say rust was the first
00:10:52.800 --> 00:10:55.660
to stick around and really tackle the hard
00:10:55.660 --> 00:10:58.540
problems i agree but at the same time i think rust
00:10:58.540 --> 00:11:02.300
is great because it can take a backseat in comparison to other languages like
00:11:02.300 --> 00:11:10.800
it can be still c for the majority of the kernel and you can take smaller bits
00:11:10.800 --> 00:11:15.400
and rewrite them in rust more or less gradually you don't have to do a big bang rewrite
00:11:15.400 --> 00:11:16.920
Interop
00:11:16.920 --> 00:11:17.760
Is the new rewrite.
00:11:17.760 --> 00:11:22.360
Can you elaborate so.
00:11:22.360 --> 00:11:26.760
I mean if you have your massive of code base in one language,
00:11:27.600 --> 00:11:30.360
like as you said, you can't rewrite the entire thing.
00:11:31.510 --> 00:11:36.990
There's no reason to do that. It doesn't make any sense. So what can you do instead?
00:11:37.370 --> 00:11:41.150
Well, what if you add a little bit of the new language, but then you have to
00:11:41.150 --> 00:11:44.830
talk, and that's interop is when one language talks to another.
00:11:45.350 --> 00:11:50.810
How can you do that from a technical perspective? How do you facilitate the
00:11:50.810 --> 00:11:54.770
interop story in the Linux kernel? Where do you even start?
00:11:55.790 --> 00:11:59.770
It's hard, and that was my biggest fear. My biggest worry was, you know,
00:11:59.910 --> 00:12:02.550
see in the kernel, we have these ideas of how we
00:12:02.550 --> 00:12:05.330
do objects how we do memory how we handle this stuff and Rust
00:12:05.330 --> 00:12:08.330
says it's very strong ideas of how it does memory and language and objects
00:12:08.330 --> 00:12:11.250
and references and stuff and those bindings it's called
00:12:11.250 --> 00:12:16.610
a binding is the magic glue right and getting that right is very very hard and
00:12:16.610 --> 00:12:21.330
we have very smart people this room and Alice to do that work and without that
00:12:21.330 --> 00:12:25.250
and that was my big fear that i don't think this can actually happen until we
00:12:25.250 --> 00:12:29.790
prove that those bindings will work and a lot of people spent the time and energy
00:12:29.790 --> 00:12:31.190
to prove that it could work.
00:12:31.510 --> 00:12:36.010
And we have working systems today as proof of that.
00:12:36.330 --> 00:12:39.970
But then you can take those bindings and just say from this point forward,
00:12:40.230 --> 00:12:43.530
hey, maybe we only write all the new graphics drivers in Rust,
00:12:43.730 --> 00:12:45.990
which the graphics driver maintainers have said.
00:12:46.490 --> 00:12:52.450
So then over time, new stuff coming in will evolve different language and we can evolve over time.
00:12:52.670 --> 00:12:56.150
The kernel evolves. We don't just do massive rewrites. Every little step of
00:12:56.150 --> 00:13:01.250
the way, every tiny change that goes in work on its own. You can't break anything.
00:13:01.470 --> 00:13:04.430
So it's slow evolution over time that change.
00:13:05.480 --> 00:13:07.360
Will work, and we'll do that.
00:13:07.940 --> 00:13:12.540
If you get the interop wrong, developer ergonomics will suffer.
00:13:12.800 --> 00:13:15.440
And we'll fix it. I mean, that's the best part of this stuff.
00:13:15.720 --> 00:13:18.380
There's nothing in these interoperability... I mean, we get it wrong all the
00:13:18.380 --> 00:13:19.280
time. We're human, right?
00:13:19.520 --> 00:13:22.940
We can fix it. There's nothing magic here. It's just code.
00:13:23.700 --> 00:13:25.660
Are there strategies to get it right the first time?
00:13:25.720 --> 00:13:26.820
Sure, we'd love to.
00:13:29.700 --> 00:13:34.580
Does it live from a lot of tension between the old world and the new world?
00:13:34.580 --> 00:13:38.200
Does it live from a lot of, let's say, inertia to say,
00:13:38.700 --> 00:13:42.860
okay, this is how we think about this abstraction in the C world,
00:13:42.860 --> 00:13:47.820
and then now we need to build a safer wrapper around this or an interrupt layer
00:13:47.820 --> 00:13:51.900
for this to communicate from the Rust side?
00:13:52.180 --> 00:13:56.800
Like, on the technical side, how do you do that? Where does the rubber hit the road?
00:13:57.680 --> 00:14:02.740
Maybe I can give one example. Let's say you have a C function that returns a
00:14:02.740 --> 00:14:08.440
pointer. Now the question is, what are the semantics of this pointer?
00:14:09.860 --> 00:14:14.760
So in Rust, there are many different pointer types. We have the reference,
00:14:14.960 --> 00:14:17.840
we have the box, we have the arc, we have a bunch of different pointer types.
00:14:18.260 --> 00:14:24.160
And so when we write a Rust version of this C function, or like a Rust wrapper
00:14:24.160 --> 00:14:30.080
that calls the C function, we have to think, okay, this pointer, what are the semantics?
00:14:30.480 --> 00:14:33.300
And then in the Rust version, we have to pick the right pointer type.
00:14:34.060 --> 00:14:40.760
And so in this sense, we're adding more details to the signature.
00:14:40.960 --> 00:14:44.600
We're saying this C function, which didn't say in the signature,
00:14:44.600 --> 00:14:48.120
what kind of pointer is it? Is it an owned pointer? Is it a borrowed pointer?
00:14:48.480 --> 00:14:53.020
We have to add that. And so in some sense, we're adding a lot more semantics
00:14:53.020 --> 00:14:57.020
in the code that were not there in the C version. And that's the tricky part.
00:14:57.320 --> 00:15:02.280
But also along those ways, we've seen APIs and interfaces that we have in C
00:15:02.280 --> 00:15:04.360
that are just simple and easy to use in C.
00:15:05.060 --> 00:15:08.560
And it turns out it would be very complex in Rust. Like famously in the driver
00:15:08.560 --> 00:15:12.040
core, there was some stuff that the driver, people writing the driver core bindings
00:15:12.040 --> 00:15:16.200
were going through hundreds and hundreds of lines of Rust code when you could do it in one line of C.
00:15:16.660 --> 00:15:19.140
Because, and I said, well, we can change the C code.
00:15:19.760 --> 00:15:22.360
And they're like, oh, and that was the big moment that, hey,
00:15:22.460 --> 00:15:25.040
if we work together, I can make C code simpler, do different,
00:15:25.180 --> 00:15:26.700
and you get to delete hundreds of Rust code.
00:15:26.940 --> 00:15:29.740
Great, let's do that. And a lot of the work we've done on the C side,
00:15:29.880 --> 00:15:33.860
if Rust were to disappear today, hey, we've made our C code better because Rust has showed up.
00:15:33.980 --> 00:15:37.160
We've had to think about how these pointers work. We've had to think about,
00:15:37.280 --> 00:15:41.220
is this a mutable pointer or not? Is it const or is it not? How do we work on this?
00:15:41.400 --> 00:15:43.580
And we changed the C code in the APIs because we can.
00:15:43.880 --> 00:15:48.180
And that's been a huge benefit for everybody. Even if they don't touch the Rust
00:15:48.180 --> 00:15:50.720
code, the C code has benefited so much from that.
00:15:51.870 --> 00:15:56.930
Hinges a lot on the collaboration of a maintainer of a subproject sure.
00:15:56.930 --> 00:16:00.010
But that's fine i mean yes and.
00:16:00.010 --> 00:16:04.650
There might be subprojects which are more open to the idea and some which are more resistant.
00:16:04.650 --> 00:16:10.310
I know where you're going with it so the big so yes there's a lot of maintainers
00:16:10.310 --> 00:16:13.910
that don't want to deal with another language right for various reasons and
00:16:13.910 --> 00:16:17.690
that's fine so in the kernel we have the rule that if you bring some code in
00:16:17.690 --> 00:16:20.750
and maintainer doesn't want to maintain it you have to be the maintainer of it.
00:16:21.290 --> 00:16:25.130
It's as simple as that. So then we have to find a responsible person that we
00:16:25.130 --> 00:16:26.270
trust to maintain this code.
00:16:26.410 --> 00:16:30.170
So for some subsystems, the maintainer that comes in for the bindings between
00:16:30.170 --> 00:16:35.610
the C and the Rust code is a new developer and they will work alongside the C maintainer, right?
00:16:35.770 --> 00:16:38.190
For other subsystems, I'll say like the driver core with me,
00:16:38.670 --> 00:16:42.470
Danilo came on and he said, I will be the, just maintain the driver core bindings,
00:16:42.570 --> 00:16:44.850
right? I said, no, no, no, no. You're going to be the maintainer of all this.
00:16:45.570 --> 00:16:48.770
So we share it. Me and Rafael and Danilo share it. We do the binder,
00:16:48.930 --> 00:16:51.250
we do the stress side, we do the seaside and we work together.
00:16:51.510 --> 00:16:54.190
So it depends on the subsystem. It's just, it's working with people.
00:16:54.330 --> 00:16:56.190
It's all just people and interactions.
00:16:56.730 --> 00:16:59.950
And so far it's worked out really well. We have some new bindings that are coming
00:16:59.950 --> 00:17:03.670
in for one subsystem and a maintainer that is like, I don't even like maintaining
00:17:03.670 --> 00:17:06.030
the subsystem in the seaside. You want to maintain them both?
00:17:06.450 --> 00:17:09.250
I mean, and maybe the person who's doing the Rust side will say,
00:17:09.330 --> 00:17:11.730
okay, great, I'll do that. And you take over and go from there.
00:17:12.030 --> 00:17:15.150
So it's just, we'll work through it as each individual subsystem.
00:17:15.230 --> 00:17:20.610
And we have a good, I want to say, a very solid mass, critical mass in the kernel
00:17:20.610 --> 00:17:24.110
of these bindings from the C to the Rust side to prove you can write real drivers.
00:17:24.230 --> 00:17:31.530
And Alice has proven you can write real drivers in code in Rust for a device that works today.
00:17:31.830 --> 00:17:34.450
So going forward, it should be much easier, much simpler.
00:17:34.870 --> 00:17:38.330
Writing the Rust driver code is much simpler than the bindings.
00:17:39.410 --> 00:17:44.770
At least from an outsider's perspective, there used to be some resistance from
00:17:44.770 --> 00:17:49.850
some subproject maintainers to give Rust a chance in their ecosystem.
00:17:50.390 --> 00:17:54.630
Has that changed to some extent? Have they changed their mind?
00:17:55.690 --> 00:18:01.730
Did you see any change in the way Rust is perceived in the Linux kernel by people
00:18:01.730 --> 00:18:05.410
who are, let's say, C veterans, very experienced C developers?
00:18:05.890 --> 00:18:10.330
I mean, I'm a C veteran. I've been doing this for C for 35 years every day.
00:18:11.090 --> 00:18:15.630
People always disagree. We have 5,500 kernel developers every year.
00:18:15.750 --> 00:18:17.590
We can only agree on one thing.
00:18:17.790 --> 00:18:21.290
And that only one thing is that we want Linux to succeed. After that,
00:18:21.350 --> 00:18:22.210
we disagree about everything.
00:18:23.890 --> 00:18:26.910
We're people, right? We don't have managers. We have the direction.
00:18:27.090 --> 00:18:31.470
But we now have a critical mass of core maintainers. We made a public statement
00:18:31.470 --> 00:18:34.950
last year saying, the Rust experiment is over. It's not an experiment.
00:18:35.110 --> 00:18:38.270
We're going to do this for real. we accept this as a project,
00:18:38.270 --> 00:18:39.810
we're going to go forward with this.
00:18:40.350 --> 00:18:41.570
Maintainers don't have to work
00:18:41.570 --> 00:18:43.510
with it and they don't have to interact with it if they don't want to.
00:18:43.630 --> 00:18:47.130
Just like any individual maintainer doesn't have to accept changes from anybody
00:18:47.130 --> 00:18:49.230
else, but everybody can be routed around.
00:18:49.550 --> 00:18:54.210
It's ostensibly this very pretty triangle of pyramid of hierarchy,
00:18:54.350 --> 00:18:57.750
but if you graph it, code flows in from everywhere, which is good.
00:18:57.810 --> 00:18:59.310
We can all modify other people's code.
00:18:59.470 --> 00:19:02.290
We can maintain other people's code. We can fix bugs in other people's code
00:19:02.290 --> 00:19:05.670
and get the code merge and get it out to users which is what matters in the
00:19:05.670 --> 00:19:07.570
end so it's a very flexible,
00:19:08.450 --> 00:19:11.510
hierarchy system and yeah some people don't want to deal with it some people
00:19:11.510 --> 00:19:15.910
don't want to deal with people who use ai generated patches and they can refuse
00:19:15.910 --> 00:19:19.850
those too i mean we have lots of rules of lots of different things not a problem.
00:19:19.850 --> 00:19:25.290
Alice do you remember the time when you entered the picture and you started
00:19:25.290 --> 00:19:31.230
to build out all of these really great interrupt abilities between Rust and C?
00:19:31.470 --> 00:19:37.070
How was that initial phase of building out all of these data structures in?
00:19:38.060 --> 00:19:42.700
Things have totally changed, right? So, okay, so one way to see it is there's
00:19:42.700 --> 00:19:48.120
this yearly conference called Linux Plumbers where all the kernel developers show up and we talk.
00:19:48.580 --> 00:19:51.780
And I've been going there for a few years now.
00:19:52.000 --> 00:19:57.400
And every single time I've gone there, things have totally changed from the last year, right?
00:19:57.460 --> 00:20:00.900
I've seen this happen like four years in a row or something like that.
00:20:01.460 --> 00:20:06.920
So, yeah, I mean, like sure, there are some subsystems left,
00:20:06.920 --> 00:20:10.820
But there are definitely subsystems that, you know, I came to them,
00:20:10.960 --> 00:20:15.220
we had 0% Rust code, and then we started writing some Rust code.
00:20:15.360 --> 00:20:19.380
And then the next year, they were like, oh, I guess you have a cute toy there.
00:20:19.500 --> 00:20:21.740
And then the next year, they have Rust code in their subsystem.
00:20:21.940 --> 00:20:25.220
And then the next year, it's no longer an experiment.
00:20:25.240 --> 00:20:30.920
Why would we not do this at all? I mean, it takes time. It's human behavior, right?
00:20:31.100 --> 00:20:34.560
It's just, we're just people. we can't do mandates but
00:20:34.560 --> 00:20:37.620
we can persuade and discuss and hopefully convince people
00:20:37.620 --> 00:20:40.320
i was convinced i was astonished this is
00:20:40.320 --> 00:20:44.040
not going to work at all and then i went to one of the Rust conferences and
00:20:44.040 --> 00:20:47.820
i think i had a whole presentation aimed directly at me for like an hour and
00:20:47.820 --> 00:20:52.020
a half to prove hey this can be done and then we went out in the hallway and
00:20:52.020 --> 00:20:55.080
discussed for another three or four hours and then over drinks that night even
00:20:55.080 --> 00:20:58.100
more and i was convinced because.
00:20:58.100 --> 00:20:59.000
Of the alcohol or.
00:20:59.000 --> 00:21:02.940
Because no no no what.
00:21:02.940 --> 00:21:06.140
Was your main concern that people wouldn't stick around until the end.
00:21:06.140 --> 00:21:09.840
Well my main concern again was back to this bindings is can we come up with
00:21:09.840 --> 00:21:13.300
a way to make these things interact in a way that's going to work good for both
00:21:13.300 --> 00:21:18.660
sides and i think Danilo and others have proven that actually it's that interaction
00:21:18.660 --> 00:21:23.160
and that binding is going to work better than i imagine so i'll call out they've
00:21:23.160 --> 00:21:24.340
done an amazing job there.
00:21:26.140 --> 00:21:31.140
Yeah a quick round of applause for that well deserved.
00:21:32.580 --> 00:21:36.460
I certainly wouldn't contribute to the C side of the kernel because I would
00:21:36.460 --> 00:21:37.640
feel intimidated, honestly.
00:21:37.900 --> 00:21:44.300
I would not know if I could contribute in a safe manner, but I would definitely
00:21:44.300 --> 00:21:47.300
be interested in contributing to the rust side nowadays.
00:21:47.680 --> 00:21:55.320
Have you ever noticed any change in maintenance or maybe contribution behavior?
00:21:55.840 --> 00:22:02.240
Are there more rust contributors now in comparison to maybe two, three years ago?
00:22:02.580 --> 00:22:07.140
If there are more people yeah for sure yeah absolutely yeah.
00:22:08.170 --> 00:22:11.170
And the other side, is that another thing where you say, okay,
00:22:11.690 --> 00:22:17.550
if you build Rust bindings for a C subset, you have to have a lot of empathy
00:22:17.550 --> 00:22:20.990
for the C code of that subset.
00:22:21.150 --> 00:22:26.790
But when you introduce new C code to the library, do you also keep Rust in mind now?
00:22:26.990 --> 00:22:31.170
And to say, well, okay, maybe eventually we want to call this from C,
00:22:31.270 --> 00:22:37.750
so maybe we build out our code very differently now in comparison to before Rust existed.
00:22:38.610 --> 00:22:42.850
Well, yes and no. So we don't try and add code to the kernel that's not used.
00:22:43.390 --> 00:22:47.170
So you can't just say, I want to add a binding and here's an interface.
00:22:47.650 --> 00:22:50.970
Sometime in the future, somebody can call us from Rust. We try not to add that.
00:22:51.070 --> 00:22:54.710
It's been a little chicken and egg problem with a lot of the bindings in the
00:22:54.710 --> 00:22:56.410
beginning because we didn't have a real user.
00:22:56.690 --> 00:23:01.390
Now we almost have a real user for everything. But when you try and add something, we need a real user.
00:23:01.390 --> 00:23:04.290
If you're just adding new functionality to the c side that nobody
00:23:04.290 --> 00:23:07.150
in the Rust side is using why would you add the binding because nobody's
00:23:07.150 --> 00:23:09.970
going to use it it's not going to be there just add it when it's needed
00:23:09.970 --> 00:23:14.790
and away you go just add stuff as needed so you see a lot of the initial bindings
00:23:14.790 --> 00:23:18.810
that are coming in for there's tiny subsets of the bindings like look at some
00:23:18.810 --> 00:23:22.230
of the stuff for the file systems that you added it was just a tiny subset of
00:23:22.230 --> 00:23:25.750
interacting with files for what was needed for one driver and then we'll grow
00:23:25.750 --> 00:23:28.210
from there or you're doing adding stuff for netlink,
00:23:28.490 --> 00:23:31.630
tiny subset of, we can't really do a full network packet, but maybe we can do
00:23:31.630 --> 00:23:33.810
a netlink message that way.
00:23:34.030 --> 00:23:36.410
So we'll just grow over time. And again, it's just evolution.
00:23:36.750 --> 00:23:39.930
You don't want to add giant chunks at once because it's unreviewable.
00:23:40.430 --> 00:23:43.770
And if it's not being used, why is it even there in the first place?
00:23:44.570 --> 00:23:48.110
Can you use a lot of the Rust standard library in the Linux kernel,
00:23:48.270 --> 00:23:50.910
or do you really have to write everything from scratch?
00:23:51.510 --> 00:23:57.550
I remember that the Linux kernel has its own allocator and there's a lot of
00:23:57.550 --> 00:24:01.310
emphasis on linked lists, for example, on the C side, but I think there's also
00:24:01.310 --> 00:24:03.410
an implementation for it on the Rust side.
00:24:04.370 --> 00:24:08.570
Is there a huge overlap between what the standard library provides and what
00:24:08.570 --> 00:24:11.250
the Linux kernel needs and where does that overlap end?
00:24:12.310 --> 00:24:15.530
So the Rust standard library is split into three parts.
00:24:15.870 --> 00:24:20.770
There's core, which is the core stuff, which we do use in the kernel.
00:24:21.630 --> 00:24:27.810
Then there's alloc, which is the things that only require an allocator and nothing else.
00:24:28.430 --> 00:24:33.530
We used to use that, but we wrote our own now. And then there's the std,
00:24:33.690 --> 00:24:37.090
the actual thing most people think of as the standard library,
00:24:37.330 --> 00:24:41.550
which provides things like files and TCP stream, But they require that there's
00:24:41.550 --> 00:24:46.110
an operating system below you, which there isn't in Linux.
00:24:46.250 --> 00:24:48.450
So it doesn't make sense to use std.
00:24:50.140 --> 00:24:55.340
Originally, we were using core and alloc, and alloc was our allocator.
00:24:56.020 --> 00:25:00.640
Eventually, we found out that the Linux kernel does a bunch of interesting stuff
00:25:00.640 --> 00:25:05.680
for its allocators, and so we wrote our own alloc crate, but yeah.
00:25:06.300 --> 00:25:07.340
What interesting stuff.
00:25:08.320 --> 00:25:12.380
Think about this. Memory needs to be allocated, and you can say, don't swap.
00:25:13.320 --> 00:25:17.660
Don't do any I.O. to give me this memory, or give it to me on that NUMA node over there,
00:25:17.660 --> 00:25:21.700
or give it over here or give it out of this bucket or that bucket or merge these
00:25:21.700 --> 00:25:26.500
buckets and memory is i say it's just a simple driver for memory chips right
00:25:26.500 --> 00:25:30.560
but that's a really complex beast underneath there so interacting with that
00:25:30.560 --> 00:25:36.140
from the rest of the kernel is not just malloc and free it's free but the malloc
00:25:36.140 --> 00:25:37.420
part is a little more nuanced.
00:25:39.000 --> 00:25:43.720
That means that allocator is very much specific to the linux kernel itself.
00:25:43.720 --> 00:25:44.087
Yes
00:25:44.087 --> 00:25:44.380
It doesn't.
00:25:44.380 --> 00:25:48.800
Make sense to use it anywhere else? Or is that a standalone library that I could
00:25:48.800 --> 00:25:50.160
use for any other purpose?
00:25:50.600 --> 00:25:52.120
For now, it's specific to the kernel.
00:25:52.280 --> 00:25:56.700
It's built on top of the C side. So I mean, it's built on top of what we have in the kernel today.
00:25:56.880 --> 00:26:00.400
So there's just some, when you call and you ask for memory, you have to give
00:26:00.400 --> 00:26:04.080
it some hints of what type of memory you want and what you're doing at that moment in time.
00:26:04.740 --> 00:26:07.720
That in a normal Rust user space program makes no sense at all.
00:26:07.980 --> 00:26:10.600
So you have to actually give it more options, more parameters.
00:26:11.400 --> 00:26:15.620
You started to adopt Rust in the Linux kernel on the, I would say,
00:26:16.240 --> 00:26:18.620
outskirts or maybe on the driver's side of things.
00:26:18.820 --> 00:26:22.800
And I think that was very clever because, well, drivers are isolated,
00:26:23.660 --> 00:26:27.740
less risk, less of a blast radius. You can experiment more there.
00:26:27.940 --> 00:26:31.680
They don't have to be in tree. You can pull them back in tree if you like to,
00:26:31.860 --> 00:26:33.460
but only once they are mature enough.
00:26:33.760 --> 00:26:40.800
Do you find that Rust finds its way into the core of Linux now with maybe eventually
00:26:40.800 --> 00:26:44.340
replacing this allocator that was written in C with rust.
00:26:45.060 --> 00:26:46.900
So actually, drivers...
00:26:47.950 --> 00:26:50.570
It makes it sound like drivers are the easiest thing to do for Rust.
00:26:50.730 --> 00:26:55.050
Drivers are the hardest thing to do because drivers consume from all the kernel.
00:26:55.670 --> 00:26:58.970
So Alice, to write a driver in Rust, had to make bindings to everything,
00:26:59.690 --> 00:27:04.870
like to the alloc crate, to do this, to do IO, to touch this file system,
00:27:04.950 --> 00:27:07.410
to do a memory malloc, to do all this stuff.
00:27:07.410 --> 00:27:13.250
So drivers are just, they look like a leaf on a tree, but they consume from the whole trunk, right?
00:27:13.370 --> 00:27:15.650
It's easier to write a core piece of the trunk in Rust.
00:27:15.790 --> 00:27:21.190
In fact, the first code in Rust was a standalone bit of code that was added to the kernel.
00:27:21.430 --> 00:27:24.310
It was to write, do the QR code when the kernel crashes.
00:27:24.730 --> 00:27:27.510
Do that logic. And famously, it had a memory bug.
00:27:29.290 --> 00:27:32.970
You can write memory unsafe code in Rust very easily in the kernel.
00:27:33.230 --> 00:27:36.690
Memory safety means something different in the kernel. So that standalone piece
00:27:36.690 --> 00:27:38.850
of Rust code was actually very easy and simple to do.
00:27:38.970 --> 00:27:41.370
It was just one C in and one C function out.
00:27:41.530 --> 00:27:46.830
But to do drivers was very much more complex. And we had to agree as a community
00:27:46.830 --> 00:27:51.110
to accept this as going to be a possibility even more to do it as a driver.
00:27:52.350 --> 00:27:58.450
Going forward who knows we'll see we still don't have full support for all architectures
00:27:58.450 --> 00:28:01.950
that we support we saw what s390 got posted again today.
00:28:01.950 --> 00:28:03.030
Yeah one.
00:28:03.030 --> 00:28:07.030
Of those architectures so we want to get all the main main architectures to
00:28:07.030 --> 00:28:11.270
have it before we'll be willing to put a core code to it because we just don't want to break systems.
00:28:11.270 --> 00:28:16.230
Yeah the big question is if you're going to have a core piece of logic is i
00:28:16.230 --> 00:28:20.430
mean if it's a new core piece then okay but if it's an existing core piece do
00:28:20.430 --> 00:28:28.630
you now have two implementations no this is this is great right so the architecture thing is pretty,
00:28:29.250 --> 00:28:33.790
important factor here in whether or not we use it in the call right.
00:28:33.790 --> 00:28:38.770
Because the allocator is shared between the rust side and the c side and it's
00:28:38.770 --> 00:28:43.630
very critical i also thought of schedulers for a moment but again we have schedulers
00:28:43.630 --> 00:28:48.190
they do work i don't even know what the latest scheduler is that is used in
00:28:48.190 --> 00:28:49.550
the default Linux kernel.
00:28:52.090 --> 00:28:55.910
You can write schedulers in BPF. So people are hacking their little steam boxes
00:28:55.910 --> 00:28:58.650
to write schedulers from user space to do better framework.
00:28:58.930 --> 00:29:02.350
And that same scheduler actually works really well in giant servers.
00:29:02.550 --> 00:29:05.650
So it's kind of fun. So you can write schedulers from user space in Rust,
00:29:06.370 --> 00:29:10.050
compiled to BPF, injected into the kernel. And people have been doing that for a few years now.
00:29:11.240 --> 00:29:16.940
What organizations use Rust for is mostly foundational to an organization.
00:29:17.160 --> 00:29:21.100
It's in the core, the thing that cannot fail. This is where you get the most
00:29:21.100 --> 00:29:24.300
leverage from the language. It's memory safe.
00:29:24.580 --> 00:29:28.300
And this is how you can hook into that system, sort of.
00:29:28.560 --> 00:29:35.520
And the thing that, in my opinion, comes closest to that in the Linux kernel would be I.O., I guess.
00:29:35.800 --> 00:29:41.060
We have a couple of different I.O. systems in the kernel. I can't list all of
00:29:41.060 --> 00:29:44.200
them, but they were poll-based ones. There's IO Uring.
00:29:45.160 --> 00:29:49.940
And Alice, you will also give a talk on IO. I'm not sure if that is related
00:29:49.940 --> 00:29:54.980
to your work on the Linux kernel, or if you can maybe share what this work is about.
00:29:55.780 --> 00:30:00.640
So I have a talk about completion based IO, but that's about user space,
00:30:00.800 --> 00:30:02.380
async Rust, and the Tokio side.
00:30:02.660 --> 00:30:07.740
Here the question is, it's actually kind of difficult to use IO Uring from async Rust,
00:30:08.300 --> 00:30:14.280
because we designed our trades to work really well with ePol and then Linux
00:30:14.280 --> 00:30:19.860
came around in 2019 and said hey we have this thing called io_uring which does
00:30:19.860 --> 00:30:26.140
not work in the same way and is not necessarily so easy to just swap them around,
00:30:27.240 --> 00:30:31.420
so yeah that's what my talk later today is going to be about.
00:30:31.420 --> 00:30:37.980
What would have to change in Tokio to support this new I.O. model in Linux better?
00:30:38.820 --> 00:30:42.820
So in some places we already have started using it actually.
00:30:43.180 --> 00:30:48.560
So if you have a file in Tokio, then you can go through,
00:30:48.900 --> 00:30:53.980
then if you enable the AyoUring stuff, then it will transparently use AyoUring
00:30:53.980 --> 00:30:59.760
instead of the normal background thread pool to do file IO, which is pretty great.
00:30:59.760 --> 00:31:05.080
But it's going to be much more tricky to use it for networking,
00:31:05.100 --> 00:31:07.380
for example. Because for networking, Yeah.
00:31:08.710 --> 00:31:12.690
Well, because networking is where ePoll is used today. And ePoll is kind of
00:31:12.690 --> 00:31:17.730
difficult to translate to our hearing without changing the API we have, right?
00:31:18.550 --> 00:31:22.190
And that's the thing, right? In user space, if we change the API,
00:31:22.870 --> 00:31:24.170
are people going to use it?
00:31:24.290 --> 00:31:28.810
You need some sense of backwards compatibility for people to adopt your new stuff.
00:31:29.530 --> 00:31:32.230
That's the mantra of the Linux kernel. Don't break user space.
00:31:32.370 --> 00:31:33.870
The kernel is always at fault then.
00:31:34.650 --> 00:31:37.530
Yeah, but we have this boundary of what's user space
00:31:37.530 --> 00:31:40.730
what's the kernel that's very well defined but within that we change everything
00:31:40.730 --> 00:31:44.610
all the time and so those are our apis are internally like you say the binding
00:31:44.610 --> 00:31:47.390
we get it wrong the first time great we'll rewrite it and make it work that's
00:31:47.390 --> 00:31:50.830
not seen by user space at all so yeah our goal is never to break user space
00:31:50.830 --> 00:31:53.170
you should always be able to update to the latest kernel and nothing should
00:31:53.170 --> 00:31:56.030
ever break if so then we did something wrong.
00:31:56.030 --> 00:31:58.850
Rust could also make that boundary a bit safer.
00:31:58.850 --> 00:32:03.810
Has nothing to do with rust sorry that
00:32:03.810 --> 00:32:06.830
that's not that's not the the issue
00:32:06.830 --> 00:32:11.950
that api of how you enter into the kernel and how you exit back out is language
00:32:11.950 --> 00:32:16.370
agnostic let's just say it's just a way of getting data in and out and rust
00:32:16.370 --> 00:32:20.810
no language matters there we traditionally have c wrappers but rust has rust
00:32:20.810 --> 00:32:25.890
wrappers and go has go wrappers and all etc how that goes.
00:32:25.890 --> 00:32:31.530
Now I find it interesting also, Ellis, that you work on both sides.
00:32:31.730 --> 00:32:37.830
You work on, say, quote-unquote, a user space application and the kernel space.
00:32:38.690 --> 00:32:43.790
And if you find a thing that maybe in Tokio could have been done differently
00:32:43.790 --> 00:32:51.730
on the Linux kernel side, is there a way for you to maybe encourage a discussion around that?
00:32:51.890 --> 00:32:57.230
To say, oh, can we change this API that would help us on the user space so much?
00:32:57.230 --> 00:33:01.910
Or do you strictly separate that work between Tokio and the Linux kernel right now?
00:33:03.140 --> 00:33:10.340
So obviously, if I want to make some Linux kernel changes motivated by my user
00:33:10.340 --> 00:33:14.440
space work, it would be a lot easier now that I know how the Linux kernel community works.
00:33:14.820 --> 00:33:18.300
I don't think we've had user space API changes per se.
00:33:18.540 --> 00:33:24.580
We have had a case where we had, I think, a kernel bug, and then I already knew
00:33:24.580 --> 00:33:27.400
the person on the Linux kernel side that I needed to speak to,
00:33:27.960 --> 00:33:31.600
and then we could figure out what the bug was and get it fixed.
00:33:31.600 --> 00:33:35.580
So in that sense, the crossover is quite useful.
00:33:36.400 --> 00:33:42.640
Now, looking ahead a little bit into the future of the Rust for Linux project,
00:33:42.640 --> 00:33:48.340
what would you say will land in the next 6 to 12 months?
00:33:48.600 --> 00:33:53.820
What can people expect there? Or is it mostly going to be fixing small little
00:33:53.820 --> 00:33:58.460
paper cuts, making the APIs a little nicer? Or are there any big plans?
00:33:59.400 --> 00:34:03.500
I mean, everybody asks what's next in Linux. It's whatever happens to show up.
00:34:04.060 --> 00:34:07.800
We don't know. We don't know what people are working on. We have hints of what people are working on.
00:34:07.980 --> 00:34:10.660
I have hints of people saying, I'm going to be doing this and this and this.
00:34:10.740 --> 00:34:12.840
Like, that's great. I believe it when I see it.
00:34:13.880 --> 00:34:17.620
So whatever shows up, we'll deal with and we'll handle it then.
00:34:17.840 --> 00:34:22.380
So we have ideas that people are, you see hints of what people are writing new
00:34:22.380 --> 00:34:23.700
drivers in that are in Rust.
00:34:23.960 --> 00:34:27.540
We have the graphics drivers being written in Rust. We have other things being written in Rust.
00:34:28.200 --> 00:34:30.220
Let's see what happens. Let's see what shows up.
00:34:30.640 --> 00:34:32.280
, what's on your agenda?
00:34:33.120 --> 00:34:33.600
Well,
00:34:34.970 --> 00:34:40.670
We have some pretty cool projects ongoing. So actually here at the conference,
00:34:40.670 --> 00:34:47.450
we have a booth where you can try to play a racing game on a device running
00:34:47.450 --> 00:34:50.950
a Rust GPU driver in the kernel.
00:34:52.250 --> 00:34:57.330
So, you know, that shows that it's working. It's very much a prototype at this point.
00:34:57.470 --> 00:35:01.350
But, you know, hopefully something will happen with that.
00:35:02.290 --> 00:35:06.530
I have one more thing. It might be a little bit political, So just tell me if
00:35:06.530 --> 00:35:07.610
you don't want to answer to this,
00:35:07.690 --> 00:35:14.270
but are maintainers now inclined to also review the Rust code or is it still
00:35:14.270 --> 00:35:20.730
okay to maybe not touch the Rust code whenever you make a patch to the C part of things,
00:35:20.930 --> 00:35:23.110
if you don't want to look at the Rust at all?
00:35:24.110 --> 00:35:27.550
There have always been some maintainers who are interested in the Rust code
00:35:27.550 --> 00:35:32.230
and have, you know, been curious to see, hey, what is this and how does it work?
00:35:33.840 --> 00:35:39.300
Remember, we also have 700 maintainers. So to classify everybody is all agreeing
00:35:39.300 --> 00:35:41.060
to do the same thing is kind of tough.
00:35:41.320 --> 00:35:47.060
We maybe have 150, 200 core maintainers. Even getting those people to agree is hard, right?
00:35:47.480 --> 00:35:52.460
So we all work differently. And the goal of the kernel, any change that comes
00:35:52.460 --> 00:35:53.720
in should never break the build.
00:35:53.960 --> 00:35:56.820
And if it breaks the build on the Rust side, we should be able to fix it up.
00:35:57.000 --> 00:36:00.840
I mean, so far, the things that have come in really haven't broken much.
00:36:01.040 --> 00:36:05.360
It hasn't been an issue. it really and it's part of that's one reason we agreed
00:36:05.360 --> 00:36:08.660
this experiment is over it looks like this is actually working because they
00:36:08.660 --> 00:36:12.620
put the work in in the past couple years we haven't had a really big roadblocks
00:36:12.620 --> 00:36:15.720
and things seem to be working okay and let's try it.
00:36:15.720 --> 00:36:22.140
But if i send in a patch to the c side and it breaks things on the rust side
00:36:22.140 --> 00:36:25.200
is is there any discussion about that or.
00:36:25.200 --> 00:36:27.800
It depends on the depends on the maintainer if i
00:36:27.800 --> 00:36:30.820
even if i notice it i'm like hey you broke that can you fix that so
00:36:30.820 --> 00:36:33.600
our build bot might just smack you
00:36:33.600 --> 00:36:36.780
down on the on the mail on the thing saying you broke the bill so you can't
00:36:36.780 --> 00:36:39.920
do that so like if you break the bill automatically i'm like okay come on let's
00:36:39.920 --> 00:36:43.740
figure this out if you say i don't know how to do the rust side great i'll offer
00:36:43.740 --> 00:36:47.480
help and things like that i don't think we've ever nobody's ever turned down
00:36:47.480 --> 00:36:50.840
help or things like that a lot of people like oh cool i've wanted to try this
00:36:50.840 --> 00:36:51.900
rust thing let me try this out.
00:36:53.300 --> 00:36:58.880
Ah that's a very interesting way to solve it because you take away this objectiveness
00:36:58.880 --> 00:37:03.620
and you make it objective by making it a technical problem you failed the build
00:37:03.620 --> 00:37:09.540
doesn't mean you made a mistake or you should have been more careful or why
00:37:09.540 --> 00:37:10.800
didn't you look at the rust part.
00:37:10.800 --> 00:37:13.520
You just say we all make mistakes right and we
00:37:13.520 --> 00:37:16.480
have checklists and that's why we have bots that run on the thing to say hey
00:37:16.480 --> 00:37:20.220
you did this wrong that's why we have like some subsystems have ai tools that
00:37:20.220 --> 00:37:23.180
are running on patch reviews and say hey that looks good but what about this
00:37:23.180 --> 00:37:26.080
what about this what about this it's like back to my very first patch i submitted
00:37:26.080 --> 00:37:29.140
i'm like oh i got all those things wrong great That's the technical thing I
00:37:29.140 --> 00:37:30.940
got wrong. I'll learn from this and move on.
00:37:31.850 --> 00:37:35.830
My next submission go. On average, it takes at least three tries to get a patch
00:37:35.830 --> 00:37:36.910
accepted into the kernel.
00:37:37.110 --> 00:37:41.810
And we're still running a huge, almost 10 changes an hour for the past 20, 15 years.
00:37:42.390 --> 00:37:46.370
Our rate of change is huge. And our rate of actually submissions and numbers
00:37:46.370 --> 00:37:48.730
of times that have to get accepted is a lot.
00:37:48.910 --> 00:37:52.170
That's normal. That's just how things go. I've had some patch series that's
00:37:52.170 --> 00:37:56.370
taken like 25 times to get a patch series in mergeable state.
00:37:56.530 --> 00:38:00.230
And that's just normal. On the C side, I don't think we've seen anything that's
00:38:00.230 --> 00:38:01.490
really broken the Rust side yet.
00:38:01.850 --> 00:38:03.410
What are our Clippy settings?
00:38:04.430 --> 00:38:06.570
I don't know. They're there.
00:38:06.690 --> 00:38:08.010
Do we use Clippy in the kernel?
00:38:08.250 --> 00:38:13.150
Yes, yeah, and they keep breaking. But as a new version of the Clippy, it's fine.
00:38:13.250 --> 00:38:16.930
And then the backport, the stable kernels, and I take patches that will fix those up.
00:38:17.130 --> 00:38:18.970
Is it already set to pedantic mode?
00:38:19.590 --> 00:38:20.390
I don't know.
00:38:21.050 --> 00:38:25.390
No, it's not set up. So we enable some set of things, right? And we do enable the...
00:38:26.130 --> 00:38:29.230
We don't enable the pedantic group, but we do enable the...
00:38:31.850 --> 00:38:38.450
Group of lints. And so when Rust compiler or Clippy adds a new lint in the default
00:38:38.450 --> 00:38:42.890
group, then suddenly we have to go update our code because we want it to be lint-free.
00:38:44.750 --> 00:38:48.530
And of course, as Craig said, then we have to backport the fix as well because
00:38:48.530 --> 00:38:53.630
we also want the code on the stable releases to be lint-free ideally.
00:38:54.670 --> 00:38:56.870
But yes, so that's a yes on Clippy.
00:38:58.170 --> 00:39:06.890
Isn't it ironic because C developers also, by osmosis, use Clippy now.
00:39:07.150 --> 00:39:13.050
If they break the build and maybe the Rust side broke and Rust uses Clippy to
00:39:13.050 --> 00:39:19.090
just enforce certain rules, also just the Rust compiler being stronger or maybe detecting more bugs,
00:39:19.730 --> 00:39:23.550
does that mean the C code also profits from that ecosystem?
00:39:23.810 --> 00:39:26.990
Does it mean the C code over time gets better? because there's more and more
00:39:26.990 --> 00:39:30.030
edge cases which get uncovered by the Rust tooling?
00:39:31.670 --> 00:39:36.530
Maybe, but we've had static code analysis in the kernel for decades.
00:39:36.870 --> 00:39:40.590
And we've fixed up things based on rules and we make sure we don't break the rules.
00:39:40.710 --> 00:39:43.770
The new version of compilers come out and they find more patterns and we fix
00:39:43.770 --> 00:39:45.050
those again, backboard those.
00:39:45.390 --> 00:39:49.870
So we're used to this. This is good code hygiene. We have no objection to this at all.
00:39:50.070 --> 00:39:54.470
So if another tool comes along and says, need to fix this up in the C code, great, we'll fix it up.
00:39:54.470 --> 00:40:00.190
I don't think clippy is so revolutionary like they have lintz in the c code too i.
00:40:00.190 --> 00:40:01.950
Think the thing that's more interesting.
00:40:01.950 --> 00:40:02.970
Is the rust format.
00:40:02.970 --> 00:40:04.410
Because on.
00:40:04.410 --> 00:40:06.250
The c side the formatting is.
00:40:06.250 --> 00:40:11.410
Very loose we have rules yes we do but they're guidelines and are not as strict
00:40:11.410 --> 00:40:16.230
as the as the Rust i but i like that but we i mean a code formatting is just
00:40:16.230 --> 00:40:20.070
there to make things universal and pattern matching you just want to have it
00:40:20.070 --> 00:40:23.450
consistent not that you agree with it or whatnot but that's why we have a code style.
00:40:23.890 --> 00:40:25.970
And now we have a Rust code style. Great, we'll follow that.
00:40:26.350 --> 00:40:27.610
Wonderful, not a big deal.
00:40:28.110 --> 00:40:30.070
Yep, just run us through the formatter.
00:40:30.650 --> 00:40:34.890
Do you use the default Rust format settings? Will you make any changes to it?
00:40:35.130 --> 00:40:38.950
I think there are some very minor changes to the settings, but almost.
00:40:40.450 --> 00:40:41.850
Is there even a C formatter?
00:40:42.570 --> 00:40:44.950
Yeah. It's been around for decades.
00:40:45.290 --> 00:40:48.210
50 years. But it's harder to introduce it than to start with one.
00:40:48.310 --> 00:40:52.510
We've had it. We've followed the rules. I gave a talk 25 years.
00:40:52.650 --> 00:40:55.470
One of my first talks I ever gave in a public thing was, here's our coding style,
00:40:55.530 --> 00:40:59.050
and here's why we have a coding style, and we abide by this coding style.
00:40:59.590 --> 00:41:03.350
Yeah, we've had that. And we've had, there's a bad little script in the kernel
00:41:03.350 --> 00:41:06.410
that will actually reformat your code to the coding style if you really want to run it.
00:41:07.470 --> 00:41:09.290
But it's been around for a long time.
00:41:09.290 --> 00:41:12.550
So it is used across all subsystems.
00:41:12.630 --> 00:41:15.150
Yeah, yeah, yeah. Any new code coming in has to follow those rules.
00:41:15.370 --> 00:41:18.610
We have some really old code in the kernel that doesn't follow the rules,
00:41:18.630 --> 00:41:21.090
but that's fine. That's nobody's touching those drivers anyway.
00:41:21.350 --> 00:41:26.250
But all new stuff, it's been, again, this is 25, 30 years we've been doing this.
00:41:26.470 --> 00:41:28.610
Yeah, that's not an argument here.
00:41:29.630 --> 00:41:33.590
It's not like we copied the C code style over to Rust. Like we're not using tabs.
00:41:34.010 --> 00:41:38.330
Yeah, I wish. That's fine. Our editors just work. It's okay.
00:41:38.930 --> 00:41:43.590
These are common standards for both the kernel side and the Rust side and we
00:41:43.590 --> 00:41:45.050
just live with them. It's all good.
00:41:45.450 --> 00:41:51.590
I remember that there was a Linux tool which was very impressive. It found...
00:41:52.730 --> 00:41:58.790
Code semantically it wasn't a grab it was more of a semantic code search.
00:41:58.790 --> 00:42:04.450
Coconut yeah exactly so yes that's what got me to move to europe wait.
00:42:04.450 --> 00:42:05.710
What that's a very.
00:42:05.710 --> 00:42:10.230
Interesting turn of events so julia luau who's a professor in paris has a tool
00:42:10.230 --> 00:42:14.430
called came up with a tool called coconut and it's a semantic tool that does
00:42:14.430 --> 00:42:19.470
code transformations and you can write code and rules and it'll, it'll fix problem.
00:42:19.610 --> 00:42:23.770
It'll transform the code based on these rules and do that long,
00:42:23.790 --> 00:42:27.950
long time ago. So you realize bugs are follow a common pattern, right?
00:42:28.130 --> 00:42:32.030
And if you define the pattern, like saying, oh, we forgot to check the error
00:42:32.030 --> 00:42:36.010
value of this function, always check that and fix it up. So we do it right.
00:42:36.150 --> 00:42:40.450
And then we have these set of rules that you can run them and make sure any
00:42:40.450 --> 00:42:42.410
new code coming in doesn't break these rules.
00:42:42.570 --> 00:42:46.290
So her work has fixed more security bugs in the kernel than anybody else.
00:42:47.300 --> 00:42:49.720
Fixed them, got them out of the code, and it prevents them from coming back
00:42:49.720 --> 00:42:52.380
in. Now, it turns out we haven't been running some of those recently,
00:42:52.380 --> 00:42:53.900
so some of those bugs are creeping back in.
00:42:54.360 --> 00:42:57.200
But there's also a graduate student here. I saw him at the conference.
00:42:57.320 --> 00:42:58.520
He's working on that tool for Rust.
00:42:58.940 --> 00:43:01.660
He's presented at the Rust conferences a few times, and he's doing some really
00:43:01.660 --> 00:43:05.300
good work at Vrye University in Amsterdam here on that.
00:43:05.400 --> 00:43:08.640
So that tool is also available for Rust for code transformations of Rust code
00:43:08.640 --> 00:43:10.040
based on semantic rules.
00:43:10.440 --> 00:43:13.860
And that's been around for a long time, and that's a good thing to be doing for Rust.
00:43:13.960 --> 00:43:17.300
And that's all these LLMs do as well. you can say look at this pattern over
00:43:17.300 --> 00:43:20.600
here is this pattern match over there and we've been doing that for the stable
00:43:20.600 --> 00:43:25.560
kernels for decades again we've fallen all the bug fixes that we didn't forget
00:43:25.560 --> 00:43:30.060
that we forgot to mark as bug fixes and now we know how to backport them just based on the semantic,
00:43:30.820 --> 00:43:35.320
analysis of the patch and the code it works well in rust now too any.
00:43:35.320 --> 00:43:43.520
Other tooling that people might not know that is used to yeah write rust code in the linux kernel So.
00:43:43.520 --> 00:43:47.800
There's a pretty cool project called KLint, which is short for kernelLint,
00:43:47.940 --> 00:43:53.080
which is a Rust compiler extension, kind of like how Clippy is a Rust compiler extension,
00:43:53.280 --> 00:44:00.720
but it's a custom run one written for the kernel, which checks some kernel specific things.
00:44:02.460 --> 00:44:05.940
One of the ones that I'm the most excited about, which is still a work in progress,
00:44:06.140 --> 00:44:08.480
but there's a lint for...
00:44:09.080 --> 00:44:12.720
So in some parts of the kernel code base you're not allowed to sleep.
00:44:13.820 --> 00:44:23.080
And then klint has this lint that checks in those pieces of code if you call something that sleep.
00:44:24.880 --> 00:44:31.100
And so basically it's a repository with custom clippy lints.
00:44:31.100 --> 00:44:36.300
Are these actually clippy lints or is that a separate system well.
00:44:36.300 --> 00:44:44.080
It's a it's a program compiler plug-in in the same way as how clippy is a compiler plug-in.
00:44:44.080 --> 00:44:52.500
Okay yeah i always find it interesting to cool yeah to to hook into the the
00:44:52.500 --> 00:44:55.320
rust kernel like that and extend it like this Because.
00:44:55.320 --> 00:45:01.040
It has to do a lot of pretty complex analysis where you run the Rust compiler,
00:45:01.040 --> 00:45:05.860
you get all the type information, and then based on that, you can then make the lint, right?
00:45:05.980 --> 00:45:10.840
You can't just do this check based on, oh, this word is here.
00:45:11.040 --> 00:45:14.900
It requires much more complex analysis. You need to run the compiler to get
00:45:14.900 --> 00:45:18.680
the information you need to do the lint. So that's why it's a compiler plugin.
00:45:19.360 --> 00:45:23.960
So we have to come to the end. And I thought about the final question for a
00:45:23.960 --> 00:45:27.400
long time. Typically, our final question is, what's your message to the Rust community?
00:45:27.660 --> 00:45:31.160
But I won't let you off the hook today that easily.
00:45:31.600 --> 00:45:37.100
So I will try to reframe it. I will rephrase it. And I have two questions.
00:45:37.240 --> 00:45:38.640
In fact, I have one for each of you.
00:45:39.520 --> 00:45:43.820
It's the same question, but just with a twist. So who wants to start?
00:45:45.880 --> 00:45:46.320
Okay.
00:45:48.440 --> 00:45:51.520
Alice, what can Rust developers learn from C developers?
00:45:57.620 --> 00:45:58.220
Empathy,
00:46:01.360 --> 00:46:05.140
what can Rust developers learn from C developers.
00:46:05.140 --> 00:46:07.480
I want to hear this it's.
00:46:07.480 --> 00:46:08.480
A bit of a curveball.
00:46:10.160 --> 00:46:11.520
That's a hard question.
00:46:13.260 --> 00:46:14.720
And your question will be the,
00:46:19.140 --> 00:46:21.640
if you like you can also answer first.
00:46:22.960 --> 00:46:24.540
Whoever has to answer first.
00:46:24.540 --> 00:46:27.020
Put you on the spot there yeah.
00:46:27.020 --> 00:46:27.560
You really did.
00:46:27.560 --> 00:46:31.360
So i i can answer so rust rust
00:46:31.360 --> 00:46:37.340
free so when switching from c to rust and back to c i'm like oh no and c i gotta
00:46:37.340 --> 00:46:41.660
okay was this pointer doing who owned this pointer who did this who where did
00:46:41.660 --> 00:46:46.500
this go and i have to like mentally think a lot more so many years ago a friend
00:46:46.500 --> 00:46:48.900
of mine said you gotta try this rust language. It's really fun.
00:46:49.120 --> 00:46:50.140
It makes programming fun again.
00:46:50.460 --> 00:46:52.780
It's like, nah, it's fine. I'm fine with C.
00:46:53.180 --> 00:46:56.700
And I should have taken him up on it. Many years later, he was right.
00:46:57.600 --> 00:47:02.400
Rust to me makes programming fun again. So it makes, it takes the,
00:47:02.560 --> 00:47:07.100
I know what logic and what I want to express and how I want to get something done.
00:47:07.320 --> 00:47:11.600
And Rust provides me with the tools to know that I'm doing so in a way that
00:47:11.600 --> 00:47:14.200
I'm not doing all the stupid things I do in C wrong.
00:47:14.360 --> 00:47:18.380
And I don't have to keep track of that it's like okay it compiles logic looks
00:47:18.380 --> 00:47:21.680
right okay now i can worry about is my logic right i'm worried about all this
00:47:21.680 --> 00:47:26.880
other meta stuff i have to keep in my head as a c programmer so my thing is
00:47:26.880 --> 00:47:30.400
so from a c programming point of view learning from the rust developers is hey
00:47:30.400 --> 00:47:32.800
thanks for making programming fun again really appreciate it,
00:47:43.620 --> 00:47:44.140
you.
00:47:44.140 --> 00:47:45.420
Can say the same thing.
00:47:45.420 --> 00:47:46.700
You can say working.
00:47:46.700 --> 00:47:48.080
With c has made.
00:47:48.080 --> 00:47:48.940
Rust fun again.
00:47:50.840 --> 00:47:51.900
Programming fun again.
00:47:51.900 --> 00:47:56.020
I mean c is pretty fun right you got to do a,
00:47:57.160 --> 00:48:00.540
You know, you get to set up all these complex things and keep it all in your
00:48:00.540 --> 00:48:04.700
head. But I don't really know if I want to learn from that.
00:48:07.520 --> 00:48:13.500
I think part of it is also maintaining a project like Linux for the better half of three decades.
00:48:14.360 --> 00:48:18.140
Is that the way to say that? Anyhow, you know what I mean. For a long time.
00:48:18.960 --> 00:48:22.980
Because you go through multiple different phases of, you know,
00:48:23.100 --> 00:48:26.200
different trends, people using different paradigms and
00:48:26.200 --> 00:48:29.300
different tooling and yeah generations of
00:48:29.300 --> 00:48:32.360
developers essentially and how do
00:48:32.360 --> 00:48:35.100
you make that work like the long-term vision of
00:48:35.100 --> 00:48:38.360
such a project that's the most impressive thing when i
00:48:38.360 --> 00:48:41.220
think about the linux kernel no matter the language really
00:48:41.220 --> 00:48:45.480
maybe in 30 years from now it might be a different language that gets added
00:48:45.480 --> 00:48:50.660
but what won't change is how you think about the project from an engineering
00:48:50.660 --> 00:48:56.100
perspective and you also think about the users So I think I can speak for all
00:48:56.100 --> 00:49:00.380
of us when I say thanks for both of your very important work,
00:49:00.540 --> 00:49:04.400
making the world a better place. Thanks. Thank you very much.
00:49:12.820 --> 00:49:16.720
Rust in Production is a podcast by Corrode. It is hosted by me,
00:49:17.000 --> 00:49:20.820
Matthias Endler, and produced by Simon Brüggen. For show notes,
00:49:21.000 --> 00:49:25.000
transcripts, and to learn more about how we can help your company make the most
00:49:25.000 --> 00:49:27.100
of Rust, visit corrode.dev.
00:49:27.280 --> 00:49:29.720
Thanks for listening to Rust in Production.