WEBVTT
00:00:00.000 -->
00:00:01.364It's 2026.
00:00:01.375 -->
00:00:04.278It's like the most scrutinized C code on the planet.
00:00:04.278 -->
00:00:07.068All of like the low hanging fruit, memory, corruption, and open.
00:00:07.578 -->
00:00:09.319An OpenSSL gone now, right?
00:00:09.319 -->
00:00:13.929Like how likely one today, so I was gonna say, like we, we were literally today.
00:00:19.112 -->
00:00:20.073Hello.
00:00:20.132 -->
00:00:22.382Welcome to Security Cryptography, whatever.
00:00:22.443 -->
00:00:23.373I'm Deirdre.
00:00:23.763 -->
00:00:24.393I am David.
00:00:25.092 -->
00:00:27.733I am a national spokesperson for the Crystal Hot Sauce company.
00:00:28.318 -->
00:00:28.858Awesome.
00:00:29.487 -->
00:00:30.448That's Thomas.
00:00:30.858 -->
00:00:31.998and we have two.
00:00:32.283 -->
00:00:35.323Special guest today we have returning champion, Alex Gaynor.
00:00:35.343 -->
00:00:36.182Hi Alex.
00:00:36.182 -->
00:00:36.302Hello.
00:00:36.393 -->
00:00:36.872Hello.
00:00:36.872 -->
00:00:38.207Thanks for having me a second time.
00:00:38.402 -->
00:00:39.843Yes, thanks for coming back.
00:00:39.843 -->
00:00:41.012We didn't scare you off.
00:00:41.103 -->
00:00:46.442And, uh, our other collaborator on the Python Cryptography Library, Paul Kehrer.
00:00:46.622 -->
00:00:47.463Hi Paul.
00:00:48.093 -->
00:00:48.593Hi everybody.
00:00:48.917 -->
00:00:50.628I'm just here on Alex's shirt tails.
00:00:51.003 -->
00:01:03.692Yes, we invited Paul and Alex today, on today because they wrote a little blog post about OpenSSL and they gave a talk about OpenSSL a couple of weeks ago, on the same topics.
00:01:03.753 -->
00:01:13.382And they have a lot of experience trying to operate around OpenSSL as maintainers of the, the Python Cryptography Library.
00:01:13.382 -->
00:01:16.173We say the, because it's literally called cryptography.
00:01:16.382 -->
00:01:23.013And so if you wanna redo anything with cryptography in Python, theirs is usually the one that you go for.
00:01:23.293 -->
00:01:24.132Alex.
00:01:24.832 -->
00:01:33.662Te tell us what prompted you to write this statement, which is also the first statement from the Python Cryptographic authority.
00:01:33.662 -->
00:01:34.472AKA you two.
00:01:34.832 -->
00:01:46.582Before you do that, can I ask one or both of you to explain to, you know, the world, why Pike cryptography is, you know, like why you guys have standing to make these complaints.
00:01:46.582 -->
00:01:47.572Like, who do you guys think you are?
00:01:49.453 -->
00:01:49.673Hi.
00:01:49.903 -->
00:01:50.192Sure.
00:01:50.272 -->
00:01:53.513I, I, I, I'll start with that and Alex can jump in as he sees, uh, necessary.
00:01:54.212 -->
00:01:57.772So, uh, the Python Cryptographic Authority is a self-proclaimed authority.
00:01:57.772 -->
00:02:08.432It's important to note that inside the context of Python, there, there used to be like, like a running gag where you would create a, an authority to define like the, basically the, the GitHub name space that you were going to use.
00:02:08.612 -->
00:02:16.592So the Python Packaging Authority, which became a very official concept in Python, was originally actually the creation of a single person who wanted to go and do some work.
00:02:16.992 -->
00:02:22.483similarly, the Python Cryptographic Authority was founded in kind of an aspirational manner back in 2013.
00:02:23.013 -->
00:02:31.723like Alex and I were working for the same employer, back then, and we looked around and discovered that there was not really a good solution for cryptography and Python.
00:02:31.992 -->
00:02:36.242In Alex's case, he was also interested in Pi, PI, support a thing that he regrets to this day.
00:02:36.543 -->
00:02:45.945Uh, but, uh, The outcome of that was ultimately that we, you know, we've embarked on a 13 year and counting, uh, adventure of taking over the, the, the world of cryptography and python.
00:02:46.205 -->
00:03:03.411and largely we've been successful, mostly because we have a, I mean, I'm sure we're gonna get into this, but we, we have a somewhat maniacal focus on the way in which we deliver the software and the way in which we, that the expectations we set for ourselves, such that we can and do have higher expectations for the things that we depend on.
00:03:03.860 -->
00:03:16.316Yeah, I can't emphasize enough how in like pre 13 years ago, so like 2011, if you tried to make an H-T-T-P-S connection from Python, it was effectively impossible.
00:03:16.756 -->
00:03:23.475you at best like had to install Py OpenSSL, which didn't have wheels, so it had to build OpenSSL from source every time.
00:03:23.656 -->
00:03:31.135And so it, and that, and of course, OpenSSL would fail to compile because it was 2011 so it was just bad.
00:03:31.360 -->
00:03:37.080And for those who don't ship a lot of python, libraries, wheels is a technical term of art.
00:03:37.170 -->
00:03:40.501It's not just ha ha wheels included, like batteries included.
00:03:40.830 -->
00:03:41.580What are wheels?
00:03:41.881 -->
00:03:42.061Yeah.
00:03:42.061 -->
00:03:45.431Wheels are, uh, python's, binary package artifacts.
00:03:45.431 -->
00:03:51.461So you can have an sist which is sourced, that you compile yourself, or a wheel that's pre-compiled by somebody else.
00:03:51.461 -->
00:03:57.670So we, we build wheels of cryptography for many, many operating systems and CPU architectures.
00:03:57.971 -->
00:03:59.980This is a, a language called Python.
00:03:59.980 -->
00:04:03.881Why isn't, why isn't it called like eggs or, or like, were called eggs.
00:04:04.300 -->
00:04:04.841oh God.
00:04:04.871 -->
00:04:05.411Damnit.
00:04:05.770 -->
00:04:06.100Okay.
00:04:06.100 -->
00:04:08.471And now they're called wheels because that makes sense.
00:04:08.531 -->
00:04:08.920All the, the.
00:04:08.996 -->
00:04:14.126Alex is the first person to be asked the same question twice on this podcast in two different episodes.
00:04:14.825 -->
00:04:17.315Yeah, you get your wheels at the cheese shop.
00:04:17.375 -->
00:04:20.906Which The cheese shop was the original name of the Python package index.
00:04:21.206 -->
00:04:21.495What?
00:04:21.846 -->
00:04:22.136Okay.
00:04:22.141 -->
00:04:22.670Okay.
00:04:22.925 -->
00:04:26.255It's cause Python was named after a Monty Pi after Monty Python.
00:04:26.255 -->
00:04:35.271And the Cheese Shop is a famous sketch, Now I'm remembering our whole episode that we, we did this whole rigamarole with like, you know, we've been running, going strong for several years now.
00:04:35.271 -->
00:04:37.940I, for, forgive me for forgetting, but, okay.
00:04:37.940 -->
00:04:46.226So, I was gonna, I was gonna say, right, so like the, the situation with like, like your biggest dependency is openness, L right.
00:04:46.901 -->
00:04:47.391correct.
00:04:47.442 -->
00:04:47.661Yes.
00:04:48.026 -->
00:04:48.146Yeah.
00:04:48.447 -->
00:05:09.521Right, and like in the universe of things that use OpenSSL, like that's gotten a lot more complicated over the last 10 years as Chrome has shifted to BoringSSL and AWS now uses their own kind of formally verified SSL and open BSD did LibreSSL, like in the universe of OpenSSL consumers, where do you guys fit in?
00:05:09.521 -->
00:05:10.271Like, what's your rank?
00:05:11.047 -->
00:05:23.466I, I don't know that there's an official leaderboard of like, OpenSSL consumers, but, I, I guess we have a lot of standing, at the very least in the Python ecosystem, we're consistently one of the most downloaded packages, on the Python package index.
00:05:23.646 -->
00:05:29.257Many millions of downloads, like if you use a Python thing, it's like quite likely you rely on us.
00:05:29.307 -->
00:05:33.396all, all of the major clouds command line interfaces include us.
00:05:33.697 -->
00:05:37.372because then like the, the cert bot, lets encrypt client includes us.
00:05:37.466 -->
00:05:37.757Yeah.
00:05:38.057 -->
00:05:44.391In the same sense as like if you were already a GO program and you use Net HTTP, you'd be pulling in go's, TLS libraries.
00:05:44.721 -->
00:06:00.232In Python world, if you're doing the equivalent of if you're doing requests or whatever, you're effectively to do TLS pulling you guys No T So TLS is the one thing that is, I think in the usual cryptography toolkit that you probably would not pull in, cryptography for, we don't have TLS APIs.
00:06:00.466 -->
00:06:09.612You're much more likely to either use the Python, standard library SSL module or PI OpenSSL, which, which we also maintain, but is a separate library.
00:06:09.911 -->
00:06:10.451Oh wow.
00:06:11.151 -->
00:06:19.161And just for the record, why aren't you just writing your Python cryptographic library in Python and shipping Python?
00:06:19.862 -->
00:06:28.326That would be a very convenient, unfortunately, it would have both security and performance implications that are, effectively insoluble inside Python, the language.
00:06:29.026 -->
00:06:45.547I, I will also say when we started cryptography, you know, 12 or 13 years ago, we made, put amounts to like a very ironic deal, which is like, I was persuaded that like I would help coordinate this library as long as the one thing we were doing was not implementing cryptography.
00:06:45.846 -->
00:06:48.007We wanted to think hard about APIs.
00:06:48.156 -->
00:06:52.896We wanted to think hard about what made a safe default and what shouldn't have a default at all.
00:06:53.016 -->
00:07:05.677We wanted to think hard about testing, but we were not trying to do something where we felt like the relevant expertise was like really outside of, you know, what a person might have if they didn't have like a PhD in cryptography.
00:07:06.112 -->
00:07:14.482and you're talking like low level primitives because like we can have a, a, a wonderful debate about what counts as implementing cryptography, but I, I know what you mean.
00:07:14.572 -->
00:07:14.632Yeah.
00:07:14.826 -->
00:07:20.826yeah, I, I would say about six months after we, you know, I had this conversation where we're not implementing cryptography.
00:07:21.156 -->
00:07:26.737One of our maintainers at the time, I guess this would've been right after HeartBleed, came up to me and said, we should implement TLS.
00:07:26.737 -->
00:07:28.776And I said, what happened to our deal?
00:07:28.867 -->
00:07:31.447And he said, TLS is a network protocol.
00:07:31.447 -->
00:07:32.826It's not cryptography.
00:07:33.127 -->
00:07:33.312Uhhuh.
00:07:33.576 -->
00:07:38.646Uh, and I think that's one of the, and I think it's like a very true statement that like really rides the line.
00:07:38.807 -->
00:07:39.096Yeah.
00:07:39.641 -->
00:07:43.331Yeah, and we've enclo we've definitely increasingly blurred it over time.
00:07:43.331 -->
00:07:53.252I mean, I think one of the first times we kind of said, oh, this isn't really cryptography, was when we did like a AES key wrap support, back in the day when like OpenSSLs implementations of it weren't very good.
00:07:53.601 -->
00:08:01.922and like we still strive not to do it unless we have to, but there are scenarios where we choose to hoist it into ourselves because we believe we can do it better.
00:08:02.232 -->
00:08:07.391I think the most recent example of that is actually something that we haven't released yet, which is the, uh, HPKE support.
00:08:07.672 -->
00:08:13.072OpenSSL does support HPKE, but we have chosen to implement it ourselves using OpenSSL as primitive.
00:08:13.771 -->
00:08:16.021And why did you choose to do that?
00:08:16.721 -->
00:08:19.331So I think there were a, a couple motivations to come into it.
00:08:19.331 -->
00:08:25.812One, it means we're, you're gonna get a really consistent HPKE implementation no matter where you get your cryptography from.
00:08:25.812 -->
00:08:35.552So like the OpenSSL HPKE implementation is not supported on LibresSL BoringSSL, aws-lc, which I, I guess we'll probably mention a lot in this, conversation.
00:08:35.552 -->
00:08:36.721So I'll just call the forks.
00:08:36.751 -->
00:08:42.461So like, implementing it ourselves means you'll get a consistent experience 'cause they'll all have the underlying cryptography.
00:08:42.981 -->
00:08:47.751it means we have a lot more control over kind of the compatibility surface.
00:08:47.841 -->
00:08:55.042We're not accidentally pulling in lax parsing behavior or other kind of unintended behavior from OpenSSL and.
00:08:55.432 -->
00:09:04.552It gives us the kind of high level of confidence that we're, we're handling all the edge cases from both the security and correctness and just like not crashing perspective.
00:09:04.611 -->
00:09:06.336Paul, I don't know if there were more motivation for you.
00:09:07.037 -->
00:09:19.517Well, I would say that the, the, the one that's kind of the elephant in the room, that's the, the core component of, part of our criticism that ac occurred in our statement is that the HPKE APIs are only accessible through the OSSL_PARAM api.
00:09:20.216 -->
00:09:21.147So, okay.
00:09:21.417 -->
00:09:22.917I wanna stop you guys there, right?
00:09:22.947 -->
00:09:32.336So I, I, so we're kind of beating around the bush and like, I think trying to like let you guys introduce the, the, the talk that you guys did yourself, but I kind of don't wanna do that.
00:09:32.336 -->
00:09:34.197Let's just like cat outta the bag.
00:09:34.376 -->
00:09:46.167You guys gave, you're, you're not super happy with the current state of OpenSSL but it's like, not for the normal reason that people usually bring up, like, you know, that's why there's Libra SL 'cause we don't tRust the code or whatever.
00:09:46.167 -->
00:09:49.496Although we can get into that later too, but for other reasons.
00:09:49.677 -->
00:09:50.067Right.
00:09:50.307 -->
00:09:57.287So, and your, your reasoning is mostly about, or is entirely about the design of OpenSSL 3.
00:09:57.761 -->
00:09:59.231The, that version of the library.
00:09:59.772 -->
00:10:05.772So I have not paid close enough attention to OpenSSL to know what the fuck OpenSSL 3 is.
00:10:06.042 -->
00:10:07.782What's roughly like the timeline here?
00:10:07.782 -->
00:10:08.562What happened?
00:10:08.861 -->
00:10:12.662Like I knew OpenSSL back on the Heartbleed days, and now there's multiple OpenSSLs.
00:10:13.121 -->
00:10:26.532So, you know, Heartbleed happened in 2013, and at that time it became like a very well known fact that OpenSSL was kind of a project on life support that didn't have the right resourcing and had various problems because of it.
00:10:27.011 -->
00:10:33.072Uh, in the wake of that OpenSSL got a surge of investment, both monetarily and also in, in, uh, human time.
00:10:33.341 -->
00:10:38.052Like there were a bunch of folks from Google, like Emelia Kaspar and others who went over there and did a bunch of work.
00:10:38.292 -->
00:10:47.471Um, there were folks who, who later founded some of the forks that were involved heavily, and then there was just like a large organization formed that had a bunch of full-time employees to do things.
00:10:48.042 -->
00:10:57.782As a component of that, the business realized that, part of the way in which you sustain in the cryptographic library is by catering to business interests around things like FIPS.
00:10:58.471 -->
00:11:02.491The structure of FIPS is such that you want to isolate it off and that.
00:11:02.927 -->
00:11:06.856Created a concept that they were, uh, that was supposed to replace engines.
00:11:07.067 -->
00:11:09.197Uh, engines are a method of plugging things into OpenSSL.
00:11:09.886 -->
00:11:11.447Providers became the new thing.
00:11:11.746 -->
00:11:14.626Providers are intended to be a superset that can do all that stuff.
00:11:14.687 -->
00:11:17.086And in the abstract it's a fine concept.
00:11:17.476 -->
00:11:30.647However, in practice, as they went down the path of this OpenSSL 3, which was an intent, a deliberate, um, ABI breakage, they skipped 2 because they were worried about version numbering based on the fact that Libre had forked themselves and they called themselves 2 at the time.
00:11:31.136 -->
00:11:41.726they went to three and the consequence, like the actions of three were basically rewrite the entire internals were without understanding the surface area of your own project.
00:11:42.116 -->
00:11:51.356Uh, this led to an 18 month alpha beta phase before release, and ultimately well over two years of delays from when they expected to be able to release it.
00:11:51.746 -->
00:11:55.767Uh, did that, when did that land, like when did OpenSSL 3 become?
00:11:56.466 -->
00:11:59.976I assume that right now OpenSSL 3 is like the mainstream version everyone's using.
00:12:00.677 -->
00:12:09.777Yeah, if you get your OpenSSL from like a Ubuntu or Debian, a Red Hat, you're gonna get an OpenSSL 3 from like a recent Linux distribution and then that landed in 2021.
00:12:10.777 -->
00:12:11.976Okay, that makes sense to me.
00:12:11.976 -->
00:12:22.006So now I know what you're about to say, but let's bring our audience up to speed on the suite of concerns that you guys have about the situation.
00:12:22.706 -->
00:12:22.917Yeah.
00:12:22.976 -->
00:12:27.331So we've got a couple, and I'm gonna say them in the order that we wrote the post in.
00:12:28.032 -->
00:12:30.792as we'll get into, I think we put them in the wrong order in the post.
00:12:31.011 -->
00:12:35.152so the, the first concern, it's, it's the most easy to quantify is performance.
00:12:35.481 -->
00:12:40.881Uh, OpenSSL 3 had some really, really significant performance regressions.
00:12:41.062 -->
00:12:47.101Things like loading, elliptic curve, public key from like a subject public key info format.
00:12:47.131 -->
00:12:49.381Really simple format to parse.
00:12:49.471 -->
00:12:58.616Got something like, eight times slower between OpenSSL 1, 1 1 and OpenSSL 3 Was it doing the, on curve math?
00:12:58.616 -->
00:13:00.552Was it checking that No, no.
00:13:00.552 -->
00:13:02.017It like, yeah.
00:13:02.096 -->
00:13:02.201Yeah.
00:13:02.201 -->
00:13:08.322It's like anything you can come up with it that's like the cryptography, the math that might make this time is not it.
00:13:08.442 -->
00:13:30.261It was that the abstraction for how providers worked and like the interplay between it and the DER parser and just like what the public, a pi, like, all that back and forth had so many indirect calls, so much allocation, so much locking, OpenSSL had a whole like format, auto detection thing that happened.
00:13:30.261 -->
00:13:36.871It was like, you know, you can find the issues in the OpenSSL bug tracker where folks are like breaking down, like, where is the time going?
00:13:36.871 -->
00:13:43.371And it's, it's truly just that the, the parsing itself had become so convoluted, that, that it was just.
00:13:43.701 -->
00:13:46.341You know, it time was spent in nonsense places.
00:13:46.522 -->
00:13:54.381we'll say since then, OpenSSL has made some improvements and now it is only three times slower than it used to be as of, the last time I measured.
00:13:54.591 -->
00:13:56.241So, like, that's, that's quite significant.
00:13:56.241 -->
00:14:00.532Like, I, I wanna just give people like a data point for like how extreme this was.
00:14:00.772 -->
00:14:25.782We, we have, our own X.509, like path validation code, what people would call like X.509 verification, and, doing our own public key parsing that is moving the public key parsing from OpenSSL to our own Rust code, no other changes was a 60% performance improvement on end-to-end X.509 validation, like that, that's just like how extreme this overhead was to like, what is empirically possible.
00:14:25.871 -->
00:14:30.331So this was kind of the first and most easily quantifiable, the performance, was insane.
00:14:30.331 -->
00:14:30.871And I, I think.
00:14:31.572 -->
00:14:32.591Well, maybe we get into this more.
00:14:32.591 -->
00:14:37.662A lot of people think, think that this is like maybe our biggest complaint'cause it's the easiest to quantify.
00:14:37.961 -->
00:14:40.631But, but it, it's not, it's, we shouldn't have put it first.
00:14:40.662 -->
00:14:45.701'cause our real complaint is, is the complexity that led to the performance regressions.
00:14:45.701 -->
00:14:53.951The fact that the provider APIs were designed or evolved in such a way that like the abstraction boundaries were unclear.
00:14:54.282 -->
00:14:55.371And, you, you had.
00:14:56.072 -->
00:15:11.731Really extreme performance regressions that came down to things like, if you load a hash algorithm, through like a function like EVP SHA-256, which is like get the SHA-256 like hash, I, object identifier like that is slower.
00:15:11.851 -->
00:15:24.121'cause what that API is doing is getting like an object that represents like a future promise that I will call the provider API later to like actually find the SHA-256 API, from your provider.
00:15:24.121 -->
00:15:26.282And like, that can change at any time in theory.
00:15:26.461 -->
00:15:32.461'cause like the provider APIs don't like say like, you know, once your program's initialized, like you can't change where the cryptography comes from.
00:15:32.701 -->
00:15:40.532So there's just like tons of back and forth, lots of indirect calls, lots of allocations, lots of locking, lots of caches to try to compensate for that.
00:15:40.532 -->
00:15:47.162Like the degree of complexity in the internals got like really extreme, but it also got pretty extreme in the public APIs.
00:15:47.511 -->
00:16:04.072Yeah, I mean, I, I won't add a lot here other than to say like the, the ultimate, uh, outcome of these like attempted fixes for performance progressions based on this, is that like we, we have like full on RCU code, which has had bugs in OpenSSL since they landed it to just to try and resolve some of these issues.
00:16:04.351 -->
00:16:09.032and like the nature of the DER parsers are actually that it chains things together.
00:16:09.152 -->
00:16:20.562Like it'll, it'll progressively try different things, uh, which led to a bunch of bugs DERing the OpenSSL 3 betas where they were leaving errors on the error stack because they, they didn't know they were doing these chain, chain attempts to parse things.
00:16:21.187 -->
00:16:22.866What they were doing different.
00:16:23.317 -->
00:16:35.287I like different implementations of dear person and they would like try one, and then if it didn't throw, we would try another So, so OpenSSL has like these auto key loader APIs where it's like, give us some DER and we'll figure it out and give you back what you want.
00:16:35.506 -->
00:16:42.797but that like necessarily means that it's going to try a bunch of things rather than simply reading the, the, the o out of it and then dispatching correctly.
00:16:43.067 -->
00:16:47.606so you get a bunch of performance issues from that was that, I have two questions, right?
00:16:47.606 -->
00:16:50.366So first of all was the, kind of the YOLO mode.
00:16:50.366 -->
00:16:52.976Just try all the different formats, key parsing thing.
00:16:52.976 -->
00:16:56.731Was that an OpenSSL one thing or did they come up with that in, in OpenSSL 3.
00:16:56.792 -->
00:17:04.376one did have the same pri uh, behavior on some limited APIs, like it's like D two i auto private key or something of that nature.
00:17:04.616 -->
00:17:12.356Uh, so that, that behavior was there, but it was a much faster one in that it dispatched based on the OID at the time, as opposed to this one, which needs the providers.
00:17:12.626 -->
00:17:17.426And like every provider doesn't, there's not like a mechanism for providers to necessarily, um.
00:17:18.057 -->
00:17:20.037Like necessarily declare exactly what they support.
00:17:20.037 -->
00:17:22.196So you have to just incrementally try them until one works.
00:17:22.896 -->
00:17:28.057I'm, I'm a little familiar with the OpenSSL code or the OpenSSL code of Old, right.
00:17:28.297 -->
00:17:32.076I don't remember Concurrency being a huge thing in that code base.
00:17:32.557 -->
00:17:39.787So you, you said that there's like a full line implementation of RCUs now in s like what is the, you guys would know way better than we would.
00:17:39.997 -->
00:17:41.557What is the concurrency situation?
00:17:41.557 -->
00:17:43.237This, if this is a, a crypto library.
00:17:43.936 -->
00:17:53.557Yeah, it's a crypto library, but like you have like various APIs for manipulating like global states, like, adding a new provider to like the global context for example.
00:17:53.856 -->
00:18:02.511And so like if you haven't done anything to preclude like that happening at arbitrary points in the runtime, wait, wait, wait, wait, wait, wait, wait.
00:18:02.692 -->
00:18:08.646Why am I, why am I adding new providers to the runtime State of a running program, I mean, that's roughly the question.
00:18:08.646 -->
00:18:12.787Like Paul and I would ask, like we think that's just like a design mistake that you can support that.
00:18:12.787 -->
00:18:21.396But like if you've chosen to support that and are now like going back to like, you know, you don't, you're not quite sure like what your API contract with users is.
00:18:21.396 -->
00:18:25.217Like, and not now you have to like deal with like concurrency bugs.
00:18:25.217 -->
00:18:29.297People are reporting like tsan issues or like performance issues.
00:18:29.297 -->
00:18:33.842Like not, not like, yeah, like, hold on, hold on, hold on.
00:18:34.021 -->
00:18:50.882Rcu are what you use when you're locking so much and so many hot paths that you can't actually do mutexes, that you actually need an optimized concurrency primitive because you're, I, I get like, you need to lock because you can add a provider, can you like add and remove providers in a hot path?
00:18:51.582 -->
00:18:53.271Nothing precludes you from doing that.
00:18:53.971 -->
00:18:54.261Yeah.
00:18:54.267 -->
00:18:54.557Yeah.
00:18:54.767 -->
00:18:58.547There, there's no, there was, there was not necessarily like, I, I wanna be charitable here.
00:18:58.547 -->
00:19:11.326So like, maybe the, the answer may not be that they didn't have a design ethos around it, but like, for whatever reason, as they went down the path of implementing this, the answer of support, everything at all times became the correct answer.
00:19:11.596 -->
00:19:14.537And like that led inexorably to these types of choices.
00:19:14.896 -->
00:19:19.277Because, because engines in OpenSSL that was there for like people doing like crypto card stuff.
00:19:19.277 -->
00:19:19.696Right.
00:19:20.267 -->
00:19:23.356And at the time it was for like people with co-processor accelerators.
00:19:23.356 -->
00:19:23.777Right?
00:19:23.807 -->
00:19:30.287So the idea here is like you're supporting a use case where somebody has like, you know, a card or something like that.
00:19:30.317 -->
00:19:35.321They're inserting and removing like several times per millisecond is why you would need rcu.
00:19:35.622 -->
00:19:39.362I, I think the, I mean the RCU component is just the, the component.
00:19:39.422 -->
00:20:01.092What comes out of the fact that they have to do locking in so many places just to check the, the actual reality is like they, as best I can understand it, again, steel manning the concepts of the provider, there is a design goal in OpenSSL that it is an abstracted substrate upon which anyone can do anything, including adding features that were never considered by the OpenSSL people.
00:20:01.692 -->
00:20:04.362Like basically linked into any arbitrary program, right?
00:20:04.362 -->
00:20:12.041So like, oh, in the future, 10 years from now, I can load arbitrary providers into some old piece of, uh, kit and get new stuff.
00:20:12.582 -->
00:20:18.521Yeah, and like you do see examples this like before OpenSSL had post quantum crypto algorithms in it.
00:20:18.791 -->
00:20:22.541There were third party providers that would provide them.
00:20:22.751 -->
00:20:26.892And so like you could get an ML KEM inside OpenSSL before OpenSSL supported it.
00:20:27.592 -->
00:20:36.311Like I, I think we would say like, this is just like not the correct allocation of resources to like towards your problems, but it, it makes me so nervous.
00:20:36.311 -->
00:20:39.372Like I, you know, I don't have, I don't maintain a project like this.
00:20:39.372 -->
00:20:44.692Like I've maintained large Rust projects, but like, no, like, is like, lib.
00:20:44.692 -->
00:20:46.551SDL works kind of like this.
00:20:47.182 -->
00:21:03.382It's designed to both be statically, which it's a, it's a like cross platform graphics library that's designed to be statically linked, but also like, sometimes it gets swapped out and replaced with a different one because, you know, maybe valve like made it run on Linux, but wants it to look like it's windows to trick some game from 10 years ago to run on Linux.
00:21:03.711 -->
00:21:05.571And you can kind of see how you end up there.
00:21:05.751 -->
00:21:19.662I don't quite understand what the, like surrounding ecosystem is where you need to do this for cryptography, um, as opposed to like games which are, I was about to say notorious for being developed once and then left on for years.
00:21:19.662 -->
00:21:21.731But then I just realized I was describing HSMs.
00:21:22.031 -->
00:21:35.096Yeah, I mean, I, I think what I would say is that like if you really sat down with a, we want to make more things pluggable, like I think there is probably a design you could get for providers where you put state in the right places.
00:21:35.096 -->
00:21:45.326You have like the indirect function call in kind of the right place in the stack that like balances the complexity and the performance and the, like, maintainability of the system.
00:21:45.326 -->
00:22:04.346Like I, you know, I, I think there are useful points on the like, trade off curve that are not just like everything is static, but I, I think the point OpenSSL ended up on where, like the SHA-256 or like the result of EVP SHA-256 is like an abstract object which will like call something on the provider later.
00:22:04.346 -->
00:22:10.277And that's allowed to change at like arbitrary points is like not a particularly useful point on that trade off Yeah.
00:22:10.616 -->
00:22:11.356No, God.
00:22:11.696 -->
00:22:14.666I was, I was, I was set off on the whole RCU thing.
00:22:14.666 -->
00:22:23.277So I was wondering, it kind of made sense to me that if they did a whole bunch of new concurrency stuff, like that's a way I could see you getting like a six x performance regression.
00:22:23.277 -->
00:22:23.727Right.
00:22:23.757 -->
00:22:24.237Because Okay.
00:22:24.237 -->
00:22:27.876You're just like, you know, you're contending on locks or something like that.
00:22:28.162 -->
00:22:28.402Mm-hmm.
00:22:28.761 -->
00:22:32.571I, I mean, and there certainly are a lot of locks, particularly in the earliest profiles.
00:22:32.886 -->
00:22:33.307Okay.
00:22:34.007 -->
00:22:40.366But it's not, it's not your sense that it's, I'm, the vibe I get from you is that it's literally just calling through bullshit indirection code.
00:22:40.547 -->
00:22:42.136It's not waiting on locks.
00:22:42.182 -->
00:22:44.642I, I, I don't wanna say there's no locks anymore.
00:22:44.701 -->
00:22:48.811Like the point of RCU is like, now you're spending less time waiting on the locks.
00:22:49.071 -->
00:22:51.652but like, like I said, it's still three x lower than we think.
00:22:51.652 -->
00:22:53.741It's like a reasonable baseline for like parsing DER.
00:22:54.422 -->
00:22:59.551And, and I, I will note that I think we've, we've fallen to the same trap because of the performance stuff is so easy to talk about.
00:22:59.882 -->
00:23:03.692Uh, but like, again, the performance is not our critical concern here.
00:23:03.692 -->
00:23:05.576It is the complexity that led to that issue.
00:23:06.122 -->
00:23:06.412Yeah.
00:23:06.412 -->
00:23:08.446Which I, I think it's worth building on kind of the.
00:23:09.076 -->
00:23:29.701Complexity in the public APIs is like a really important other, thing that is, is like honestly like maybe the most pressing thing for us, Yeah, you mentioned OSLs, O-S-L-O-S-S-L, param, and like, you know, I'm, I'm poking around in there, in, in OpenSSL for, you know, post quantum stuff, and I'm seeing this all over the place.
00:23:29.761 -->
00:23:34.507And like, tell what, where does OSL param come from and why, why is it there?
00:23:34.967 -->
00:23:35.207Yeah.
00:23:35.207 -->
00:23:39.586So OSL pram is, is another one of the kind of new APIs from OpenSSL 3.
00:23:40.317 -->
00:23:44.007And like it is effectively, like many, many public API functions.
00:23:44.007 -->
00:23:52.747Instead of like taking like a list of like arguments to a function, take, a an array of OSL paras, which is basically an array of key value pairs.
00:23:53.047 -->
00:23:59.487So like, instead of passing, I don't know if you, you're calling like argon, two, like you've got a key derivation function.
00:23:59.487 -->
00:24:02.096Like you've got, uh, you've got your key material.
00:24:02.217 -->
00:24:05.906You've got like your salt and your salt le maybe a length.
00:24:05.967 -->
00:24:10.406'cause you know it's c so you got your pointer and your length fine, uh, and your like desired output.
00:24:10.557 -->
00:24:24.227And, the RN and I can't remember the third parameters name, like e instead of like, you pass each one of those to function to a function as like arguments, the way this works in OpenSSL 3 is that you create an OSL parameter.
00:24:24.926 -->
00:24:34.777and it's got, you know, you know, string key pointer to the key string, uh, salt pointer to the past salt len and like.
00:24:35.477 -->
00:24:36.586That is like how it works.
00:24:36.586 -->
00:24:38.596And you know, you've got types for each of the values.
00:24:38.626 -->
00:24:41.507'cause like you wouldn't wanna Yeah, I was, I'm looking at that now.
00:24:41.957 -->
00:24:47.237I'm looking at the, the Claude summary on this now, and it's like integer, unset integer, UTF string octe.
00:24:47.936 -->
00:24:49.346Does it really have octet string?
00:24:49.991 -->
00:24:50.682it does.
00:24:51.382 -->
00:24:53.692you wanna pass like arbitrary bites, like what you want.
00:24:53.692 -->
00:24:55.491You think your salt should be like UTF eight?
00:24:56.192 -->
00:24:59.892I mean, it sounds like they've created C-J-S-O-N Oh goodness.
00:24:59.951 -->
00:25:00.582Yeah.
00:25:00.642 -->
00:25:01.001But like.
00:25:01.166 -->
00:25:02.967C JSON TypeScript interface.
00:25:03.326 -->
00:25:03.596Yeah.
00:25:03.596 -->
00:25:18.086And like, you know, a again, to like try to steel man, this, like our understanding of like, this is part of just like the make things very abstract theory of like, well you could write a program that like, I don't know, has a configuration file and like read some parameters from it.
00:25:18.477 -->
00:25:27.217And like that would mean that like somebody could bring a new algorithm and new types of parameters and like, you'd never have to update your program 'cause like it all flows through like this abstract OSSL_PARAM.
00:25:27.596 -->
00:25:34.227In practice, our experience for like the things we are trying to do is, it means, it's very difficult to tell what arguments the function takes.
00:25:34.317 -->
00:25:36.446It's very difficult to tell, you're passing 'em correctly.
00:25:36.446 -->
00:25:39.027You're, you are losing a whole bunch of like static like.
00:25:39.086 -->
00:25:42.356Static type checking that you would normally get from a computer program.
00:25:42.717 -->
00:25:46.707It makes things slow and it makes the OpenSSL code like much more complex, right?
00:25:46.707 -->
00:25:49.287Like if you think like, oh, I have the name of a variable.
00:25:49.287 -->
00:25:50.517That's an argument to my function.
00:25:50.696 -->
00:25:54.537Like that's just like clearly much simpler than I'm gonna go root around in the array.
00:25:54.717 -->
00:26:04.826And in fact, many C Source files in OpenSSL's source now have a custom Perl pre-processor to like make dealing wait, Oh, no, no.
00:26:04.961 -->
00:26:06.846wait, wait, So wait, wait.
00:26:06.957 -->
00:26:08.922Hold on, hold on, hold on.
00:26:09.071 -->
00:26:15.461In the new OSSL_PARAM world and OpenSSL 3, none of the cryptography interfaces are type safe anymore.
00:26:15.791 -->
00:26:18.311They all just take abstract arrays of parameters.
00:26:18.582 -->
00:26:20.906All the checking is, none of the new ones.
00:26:20.906 -->
00:26:24.551Like they're still old APIs it's all run, it's all runtime checking.
00:26:24.551 -->
00:26:26.902Now they're just like, okay.
00:26:27.352 -->
00:26:27.832Correct.
00:26:27.832 -->
00:26:27.981Yeah.
00:26:28.231 -->
00:26:29.701So we're going backwards.
00:26:29.836 -->
00:26:41.727yeah, any new interface like Eeb, E-V-P-K-D-F, and E-V-P-A-E-A-D, those are all interfaces that now require OSSL Perra, uh, and almost any new feature added, does require it.
00:26:41.727 -->
00:26:52.436have spent a lot of time and energy like trying to not use OSSL_PARAM except where necessary, and I think we currently have two places we consume it, but we actually abstracted away by pushing it into Rust OpenSSL.
00:26:53.021 -->
00:26:54.642Let me, let me steal man.
00:26:54.642 -->
00:26:55.271This design.
00:26:55.422 -->
00:26:56.021All right.
00:26:56.412 -->
00:27:02.771What if the idea here is that no normal person is ever supposed to use this interface, which is the only way I can think.
00:27:03.281 -->
00:27:06.192The only way I can think to describe a, A cryptography.
00:27:06.612 -->
00:27:11.112A cryptography interface where like the IV isn't type safe at, at compile time.
00:27:11.321 -->
00:27:16.717So you're saying me and my collaborators are not normal 'cause we did just that the other day.
00:27:16.856 -->
00:27:18.027on, hold on.
00:27:18.267 -->
00:27:28.977I'm saying only two people in the world are ever supposed to consume OSSL_PARAM, and it's Alex and Paul, and what you guys are supposed to do is take that and then turn it into something reasonable.
00:27:29.676 -->
00:27:41.307So I, I mean, I think what I would say is like, if you, you like, you'd come to the idea that like your, internal APIs needed to have this for whatever reason, right.
00:27:41.307 -->
00:27:47.477You, you needed, you know, more flexibility and like the provider API, 'cause like that a BI had to be the same for forever.
00:27:47.477 -->
00:27:51.017Like, if you, if you had like a theory of that, like this needed to be kind of your internals.
00:27:51.376 -->
00:28:04.196I like, I think the, like I, I'm not sure I would ever reach that conclusion, but like if I did, it seems like what I would want to do is like, have public APIs that are like entirely like type, safe and like construct these things internally.
00:28:04.376 -->
00:28:05.906And it's just like not the case.
00:28:05.906 -->
00:28:20.017Like if, for example, if the thing you would like to do is configure, OpenSSL to do elliptic curve, signatures that use, uh, deterministic nonces like specified in RFC, like the way you do that.
00:28:20.257 -->
00:28:20.646Yeah.
00:28:20.737 -->
00:28:21.487on, hold on.
00:28:21.936 -->
00:28:22.626Hold on a second.
00:28:23.326 -->
00:28:25.126You're telling me that in the new system.
00:28:25.826 -->
00:28:34.531If I wanna like do GCM or something and I'm passing a nonce, it has to do a string compared to find the nonce key.
00:28:35.166 -->
00:28:35.737Correct.
00:28:35.767 -->
00:28:53.096like I, I mean, I think in practice there's like old APIs that were type safe for like things like just like doing a symmetric encryption, but I would say, Alex, they've been significantly re-implementing some of the old APIs, using the new APIs so that like underneath the hood, they generate OSSL pers, and then there's definitely a string comparison.
00:28:53.396 -->
00:28:55.467So like, that's better than making me know about them.
00:28:55.467 -->
00:29:09.656'cause like literally, if you want to do an elliptic curve, private key signature, and you wanna use a deterministic nonce RFC 69 79, the way you do that is you create an OSSL paras with two entries, one of which is a bool indicating true.
00:29:09.777 -->
00:29:12.446And the other is like the marker for like, this is the end of the params.
00:29:12.717 -->
00:29:17.517And like you pass that to like EVPP key, like set paras or whatever the function's named.
00:29:17.906 -->
00:29:21.757Like, you can't just pass bull to like, you know, enable deterministic nonce.
00:29:21.777 -->
00:29:27.301Like you create It is like, it's like they're implementing Ruby, I, yes.
00:29:27.301 -->
00:29:30.632So this like, I, so like I used to be a programming languages person.
00:29:30.632 -->
00:29:34.172Like I, like Paul mentioned, like I got my start working on cryptography stuff.
00:29:34.172 -->
00:29:44.521'cause I, I was working on pi pi, the Python implementation and like, yes, this is what it looks like if you're like writing an interpreter and like, you know, a user can define arbitrary length functions.
00:29:44.521 -->
00:29:46.321Like, so you have to have an array of parameters.
00:29:46.321 -->
00:29:52.217Like, yeah, this one interpreter looks like, It It you don't get to get out of this discussion without noting.
00:29:52.217 -->
00:29:53.582You also worked on pie Ruby.
00:29:53.981 -->
00:29:58.211it was not called pyro, but it was called Topaz, but yes, yes, I created a Ruby interpreter.
00:29:58.912 -->
00:29:59.122Yeah.
00:29:59.821 -->
00:30:07.082I will say every time I end up doing OSSL_PARAM things, what it feels like to me is actually like, Apple's design aesthetic from next step era.
00:30:07.231 -->
00:30:23.231Like Ns, mutable dictionaries everywhere with all the same sort of challenges where like maybe the, like maybe the, principle like golden path has been tested, but God help you if you pass anything outta the ordinary, who knows what's gonna happen, which I think I gotta say, to testing.
00:30:23.932 -->
00:30:31.612I gotta say, passing in its parameters to a function, a dictionary of random string keys and values does feel very python to me.
00:30:32.311 -->
00:30:44.596Hmm, but it's, but this is the part that's in OpenSSL, supposed to be C library, but apparently it's getting OSL with param to just rename OSL per to qua, and suddenly you guys would be all over it.
00:30:45.297 -->
00:30:48.116I mean, it's not really our design aesthetic for Python.
00:30:48.116 -->
00:30:58.287Like, you know, I think we have like pretty strongly typed APIs within the Python world, but like y yeah, like that would be a recognizable like di dynamic programming language aesthetic.
00:30:58.287 -->
00:30:59.757That is like definitely true.
00:31:00.027 -->
00:31:05.301I just don't see why you would bring that aesthetic to C like, that's not Steve's problem.
00:31:05.602 -->
00:31:15.606So all of this dynamic stuff and lack of static validation and verification seems like it would make it, hard to test, or at least harder to test.
00:31:16.307 -->
00:31:19.666So we definitely, it's, it's definitely a difficult thing to test.
00:31:19.717 -->
00:31:23.946it is also the case that the OpenSSL project was founded in the nineties when aesthetics were different.
00:31:24.346 -->
00:31:35.926however, over the course of the, the decades the Open SS l project was been around and including the time now where they are a very well-funded organization with more full-time engineers than work on any of the forks, to our knowledge.
00:31:36.527 -->
00:31:37.971They, yeah.
00:31:38.001 -->
00:31:38.751And a foundation.
00:31:39.021 -->
00:31:43.942And a corporation and, 17 different, like interest groups at this point.
00:31:44.352 -->
00:31:44.622they.
00:31:44.842 -->
00:31:45.951they still struggle with testing.
00:31:46.011 -->
00:31:53.422So like Alex and I consistently joke that like the Python Cryptographic Authority is a CI engineering project that incidentally produces a cryptography library.
00:31:53.961 -->
00:32:09.231and part of the reason we make that joke is that like, it, it, it reflects our real belief that like, that type of infa investment in continuous, in integration and testing pays dividends in terms of like software engineering velocity and the quality of, of the product we deliver.
00:32:09.892 -->
00:32:13.912we spend so much time on it that like it can almost make the other work we do seem trivial.
00:32:14.382 -->
00:32:18.971unfortunately the OpenSSL project, I mean, we've worked with a lot of these folks.
00:32:18.971 -->
00:32:20.622We, we, we, we like these people.
00:32:20.622 -->
00:32:23.981But like it, it's important to, to note where, where struggles continue.
00:32:24.102 -->
00:32:30.432And one of those things is that despite all this time the OpenSSL project does not prize testing in the way that we prize testing.
00:32:30.981 -->
00:32:31.942we have seen.
00:32:32.642 -->
00:32:50.082There are, there are many ways, we, in which you can judge this, but one of the ways which is fairly prominent is go and look at any bug fix or new feature set that lands on, uh, OpenSSL and look to see whether or not there are tests, now new features typically, yes, bug fixes frequently no.
00:32:50.612 -->
00:32:54.961and if you ask about them, they won't say they shouldn't be tests, but they may not happen.
00:32:54.991 -->
00:32:58.352They're not prioritized in the way that you would expect a project like this.
00:32:58.951 -->
00:33:01.471Similarly, you have a large CI matrix.
00:33:02.172 -->
00:33:08.027Alex and I spend a lot of time and energy making sure our CI matrix is clean and fast because not otherwise is very painful.
00:33:08.622 -->
00:33:17.442In fact, uh, as, as a, as a related note, OpenSSL ship, the bug fix release today, which meant we ship the release because we statically link OpenSSL in our binary artifacts, the wheels.
00:33:18.142 -->
00:33:24.172And because we've spent two weeks with Windows arm 64 builders failing, we removed Windows, arm 64 support.
00:33:24.442 -->
00:33:26.122That's how serious we are about this sort of thing.
00:33:26.821 -->
00:33:27.451Failing.
00:33:27.451 -->
00:33:33.122Not like, 'cause we landed a regression, but due to like a platform issue in GitHub's, uh, windows arm 64 runners.
00:33:33.767 -->
00:33:34.247Yeah, yeah.
00:33:34.247 -->
00:33:37.261This, it wa it was because of the, the runners.
00:33:37.457 -->
00:33:37.747Yeah.
00:33:37.997 -->
00:33:42.287The runners themselves have platform and it's not the first time we've had issues like this.
00:33:42.317 -->
00:33:49.007This is not a, a conversation about how Microsoft owns GitHub and yet somehow they don't prioritize Windows Arm 64 at all.
00:33:49.156 -->
00:33:56.656But like, we, we've had enough issues and that like be, we gave them a lot of time, but we care very deeply about our CI working and being performant.
00:33:56.686 -->
00:33:59.626And so we were willing to remove it even though it, it, it is painful.
00:34:00.221 -->
00:34:03.461that's indefinitely until those things can work again.
00:34:03.866 -->
00:34:04.047Yeah.
00:34:04.047 -->
00:34:06.761When, and they work and we have confidence they'll stay working.
00:34:07.332 -->
00:34:07.737Exactly.
00:34:07.737 -->
00:34:08.996I need a track record behind it.
00:34:09.027 -->
00:34:15.896'cause fixing it is fine, but like I need to see it working for an extended period of time and that like, they actually respond in a timely fashion to future issues.
00:34:16.461 -->
00:34:22.012a failure is just flapping all the time, then it's, it's a noisy signal and it's not a useful signal, and then it's yeah.
00:34:22.072 -->
00:34:28.081Flip flapping at the all the time is like a, a useful segue into like, that is a big problem with Yeah.
00:34:28.561 -->
00:34:39.192really the apex of this was OpenSSL 3.0 0.4 had a buffer overflow in the RSA implementation, for a VX five 12 capable CPUs.
00:34:39.461 -->
00:34:47.271And in fact, this failed in CI sometimes when the CI runner happened to be allocated on a machine with AVX five 12.
00:34:47.481 -->
00:34:52.521But it didn't but, but it was doing a different architecture, like.
00:34:52.521 -->
00:34:54.817it was randomly GitHub action.
00:34:54.817 -->
00:34:58.626Well, so GitHub actions doesn't guarantee like which CPU class you'll get.
00:34:58.777 -->
00:35:02.737And so like sometimes you built on a machine with AVX five 12, sometimes you didn't.
00:35:02.856 -->
00:35:08.387And so like when you were on AVX five 12, your tests probably failed, like as a buffer overflow.
00:35:08.387 -->
00:35:11.606So there's always an element of luck, but it was not noticed.
00:35:11.697 -->
00:35:13.737'cause tests were kind of always flaky.
00:35:14.126 -->
00:35:16.146And, you know, I think this reflects two issues.
00:35:16.146 -->
00:35:19.657One tests being flaky all the time, which is a really persistent issue.
00:35:19.657 -->
00:35:27.472Like the day we wrote our slides, for the original talk version of this, five of the 10 recent commits had failing, ci checks.
00:35:27.561 -->
00:35:33.902And when we checked again the day before we get delivered, our talk, every single commit, had failing builds for cross compilation.
00:35:34.021 -->
00:35:37.291This is like the first issue, just like lack of really prioritizing stability.
00:35:37.561 -->
00:35:41.431The second issue is like lack of investment in the kind of infrastructure.
00:35:42.181 -->
00:35:47.581for like open a sales, a project with lots of like per platform assembler, per platform assembly.
00:35:47.702 -->
00:35:54.902And Intel in fact, offers a tool called Intel, SDE, which basically lets you dynamically toggle, uh, CPU features on and off.
00:35:54.902 -->
00:36:00.871So you can, you know, simulate A CPU without a VX five 12 A CPU without SSE three.
00:36:00.961 -->
00:36:04.231And so you can in fact test all of the combinations of assembly you have.
00:36:04.532 -->
00:36:08.882And there are forks that use this in their CI to verify all of their assembly.
00:36:08.882 -->
00:36:12.851And it, you know, this is not present in OpenSSL ci.
00:36:12.851 -->
00:36:13.992And like, we think that's a real miss.
00:36:13.992 -->
00:36:15.612Like that's just a missed opportunity.
00:36:15.617 -->
00:36:23.306Like I'm, look, I'm, I was, I was stunned that you're like, you can't make sure that you get like an AVX backend runner in GitHub actions.
00:36:23.306 -->
00:36:31.527And like the answer is okay, maybe you can't, but that means you put your A VX specific stuff behind a flag.
00:36:31.706 -->
00:36:34.887You put, online your own custom runner.
00:36:35.157 -->
00:36:50.306You, you rack your own hardware or you pay someone to rack the hardware, and then you have the flag so that when you do allocate to your custom runner that always runs on AV X, then you unflag your flag and then you test your AV X specific implementation.
00:36:50.516 -->
00:36:52.887I know lots of projects that do this.
00:36:53.442 -->
00:36:53.742Yeah.
00:36:53.802 -->
00:36:57.101now a word from our sponsor, fly.io, provider of a VA escapable.
00:36:57.802 -->
00:37:02.422And from the official hot sauce of security, cryptography, whatever, crystal hot sauce.
00:37:03.121 -->
00:37:04.472that a salt brand?
00:37:05.172 -->
00:37:08.931It's also you're thinking of a diamond.
00:37:08.961 -->
00:37:09.681Diamond crystal.
00:37:10.222 -->
00:37:10.762Yes.
00:37:10.762 -->
00:37:11.242Yes.
00:37:11.632 -->
00:37:14.541I am gonna be very disappointed if that doesn't make it into the co podcast.
00:37:14.822 -->
00:37:23.007Leaving the, uh, so like, I mean there's, as you noted, Deirdre, there's a bunch of ways where you can, you can slice this such that you get the type of testing coverage you want.
00:37:23.336 -->
00:37:28.436It's also the case that OpenSSL is at this point, a big tent with a wide variety of supported things.
00:37:28.586 -->
00:37:31.286And they actually do have leverage in this ecosystem.
00:37:31.527 -->
00:37:35.817So like if and when someone comes and says, I would like you to land.
00:37:36.402 -->
00:37:43.181Like architecture specific assembly for my pet architecture, it would not be out of the bounds for them to say.
00:37:43.882 -->
00:38:08.331Supporting that in our, in our system looks like the following and it look would pro, frankly, it would probably look similar to what Alex and I have said in the past around Power PC 64, little endian support or Windows arm 64, which is you will provide, you know, ephemeral runners that can maintain no state and integrate into our CI such that we do not have the responsibility of, of like managing them and they will work in the following fashion, And if they do not, we drop you.
00:38:08.731 -->
00:38:09.336yeah, yeah.
00:38:09.336 -->
00:38:11.467I mean, I give a real concrete example there.
00:38:11.496 -->
00:38:21.197There's an open bug right now against OpenSSL for the assembly, for Spark for doing, I don't even know what, and it's, it's got some bug in an optimization pass.
00:38:21.456 -->
00:38:26.257And you know, the OpenSSL folks have basically said like, look, SPARC assembly is not maintained by the OpenSSL core team.
00:38:26.257 -->
00:38:27.637Like this is community maintained.
00:38:27.907 -->
00:38:31.237Like what I would encourage them to do is say like.
00:38:31.936 -->
00:38:32.657This is a bug.
00:38:32.657 -->
00:38:41.597Therefore, like we're disabling this optimization If like the maintainer of the SPARC platform wants to like, contribute CI or contribute fixes, like we will accept those.
00:38:41.597 -->
00:38:52.306But like we don't wanna ship this buggy thing and we don't wanna give users like the impression that like, you know, give them like, we, we don't wanna carry all of this performance sensitive and buggy code.
00:38:52.487 -->
00:39:03.226We would rather slip the ship, the slower thing that's guaranteed to work and put really the onus on like, if the spark owner wants that to be a fast OpenSSL supported thing, like they should do the work.
00:39:03.257 -->
00:39:07.577And so like, you know, if the OpenSSL, project like pushed on things like that, we'd be very supportive.
00:39:07.976 -->
00:39:19.617we haven't even touched on all the work that the, the Python Cryptography Authority has done on moving towards doing a lot of the riskiest stuff in, a memory safe language like Rust.
00:39:20.007 -->
00:39:32.586But like you, just, the two of you and your project did it a ton of stuff on your own, and we're not, it's 20, it's 2025 at this it's All, all of it's 20.
00:39:32.586 -->
00:39:34.327I'm sorry, it's 2026.
00:39:35.027 -->
00:39:36.391It's 2026.
00:39:36.782 -->
00:39:39.692It's like the most scrutinized C code on the planet.
00:39:40.112 -->
00:39:42.902All of like the low hanging fruit, memory, corruption, and open.
00:39:43.411 -->
00:39:45.152An OpenSSL gone now, right?
00:39:45.152 -->
00:39:49.762Like how likely one today, so I was gonna say, like we, we were literally today.
00:39:49.891 -->
00:39:52.336we were, or Tuesday the 27th.
00:39:52.637 -->
00:39:59.911We were originally scheduled to record this podcast yesterday, and if we had recorded this podcast yesterday, we would not have been able to discuss it.
00:39:59.911 -->
00:40:04.322There were several pieces of memory corruption in OpenSSL that were disclosed today.
00:40:04.472 -->
00:40:08.677And like Goodie If we had recorded yesterday, I would've had you with that argument.
00:40:09.376 -->
00:40:09.496I.
00:40:09.896 -->
00:40:17.637I'm kind of like with you, Thomas, like adults understand how to write small bits of C code without like totally screwing it up.
00:40:17.907 -->
00:40:25.277Part of that is having the judgment to not write large bits of c code, parsers for like length, type value.
00:40:25.726 -->
00:40:30.067And, you know, deterministically bites in, bites out functions.
00:40:30.067 -->
00:40:37.967Like we should be able to write in Rust or, or excuse me, in assembly in sea without making huge mistakes.
00:40:38.112 -->
00:40:42.311So, the people that are working on OpenSSL don't seem to be able to do this.
00:40:42.641 -->
00:40:45.402Many people exist who are capable of doing this.
00:40:45.641 -->
00:40:47.621Many of them have been guests on this podcast.
00:40:47.922 -->
00:40:48.036Huh.
00:40:48.251 -->
00:40:50.112So I I, I'll agree partially with that.
00:40:50.112 -->
00:41:06.251Like f folks who are familiar with my work know that, like I talk a lot about memory safety as a language level concern and about how in the long term we need to be looking at CNC plus plus replacements or, or to the extent the CNC plus plus standards committees have any openness to make their languages memory safe.
00:41:06.521 -->
00:41:18.282But it is absolutely true that you can write small, where small is like probably less than a few thousand lines of like c code for well-defined tasks that don't change very often and maintain memory safety.
00:41:18.402 -->
00:41:28.121Like that's, that's like an observable fact that like boring SSL is a code base like this that has like a very low rate of vulnerabilities because it's maintainers are like very diligent maintainers.
00:41:28.211 -->
00:41:29.561They take testing seriously.
00:41:29.711 -->
00:41:31.842They just like think hard about the changes they're making.
00:41:32.472 -->
00:41:47.902But it is also just true that complexity and velocity are real world phenomena and, Part of how BoringSSL is able to be that diligent is by having a very narrow scope that is like roughly the set of things Google exclusively cares about.
00:41:48.081 -->
00:41:51.532And open cell does intend to cater to a larger audience.
00:41:51.532 -->
00:41:53.842It has more features, supports more functionality.
00:41:53.842 -->
00:41:55.911And that's like reasonable and like very useful.
00:41:56.092 -->
00:42:02.722Like when we compare the set of features, open cell has that boring, a cell does not, like, there are things missing that we would like, for example.
00:42:02.961 -->
00:42:05.992so like complexity and velocity are real world phenomena.
00:42:06.231 -->
00:42:07.791Scope is a real world phenomena.
00:42:07.851 -->
00:42:14.001And so you need approaches to security that are responsive to the real world, to those real world constraints.
00:42:14.121 -->
00:42:24.202And so, we think you just have to have a design approach and like a memory, safe programming language is by far the strongest one to not have certain classes of vulnerabilities.
00:42:24.202 -->
00:42:27.742Like formal verification is another thing from that bucket for like right.
00:42:27.742 -->
00:42:30.112How do you write certain types of programs?
00:42:30.382 -->
00:42:31.902Very, very safely.
00:42:32.601 -->
00:42:40.501I wanna, I wanna talk a little bit more about how you guys are SSL of Theseus-ying, the, uh, OpenSSL library with your own Rust code.
00:42:40.681 -->
00:42:59.831But before we do that, like, the thing today was if you give open LAP seven file that, um, is encrypted with an A 80 cipher is encrypted with GCM or whatever the EVP code, the high level OpenSSL library when it goes and tries to parse the P seven file and like pull the nons out of the it's DERight?
00:42:59.851 -->
00:43:01.697Or it's BER, is it BER or DER in P seven.
00:43:01.996 -->
00:43:02.806It can be either.
00:43:03.507 -->
00:43:03.987Wonderful.
00:43:04.106 -->
00:43:04.467Okay.
00:43:04.617 -->
00:43:09.777When it goes to, when it goes to pull out the, the, you know, the, the nons slot of the goo right.
00:43:09.896 -->
00:43:13.061it, it, it spills that goo all over the stack, right?
00:43:13.092 -->
00:43:20.902So, like, I guess the, the first question I have is, does that, we were, we were talking about this earlier, but I'm still wondering if this does or does not hit you guys.
00:43:21.601 -->
00:43:22.521this doesn't hit us.
00:43:22.782 -->
00:43:24.931while we do have, some, uh.
00:43:25.467 -->
00:43:35.282Pkcs, seven APIs that do still use OpenSSL for parsing, for reasons we can talk about our pkcs, decryption APIs use our own, Rust, DER library.
00:43:35.552 -->
00:43:38.222And it's a, you know, totally memory safe parsing.
00:43:39.101 -->
00:43:46.942are like, this is a little, this is a little off topic, but like for somebody who's asking what the attack surface is for P seven files, what does that look like?
00:43:47.291 -->
00:43:53.681so Pkcs seven is like a pretty widely used container format, like it pops up in all sorts of places.
00:43:53.681 -->
00:43:55.541Like Mime is A-P-K-C-S.
00:43:55.541 -->
00:44:06.231So like, if you're doing like, encrypted email, like not pg, not like pg p, but like, you know what, like what, what Microsoft Exchange will like, encrypt your emails with, like, that's a PKC S seven format.
00:44:06.811 -->
00:44:13.172I think Microsoft's like code signing format does it, like it pops up in all sorts of like places like this.
00:44:13.172 -->
00:44:18.481Like if the thing you were shipping is like roughly a signature and encrypted blob and like some metadata and like.
00:44:19.036 -->
00:44:26.956Particularly if it's like maybe a slightly older standard, like I, I think modern es, modern cryptographic aesthetics or like container formats like this are not particularly useful.
00:44:26.956 -->
00:44:30.166But like in, in older things they were like super common.
00:44:30.782 -->
00:44:31.711Yeah, it's actually, uh.
00:44:32.202 -->
00:44:41.981I forget what the underlying weird name for the like, uh, standard is, but Apple Pay also actually uses Pkcs seven signatures, uh, in the backend.
00:44:42.681 -->
00:44:58.146But not with OpenSSLI.
00:44:59.286 -->
00:45:02.016I looked, I looked today, and Libra SSL didn't have the bug.
00:45:02.016 -->
00:45:06.876Libra ss l in the code where they pull the, the, the non Sout, is doing an explicit length check.
00:45:06.907 -->
00:45:08.347Does boring SSL have the bug?
00:45:08.887 -->
00:45:09.556I believe so.
00:45:09.556 -->
00:45:09.856No.
00:45:09.856 -->
00:45:13.132I, I. Pretty so BoringSSL actually published today.
00:45:13.132 -->
00:45:19.041A whole bunch of like notes from past OpenSSL, vulnerabilities on, you know, what, whether they were infected.
00:45:19.041 -->
00:45:21.411Like what, what customers of BoringSSL need to know.
00:45:21.742 -->
00:45:28.036And I'm almost positive that this is one of the ones that marked like this bug was introduced after we forked Mm-hmm.
00:45:28.737 -->
00:45:28.947Okay.
00:45:29.646 -->
00:45:32.077Ah, So like, you guys missed this bug.
00:45:32.331 -->
00:45:46.052You, you missed out on all the fun because you rewrote part of OpenSSL I wouldn't say they're missing it, Yeah, we, we missed this bug there, there were a handful of other bugs, in today's release that, that did impact us.
00:45:46.152 -->
00:45:53.092I, I'd have to go back and like, look, look at the full list to tell you which ones, but How, how much of OpenSSL are you guys gonna rewrite in Rust?
00:45:53.556 -->
00:45:53.791Hmm.
00:45:53.791 -->
00:45:57.121Anything that's not cryptographic crypto cryptography itself.
00:45:57.152 -->
00:46:02.911The core crypto like routines are that one before we opened with how that's false.
00:46:03.302 -->
00:46:03.632Yeah.
00:46:03.661 -->
00:46:09.552so like, we're, the things I would say is like, we will do anything that is like parsing that.
00:46:09.552 -->
00:46:16.751It, it's, you know, serialization, deserialization that is like orchestrating cryptography, like kind of the HPKE that we mentioned.
00:46:17.141 -->
00:46:18.641But encodings.
00:46:19.032 -->
00:46:22.661yeah, we're pretty close to saturated on this stuff.
00:46:22.661 -->
00:46:32.831Like almost all parsing at this point, whether it's, you know, public keys, private keys, whether it's like X.509 certificates and CRLs, all, all of that stuff is in Rust.
00:46:33.112 -->
00:46:35.211Yeah, I think the 5 0 9 path building.
00:46:35.692 -->
00:46:36.112Yeah.
00:46:36.532 -->
00:46:52.492the only exception to this point is actually what Alex alluded to earlier, which is that because, p Pkcs seven does support BER in addition to DER we do have in one code path a fallback where if we can't parse it using our DER parser, we hand it to OpenSSL because we have not implemented BER and Rust.
00:46:53.192 -->
00:46:54.556For, for and for but you're going to.
00:46:55.257 -->
00:46:57.152no, we probably won't for, for those Yeah.
00:46:57.152 -->
00:47:10.072For those who are in this like alphabet soup of like BER and DER just as like causing their eyes to glaze over, they're, the, the really short version is BER and DER are two different ways of serializing, kind of the same as n one data.
00:47:10.382 -->
00:47:16.052And DER is a subset of BER that like basically takes away all of the flexibility that BER has.
00:47:16.052 -->
00:47:32.192Like BER will let you do kind of like very bizarre things that makes it like much more complicated to parse and like, we basically decided like BER is a bad idea for all modern cryptographic standards and like DER is just like a mu much more compact from a, like a surface perspective.
00:47:32.192 -->
00:47:47.902So like we, we restrict ourselves to DR and like, that's the only thing you need to care about for things like X.509 certificates, kind of standards that are less well tended than X.509 is, maybe I'll say it like PKCS seven still have a lot of BER in their ecosystems, and.
00:47:48.336 -->
00:47:53.697And when we refuse to support BER, and so we'll use OpenSSL in the places we have a compatibility need to parse BER.
00:47:54.467 -->
00:47:56.356So has that, how has that gone for you guys?
00:47:56.356 -->
00:48:06.226How has that I, I'm, I keep stepping on Deirdre and I'm sorry, but I, I, I want to hear, like, I, I want to hear the story about how it's gone taking PICA cryptography and making it a Rust project.
00:48:06.927 -->
00:48:07.737I think it's gone.
00:48:07.751 -->
00:48:09.597that's like a whole podcast of its own, but.
00:48:09.601 -->
00:48:09.891yeah.
00:48:09.896 -->
00:48:10.376Yeah.
00:48:10.552 -->
00:48:14.996And, and So it was, it was complicated and dramatic, and not a great experience is what I'm hearing.
00:48:15.356 -->
00:48:16.407No, no.
00:48:16.407 -->
00:48:24.097I mean, so we, we gave a talk at, about this at Pycon a couple of years ago that really focuses on like the initial release and like what that migration looks like.
00:48:24.456 -->
00:48:32.081but and, and to be clear, there, there were challenges and some drama early on where like, we, we, is early on, like how many years ago was this?
00:48:32.606 -->
00:48:36.1172022, I wanna say not that long ago.
00:48:36.117 -->
00:48:36.126Oh.
00:48:36.427 -->
00:48:37.3172021.
00:48:37.467 -->
00:48:38.367No, it's 2021.
00:48:38.367 -->
00:48:39.416Alex, I'm, I'm Okay.
00:48:39.416 -->
00:48:39.95620. Yeah.
00:48:39.987 -->
00:48:41.876It's like February, 2021.
00:48:42.266 -->
00:48:42.507Yeah.
00:48:42.507 -->
00:48:42.686Yeah.
00:48:42.686 -->
00:48:45.086February, 2021, we do our initial release.
00:48:45.237 -->
00:48:46.916So it's like five years ago now.
00:48:47.186 -->
00:48:49.047and the initial release has some, has some drama.
00:48:49.047 -->
00:48:52.976Like we, we were pretty aggressive in like pushing Rust into the ecosystem.
00:48:53.217 -->
00:49:07.157Users who were getting not getting wheels, they had to compile themselves, like were, and if they weren't per pinning their versions, like they woke up one morning and were like, why is my, Ansible CI pipeline failing with like no Rust sea on my path?
00:49:07.396 -->
00:49:08.567Like, what is this garbage?
00:49:08.847 -->
00:49:14.496but like we have with lots of help from like other folks in the community, we have like pushed past that.
00:49:14.496 -->
00:49:19.387Like we now we ship wheels for a great many platforms like that is no longer a problem.
00:49:19.657 -->
00:49:24.757And so like I, you know, I think your question is mostly about like, how has migrating our own code been?
00:49:24.757 -->
00:49:26.347And like, I think it's pretty.
00:49:26.751 -->
00:49:29.001Much just been like an across the board win.
00:49:29.271 -->
00:49:32.661We have much better performance on like all of our parsing APIs.
00:49:32.661 -->
00:49:34.672As we kind of alluded to earlier.
00:49:34.791 -->
00:49:38.242We have a much clearer compatibility surface.
00:49:38.242 -->
00:49:50.541Like, because we own the parsing, we understand exactly what are the places we're being lax because like, you know, the, the specification is like kind of in an ht ML style, like diverged from like common practice.
00:49:50.632 -->
00:49:52.192And we know where we're being strict.
00:49:52.552 -->
00:49:57.141we have a much, just much clearer, abstraction layers between things.
00:49:57.141 -->
00:49:59.512So like, I'll give you a concrete example of, I think that's just like better.
00:49:59.782 -->
00:50:03.382so we used to, we, we have a bunch of Acts 5 0 9, like certificate APIs.
00:50:03.871 -->
00:50:07.331we used to implement those on top of OpenSSL X.509.
00:50:07.391 -->
00:50:08.021APIs.
00:50:08.172 -->
00:50:12.222And so when you did something like sign an X.509 certificate, right?
00:50:12.431 -->
00:50:21.621Creating a new X five nine certificate and you do a signature, and you'd pass in a private key to like do that signature, but actually that private key had to be an OpenSSL private key.
00:50:21.831 -->
00:50:32.422Like we, we nominally had these like abstract APIs that like you could implement, for a private key, but like if you didn't pass an OpenSSL private key, we didn't have a private key to pass to the OpenSSL signature API.
00:50:32.661 -->
00:50:36.501So like there's this real like abstraction failure and it's, it's not an uncommon one.
00:50:36.501 -->
00:50:40.161Like you, you see this in a bunch of things that like try to do, abstraction layers like this.
00:50:40.161 -->
00:50:47.251Like I believe the Java crypto APIs have like a whole bunch of like fast paths and slow paths depending on what kind of, pri private key you've got.
00:50:47.402 -->
00:50:50.012But now that we own X five nine parsing top to bottom.
00:50:50.597 -->
00:51:12.157What happens when you try to do a, a sign a new certificate is that we use the public sign, API and like any private key you've got, whether it's one of the ones we provide or you have like a third party implementation of, of our private key APIs that uses that, I don't know, talks like A-W-S-K-M-S for example, or GCP or like any, any cloud providers key management.
00:51:12.246 -->
00:51:13.327Now that just works.
00:51:13.416 -->
00:51:16.507So like we have a much cleaner compatibility surface.
00:51:16.626 -->
00:51:20.827We have much more coherent story for things like third party keys.
00:51:20.856 -->
00:51:30.206Like, it's just like, like I don't have any bad to say about the migration besides like the initial like stumbling with like, the kind of pain of lots of people having it to adopt to Rust.
00:51:30.206 -->
00:51:32.697But Paul, I don't know if you have like a different reflection on this.
00:51:33.396 -->
00:51:40.626Yeah, I mean, I, I think I generally agree like pa past the initial teething, like one of the components of that, that entire project is Alex and I decided that.
00:51:41.327 -->
00:51:43.217This was worth the breakage budget.
00:51:43.246 -->
00:51:45.436We knew our position in the ecosystem was important.
00:51:45.436 -->
00:51:49.186We knew that it was going to cause us, us pain as well as some user pain.
00:51:49.396 -->
00:51:58.697But we wanted to, you know, both manage, manage our, uh, manage our long-term pain as maintainers and also drive down the pain for the adopters as quickly as possible.
00:51:58.876 -->
00:52:07.007So we were able to work across the ecosystem, basically blaze the trail, such that like future Rust, Python projects have effectively the ability to deploy with no fear.
00:52:07.036 -->
00:52:10.007Where we, five years ago, obviously had a lot of work to do.
00:52:10.547 -->
00:52:12.766on the actual Rust development side, I would say that like.
00:52:13.456 -->
00:52:23.996The only piece of pain that we've really experienced, I'm not, not even MSRV, although MSRV has its own like that's minimum supported Rust version for, for folks who are not deep in the Rust ecosystem.
00:52:24.387 -->
00:52:31.567Uh, there was some work there we had to do, but like the only real thing was that in our ci Rust compilation is slower than what we had before.
00:52:31.626 -->
00:52:38.677And so we ended up spending a lot of time and energy looking at what it meant to cache intermediate artifacts and make sure the caches don't do bad things.
00:52:38.706 -->
00:52:41.317'cause we had a few incidents where our caches were pretty bad.
00:52:41.737 -->
00:52:44.106but once we got past that, we were in really good shape.
00:52:44.226 -->
00:52:55.097And like, I, I mean, even right now there's, there's a current, feature Alex and I are working on where we were unhappy with the Python APIs, uh, that would allow us to express what we needed.
00:52:55.186 -->
00:53:00.077And so we're likely to rewrite this piece in Rust simply so we can have the visibility control that we want.
00:53:00.416 -->
00:53:00.706Yeah.
00:53:00.742 -->
00:53:12.407Yeah, I, so like, I don't know if I had to register a complaint, like my number one complaint might be like, the Rust coverage support is, not quite as stable as like pythons coverage pie.
00:53:12.677 -->
00:53:18.461Like, th this is like the level of like complaint we have, you know, about five years of like maintaining this Rust code.
00:53:19.161 -->
00:53:33.931so you, you ate the pain of supporting Rust in your extremely widely supported, in terms of platforms, a diversity of platforms where it is, is used and it needs to be supported.
00:53:34.141 -->
00:53:40.981Did you have to drop any platforms that PI Crypto was supported on in order to ship the Rust stuff?
00:53:41.387 -->
00:53:45.166We dropped no platforms we officially supported is a good way to say that.
00:53:45.416 -->
00:53:50.097one of the tricky bits of a migration like this, and I'm sorry Alex, I'll, I'll no, you we're gonna say the same thing.
00:53:50.396 -->
00:53:51.842but yeah, one of the tricky bits about.
00:53:52.541 -->
00:53:59.652Having a project that's just C and Python is that C and like Python is not a compiled language and c is a compiler that exists.
00:53:59.711 -->
00:54:05.532Like there's a compiler that exists for every platform under the sun, including things that are weird, like 31 bit architectures.
00:54:06.016 -->
00:54:21.277and so like implicitly, when you ship software like this, you end up with consumers at some level who maybe they only want to compile it to prove they can, but there's a set of folks who are like, we, like I was able to compile this and therefore it should be supported.
00:54:21.516 -->
00:54:33.126And so one of the things that did come out of our Rust migration was a much more obvious, like a much more clear and obvious statement of like, we support architectures that have enough support that LLVM has been ported to it.
00:54:33.577 -->
00:54:48.217Uh, and one of the, one of the, perhaps most prominent, and Alex and I might be overly patting ourselves on the back for it, but also it really feels like we, we might be responsible is that IBM recent, I guess now it's about a year ago, but, uh, IBM ported Rust to AIX.
00:54:48.472 -->
00:54:53.887And one of their headline messages on the blog post where they announced it was Python Cryptography will work.
00:54:54.586 -->
00:54:56.146Oh, that's pretty nice.
00:54:56.652 -->
00:54:56.847Yeah.
00:54:56.847 -->
00:55:00.496And like this is, You're perpetuating A IX.
00:55:00.527 -->
00:55:01.177It's not nice.
00:55:01.806 -->
00:55:10.356I know, but like, the fact that they, like they are that good at This is a positive sum interaction, is the way I would say it.
00:55:10.356 -->
00:55:15.277Like I, IBM like gets better support for whatever customers A IX has.
00:55:15.697 -->
00:55:40.447We can point users at a IX who like show up at our issue tracker and like IBM, like they maintain your stuff and like, I think this is like a good message to like projects, to like when you get requests for like weirdo, particularly commercial operating systems or architectures, like push back and like try, try to make people go to like the company they have a support contract with and not like pawn it off on you.
00:55:40.746 -->
00:55:43.027You know, if you, if you're not interested in maintaining that.
00:55:43.327 -->
00:55:48.871Let's, let's talk about like that aspect a little more before we get back to like broader points about OpenSSL.
00:55:49.351 -->
00:56:02.271both you, Alex and Paul have been doing like, open, open source, I say in quotes, for, you know, like 13 years here, you have more than that, but this specific project, it's very widely used.
00:56:02.702 -->
00:56:07.442but, well I, I don't know Paul as well, but Alex seems like a fairly emotionally stable person.
00:56:07.831 -->
00:56:15.442and, Like you see a lot of discourse around open source of like, oh, there's all these like, people freeloading off me.
00:56:15.442 -->
00:56:16.791I don't have enough time now.
00:56:16.791 -->
00:56:17.902I hate this project.
00:56:18.492 -->
00:56:31.322but at the same time and kinda look over at you guys and you've been able to both kind of keep some amount of support for like a lot of, or, or, or keep a lot of support for a lot of platforms you've been able to get Intel to do.
00:56:31.842 -->
00:56:34.652or excuse me, or who, whoever to do AIX.
00:56:35.001 -->
00:56:48.532and like you're, you're still chugging along and to some extent, like the Rust thing, like you chose to do that in Rust in part because of your like personal preferences around memory safety, which I agree with.
00:56:48.771 -->
00:56:53.632But like you could have also just been like, well, we'll control our own parer by writing c code that doesn't like suck.
00:56:54.342 -->
00:56:58.692so like you're able to like kind of push your personal opinions into the project and.
00:56:59.391 -->
00:57:01.251Kind of have fun without dying.
00:57:01.311 -->
00:57:13.822Like what's your approach to this that like, lets you both kind of keep doing this, but also like doesn't necessarily result in either the drama or BERnout that you hear a lot about in open source?
00:57:13.822 -->
00:57:17.782And how do you feel about, like, like are you being funded sufficiently for this?
00:57:17.782 -->
00:57:26.632Or like, like how does money fit into this, if at So very early on in the project's history, actually, like Paul and my employer, like, gave us both time to work on it.
00:57:26.811 -->
00:57:29.782That's close to 10 years ago at this point for me.
00:57:29.931 -->
00:57:31.461Maybe it's more than 10 years ago.
00:57:31.641 -->
00:57:37.822So it's, it's, you know, for the last 10 years at least, this has been a, you know, labor of, you know, my, my personal time.
00:57:38.181 -->
00:57:41.331as for like, I guess how we think about it, I think.
00:57:42.032 -->
00:57:56.141You know, I'm not, I'm not sure we have like a documented philosophy of this or something, but like, we would like working on this project to be, sustainable and enjoyable and a, product that is like, you know, advances things we care about in the world, right?
00:57:56.141 -->
00:58:00.402Like we think, you know, crypto cryptography, the ability to build secure systems is important.
00:58:00.402 -->
00:58:02.322Like we're, this is like a positive contribution.
00:58:02.322 -->
00:58:08.771If we can make that easier in the Python ecosystem, if we can advance memory safety like that is, that is a, like, good thing for the world.
00:58:09.012 -->
00:58:15.101And so, like part of what you hear when you hear us talking about like the importance of like really robust testing is that.
00:58:15.802 -->
00:58:25.851For a thing you work on in your spare time, it's like really, really valuable for the way things come to you to be predictable and not emergencies, for example.
00:58:25.851 -->
00:58:36.762So like, it is a thousand times more preferable to like spend some time, you know, getting a PR degree, because like I control when I'm working on that than to have like a vulnerability reported to us.
00:58:36.762 -->
00:58:49.541Like I, you know, if you're like a company, you talk about this in terms of like shift left and like your developer productivity, but like for in your a volunteer project, like what I'm saying is like, it's really good to like not have a vulnerability get reported.
00:58:49.541 -->
00:58:56.876Like when I'm busy with life, like I can, you know, I can take a week or two off this project because it is stable and like, that's, that's really good.
00:58:57.177 -->
00:58:59.902and the stuff that you're working on, the time is not like.
00:59:00.601 -->
00:59:14.161Just trying to debug a flap, ECI or like trying to fig, like wasting time just trying to make something functional and workable as opposed to like, Hey, here's a new feature that people want.
00:59:14.161 -->
00:59:15.391I need to go ship it.
00:59:15.391 -->
00:59:25.527Or a new primitive or making something better, like we put a lot of time to ci like, you know, it, I don't know if it's literally the majority of our time, but it, it's really plausible.
00:59:25.527 -->
00:59:36.746It's the majority of our time, but it's almost just overwhelmingly like proactive time, like improving it so that like when there's a feature work site about like, we're just working on that feature, that's the thing you can schedule on.
00:59:36.777 -->
00:59:52.137I mean, did David, to your like, question about like funding, I mean the, the way I think I would say it, like particularly for the last 10 years is like, I think working on this project has been incredibly, like, positive some, and I, I think it's a concept that like doesn't get, it's due these days just in the sense of like.
00:59:52.836 -->
00:59:55.806We think we provided things that's like valuable to many companies.
00:59:55.887 -->
01:00:00.237and even though it's the case, like at least I have not been paid for the, to work on this in quite a while.
01:00:00.387 -->
01:00:02.067Like, I'm like very confident.
01:00:02.067 -->
01:00:18.896I've gotten lots of opportunities, whether it's to travel and give talks or meet people or just professional opportunities because like I work on this and like that, that is for me, like an exchange of like, you know, my time and like, for like remuneration that like really seems fair and like very positive sums.
01:00:18.896 -->
01:00:19.617So like, I, I don't know.
01:00:19.617 -->
01:00:20.036I don't know.
01:00:20.157 -->
01:00:20.936I don't have any complaints.
01:00:20.936 -->
01:00:21.867How you feeling Paul?
01:00:22.266 -->
01:00:28.987I mean, I think I found that like over the course of 13 years, like again, yeah, I got time for my employer when we first started.
01:00:28.987 -->
01:00:36.097I have not gotten time since I left that employer, so it has been 10 years roughly since I, I was, I was quote, paid to work on this project.
01:00:36.467 -->
01:00:42.907We, I mean, I think a lot of the sustainability for this is that, like Alex and I are a small team, right?
01:00:42.936 -->
01:00:50.737It's, it's two primary contributors to this project, which means that when we wanna make major decisions, this is not like a large scale, long-term effort.
01:00:50.737 -->
01:00:54.126It is a conversation between two people who largely think very similarly.
01:00:54.556 -->
01:01:03.876that means we can, we can make big decisions, we can execute against them, without a lot of, BEReaucratic time or a lot of like, politicking to determine what's good or bad.
01:01:04.086 -->
01:01:08.827Now, that has its disadvantages too, but so far inside this project that they've been largely advantages.
01:01:09.226 -->
01:01:18.646it is also the case that like we have built the project such that like when people come and they ask for something unreasonable, we feel comfortable pushing back.
01:01:18.976 -->
01:01:25.277Uh, we feel comfortable telling people they should not be, you know, they shouldn't behave certain ways in an issue tracker when they behave in ways that are inappropriate.
01:01:25.666 -->
01:01:26.206Uh, and we.
01:01:26.762 -->
01:01:28.952Built the system so that it's enjoyable to use, right?
01:01:28.952 -->
01:01:37.387Like, uh, one of the unofficial rules of thumb Alex and I have used for a long, long time now is anytime the CI exceeds 10 minutes, we spend time on it because it annoys us.
01:01:38.186 -->
01:01:42.356and so like those types of things just make it continuously pleasurable to work on.
01:01:42.586 -->
01:01:45.317it is also the case that it is a large bully pulpit, right?
01:01:45.317 -->
01:01:57.677Like we, we have a lot of influence and, and we are cognizant and respectful of the fact that we can use that bully pulpit, but we do want to be able to use it to like, pursue things we consider, worthwhile in the ecosystem.
01:01:57.782 -->
01:01:58.246Y yeah.
01:01:58.246 -->
01:02:01.036May maybe to really tie these two together.
01:02:01.257 -->
01:02:21.831I, I think for Paul and I, a really big indicator, what, one of the things that, at least for me, more than anything, led to like us giving this talk and writing this, statement about where OpenSSL was, is that we found that working with these new SSL APIs, looking at things like argon two or ML KEM, it, it was becoming like really unpleasant.
01:02:21.831 -->
01:02:29.161Like it, like if you had like a log of all of our chats, you would see like the profanity really went up.
01:02:29.161 -->
01:02:32.606Like we were experiencing a lot of fRustration and like.
01:02:33.306 -->
01:02:42.847That, that's like a pretty marked difference from, what came before, like that, that was just like, kind of not the emotional valence we had about, you know, adding new features before.
01:02:42.996 -->
01:02:45.516And so like that, that was a big signal to us.
01:02:45.516 -->
01:02:48.186Like, Hey, something has gone wrong.
01:02:48.186 -->
01:02:51.246Like we, we need to take stock of like, what's, what's happening here?
01:02:51.947 -->
01:02:55.336and so that basically leads you to the two of you.
01:02:55.396 -->
01:03:08.987You're generally on the same page about everything, and it's not that difficult for you to finally de declare and come to the conclusion that you basically have to move away from OpenSSL for at least some things for Python cryptography.
01:03:09.686 -->
01:03:15.487Yeah, so the, the, the core thesis of the statement we, we issued is that OpenSSL as a project has.
01:03:16.186 -->
01:03:28.947Slowly diverged toward, from, from paths that we find aesthetically appealing, technically acceptable, et cetera, uh, to the point where we are now actively seeking mechanisms by which we could end our dependence.
01:03:29.606 -->
01:03:31.496Now, that is a difficult thing to do.
01:03:31.527 -->
01:03:34.197There are a variety of reasons why, like compliance reasons.
01:03:34.197 -->
01:03:37.757We have downstream consumers that care deeply about the support of OpenSSL.
01:03:38.036 -->
01:03:41.577We have feature gaps based on like what, what the different forks support.
01:03:41.697 -->
01:03:50.137There's a bunch of things that make this challenging, but like while OpenSSL continues on its current trajectory, and to be clear, it is possible for them to course correct.
01:03:50.186 -->
01:03:53.887but it is a difficult thing, based on the fact that we've spent years advocating for it.
01:03:54.356 -->
01:04:06.396but if they fail to course correct, if they fail to, to, provide material improvement in the, on the axes that we've defined, then we will be trying to at the very least, remove OpenSSL as our default wheel configuration.
01:04:06.577 -->
01:04:13.391And potentially based on the, what Alex and I consider sustainable for ourselves in the long term OpenSSL entirely.
01:04:14.092 -->
01:04:17.686Do you have any other wheel configurations besides OpenSSL right now?
01:04:18.387 -->
01:04:23.007we support build against, LibreSSL, BoringSSL or aws-lc.
01:04:23.186 -->
01:04:24.146But you have to bring your own.
01:04:24.146 -->
01:04:26.927We don't, we don't, uh, ship, prebuilt, wheels for those.
01:04:27.297 -->
01:04:33.447But the, it, it's not difficult for y'all to just sort of start, experiment with just building those in.
01:04:33.476 -->
01:04:39.101'cause you already have the work to, to link, to hook them yeah, so we're perfectly capable of doing it right now.
01:04:39.132 -->
01:04:46.302The challenge is that like there's no, there's no concept of like wheel variance that would say like, oh, I want cryptography, but with a different backend.
01:04:46.492 -->
01:05:04.172and even if there was that concept, which there's actually some PEPs that might make that possible, even if that concept did exist, I think that's not a thing I, I can expect a consumer to understand the consequences of, Alex and I believe pretty deeply in having this library be the sort of like drop in, in just work and it has secure defaults.
01:05:04.382 -->
01:05:08.311And so I think like what we wanna do is get to a point where we would just swap the default.
01:05:09.012 -->
01:05:09.311Right.
01:05:09.311 -->
01:05:12.222So everyone is just like in, in to a first approximation.
01:05:12.521 -->
01:05:17.507Everyone's just gonna use the default, and if the default has feature gaps, then you're randomly gonna blow things up for people.
01:05:17.806 -->
01:05:18.416Exactly right.
01:05:18.447 -->
01:05:27.146So like for example, the current state of the state of the world is that we sh support and ship Argon2, ID and scrypt, both which are APIs that live natively inside of OpenSSL.
01:05:27.416 -->
01:05:35.586Not all the forks support either of those, and that meant if we, if we shipped a wheel with those, we would lose the support for those, algorithms immediately.
01:05:36.621 -->
01:05:40.972But like it's more likely that you'll like, this is kind of a normal engineering problem, right?
01:05:40.972 -->
01:05:43.251Like a flag day problem kind of deal, right?
01:05:43.282 -->
01:05:47.001It's more likely that you'll resolve that problem than that.
01:05:47.007 -->
01:05:49.487OpenSSL 3 is gonna get rid of O SSL param.
01:05:50.161 -->
01:05:53.536I won't try and speculate, but I, I, you might be right.
01:05:53.681 -->
01:05:53.972Yeah.
01:05:54.092 -->
01:05:56.492a betting person, but like, it, it seems plausible.
01:05:56.711 -->
01:05:57.612It seems plausible.
01:05:58.061 -->
01:05:58.362yeah.
01:05:58.961 -->
01:06:01.782It seems like one of us could just sign up for Poly Market and set this up.
01:06:02.181 -->
01:06:14.277That sounds like a long, long, but I mean, it seems like all you have to do is violate your first rule, which you also opened by saying you've already repeatedly violated, which is just like write a little bit of your own cryptography.
01:06:14.976 -->
01:06:19.581Uh, then, then I gotta get involved in FIPS more heavily, and that's already too much of my life.
01:06:19.882 -->
01:06:24.501Like you, you end up with things like if you actually like go inside the Phipps.
01:06:24.751 -->
01:06:26.702Not yet, hopefully sometime soon.
01:06:27.146 -->
01:06:36.202i, I If, and when NIST finishes their, starts, I, I lost track of the status of their, key derivation function work, but like in theory, maybe someday.
01:06:36.652 -->
01:06:43.297but like another example like Ed 4, 4 8 is I think not supported by any of the forks it is do you need it?
01:06:44.132 -->
01:06:48.016do We, whether or not it was wild wise, we, we did expose it at one point.
01:06:48.347 -->
01:06:49.217Oh Lord.
01:06:49.916 -->
01:06:52.407And like this is one of those things where like.
01:06:52.842 -->
01:06:59.802We could, we can make, we in fact have made the list of like, what are the things that, we ship that like we think are not stupid?
01:06:59.802 -->
01:07:06.282Like there, there's some stuff that like, we expose that like we don't care, like I, I don't know, SEC t whatever.
01:07:06.282 -->
01:07:09.851Like there's weird elliptic curves that like, if we dropped them, we don't think many people would care.
01:07:10.121 -->
01:07:11.351We, we would be okay with it.
01:07:11.501 -->
01:07:18.851But there's things like X ED 4, 4 8 or like, different kds or like, I think various like a EAD modes.
01:07:18.851 -->
01:07:26.152Like we've got a list somewhere that like any one of them, like you might look at and you might say, ah, how likely do we think it is?
01:07:26.152 -->
01:07:26.692People use this.
01:07:26.692 -->
01:07:30.501We can make an assessment of like, how much breakage would it cost to do this?
01:07:30.652 -->
01:07:31.791You know, what's the likelihood?
01:07:31.791 -->
01:07:33.472Are there standards that implement this?
01:07:33.472 -->
01:07:37.492Like, you know, if I search GitHub, does it like, seem like there's some projects that use it?
01:07:38.012 -->
01:07:38.251but.
01:07:38.952 -->
01:07:41.081For any of the given forks, we're looking at like, yeah.
01:07:41.141 -->
01:07:45.411Decently sized list of these like, I don't know, 10, 10 or so, algorithms.
01:07:45.771 -->
01:07:52.132And that's like, I don't know, 10 times the breakage budget of the, you know, mean of them.
01:07:52.282 -->
01:07:53.871So it's, it's potentially non-trivial.
01:07:53.871 -->
01:07:55.161So like Hmm.
01:07:55.461 -->
01:07:58.387we, we would like, we wanna spend our breakage budget.
01:07:58.387 -->
01:08:16.082Well, is I think the way, I would say it like we we're prepared to break things when we think like, the ecosystem, our users get like substantial benefits, but, we have a lot more ability to make changes like avoiding OpenSSL if we were reduce, uh, the level of breakage, that, that we incur in doing so.
01:08:16.381 -->
01:08:16.662Mm-hmm.
01:08:17.362 -->
01:08:28.046That does sound like, like you're incentivized to implement some of this stuff in Rust to paper over the migration, away from OpenSSL to another default backend.
01:08:28.777 -->
01:08:33.471That that really does, that does sound like what you're it's not impossible.
01:08:33.502 -->
01:08:54.051But there are, there, there are challenges there too because depending upon what we need to re-implement, like we don't want our downstream consumers to be silently surprised when, when the thing they thought was FIPS no longer can be, similarly, like there are, there are various like dependency requirements in, in the Rust cinematic universe of, of cryptography that are somewhat challenging for us in some scenarios.
01:08:54.442 -->
01:09:00.112and so like one of the things we mentioned in our, in our statement was, graviola a pure Rust cryptography library.
01:09:00.436 -->
01:09:02.537that is interesting to us in the abstract.
01:09:02.537 -->
01:09:04.516It's nowhere near where we would need it right now.
01:09:04.516 -->
01:09:15.266It's like we've spoken briefly with the author and like they're, that's not currently their focus for like, its adoption, but it's something that we're watching with interest as like maybe a long-term solution, as we go forward.
01:09:15.296 -->
01:09:23.016And when I say long-term, I mean like on the 5, 10, 15 year time horizon, Alex and I are genuinely thinking in the long term here.
01:09:23.466 -->
01:09:32.617because, and, and it's also why we spoke up when we did, because we gave it several years, but also knowing that it will take us many years to, to migrate off.
01:09:32.766 -->
01:09:37.207We want people to be aware of the problems rather than springing this on people at the last second.
01:09:37.612 -->
01:09:38.002Yeah.
01:09:38.002 -->
01:09:53.627Yeah, I, I mean, just in that space of like, you know, compliance questions and libraries, I, I do, I am very hopeful that, Filippo Valsorda's work on FIPS support for Go is going to prove as a model that is very valuable.
01:09:53.627 -->
01:10:07.556So, like, I would say in the, at least in the open source cryptography world, OpenSSL has been one of, it is like the most default choice, like maybe even more of a default choice than like OpenSSL, just as like open source cryptography is in general.
01:10:07.737 -->
01:10:12.716Like they, they had, you know, done the work to like build the, like back when it was called like the FIPS canister.
01:10:13.046 -->
01:10:22.346and I think Felipo has really demonstrated that it is possible if you were a diligent and knowledgeable cryptography maintainer maintainer to like take on this project.
01:10:22.396 -->
01:10:24.466is not, you know, impossible.
01:10:24.466 -->
01:10:25.516It's not all consuming.
01:10:25.606 -->
01:10:35.896It doesn't even require changing your library in that like, disturbing of ways, Go's cryptography modules are still kind of like a model of like clarity.
01:10:36.136 -->
01:10:36.327Yeah.
01:10:36.377 -->
01:10:44.577so like, I, I am hopeful that serves as like, a model that, like organizations that have felt like I only have one choice will look and be like, oh, it is actually possible.
01:10:44.577 -->
01:10:45.837There are other directions.
01:10:46.256 -->
01:10:58.077I, I really hope the way that he was able to get that working for Golan will be a, a model for like a future possible Rust, like alternative, like Rust does not have a standard library.
01:10:58.377 -->
01:11:04.596The way that Golan has a standard library, including all the cryptography that comes, including TLS, that comes with go.
01:11:04.867 -->
01:11:14.962But I could totally see a. Library project in Rust, trying to be validated as a FIPS module the same way that the the Go Lang one did.
01:11:15.171 -->
01:11:20.542It just would be its own project as opposed to a piece of the standard library of a language or something like that.
01:11:21.242 -->
01:11:27.931So I've always said, instead of rolling your own crypto, you should build it for someone else and then charge them for the FIP certificate.
01:11:28.631 -->
01:11:28.902Yeah.
01:11:29.652 -->
01:11:30.221cool.
01:11:30.341 -->
01:11:32.771Um, good luck in Godspeed.
01:11:32.771 -->
01:11:34.271You've come 10 years.
01:11:34.452 -->
01:11:35.622What's another 10?
01:11:35.652 -->
01:11:41.231To actually get away from the fundamental, underlying foundation of your project.
01:11:41.631 -->
01:11:46.521I'm sure Alex and I will, will be, complaining about this in our retirement, next to each other in the old folks home.
01:11:46.921 -->
01:11:48.002Looking forward to it.
01:11:48.702 -->
01:11:50.051Aw, that, that's sweet.
01:11:50.452 -->
01:11:53.992Tha thank you guys very much for introducing me to OSSL_PARAM.
01:11:54.012 -->
01:11:56.203we're, we're glad to help and thank you guys for having us.
01:11:56.203 -->
01:11:56.262No.
01:11:56.502 -->
01:11:57.193Absolutely.
01:11:57.943 -->
01:12:00.313Paul, Alex, thank you so much.
01:12:00.313 -->
01:12:05.533Security, cryptography, whatever is a side project from Dear De Connolly, Thomas Patrick and David Adrian.
01:12:05.773 -->
01:12:07.002Our editor is Netty Smith.
01:12:07.363 -->
01:12:07.943oh my God.
01:12:08.042 -->
01:12:09.028What's the rest of it?
01:12:09.087 -->
01:12:10.137I haven't done this in so long.
01:12:10.347 -->
01:12:16.198You can find the podcast online@scwpod and the hosts online@durumcrustulum, @tqbf, and @davidcadrian.
01:12:16.467 -->
01:12:19.717You can buy merchandise at merch dot security cryptography whatever dot com.
01:12:19.738 -->
01:12:23.097If you like the pod, give us a five star review wherever you rate your favorite podcast.
01:12:23.368 -->
01:12:24.297Thank you for listening.
