We Spent Decades Building Programming Languages. Now We Think They're Obsolete.

WEBVTT
00:00:00.000 --> 00:00:05.839
I'm pleased to be physically hosting you in a physically co-located in San Francisco edition of Computers, Coffee and Beer.
00:00:05.919 --> 00:00:08.160
This time, no coffee, no beer, but all computers.
00:00:08.480 --> 00:00:08.880
Decaf.
00:00:09.199 --> 00:00:09.839
Decaf copy.
00:00:09.919 --> 00:00:10.320
I guess it counts.
00:00:10.480 --> 00:00:10.720
That counts.
00:00:10.880 --> 00:00:12.240
It's a pleasure seeing you, my old friend.
00:00:12.480 --> 00:00:13.679
Uh very good to be here.
00:00:13.839 --> 00:00:14.880
Yeah, thank you for being here.
00:00:15.039 --> 00:00:16.399
There's tons of stuff we could talk about.
00:00:16.480 --> 00:00:21.039
We are we're physically here in Pebble Bed Intergalactic HQ in San Francisco's mission district.
00:00:21.199 --> 00:00:23.359
Um we're investors who build.
00:00:23.440 --> 00:00:24.719
You're the CEO of Skip Labs.
00:00:24.800 --> 00:00:26.320
You've got a million things going on.
00:00:26.480 --> 00:00:34.399
Um, but our default equilibrium over the last couple decades as a you know aficionado of programming languages and a programming language professional.
00:00:34.479 --> 00:00:36.560
Um usually talk PL.
00:00:37.119 --> 00:00:48.479
And uh one of the things that I find myself really confused about these days is what it means to do programming language research or or to build new programming languages in this kind of post-LLM world, right?
00:00:48.719 --> 00:00:55.119
Because I've made sense of programming languages as an intellectual discipline, as a kind of applied psychology so far, right?
00:00:55.200 --> 00:01:04.079
You're you're bridging this gap between these very inhuman machines and the kind of limited psychologies of human beings, and we have our strengths as well.
00:01:04.159 --> 00:01:08.079
Um, and you're trying to translate those strengths of humans into the these machines.
00:01:08.319 --> 00:01:12.319
And now a lot of the code is going to get written by very different psyches, it seems like.
00:01:12.560 --> 00:01:14.560
And does that change PL?
00:01:14.719 --> 00:01:15.840
Should it change PL?
00:01:16.000 --> 00:01:17.200
Does anybody know?
00:01:17.599 --> 00:01:18.640
I think nobody knows.
00:01:18.719 --> 00:01:19.599
Let's start with that.
00:01:20.079 --> 00:01:24.560
Then I think um there will probably be different phases, right?
00:01:24.719 --> 00:01:31.359
And I think phase one, uh, what's going to matter is going to be where's the data coming from.
00:01:31.599 --> 00:01:35.840
So right now you see an uptick in TypeScript, for example.
00:01:36.000 --> 00:01:38.959
And that's because LLMs are very good at TypeScript.
00:01:39.200 --> 00:01:46.959
And then it's uh it's you know uh it's it's this virtuous or vicious circle, depending on how you want to look at it.
00:01:47.200 --> 00:02:03.439
Positive feedback where um where the more code there is out there that is written in TypeScript, the more you know AIs are gonna pick up that data and are gonna be trained on that data, and the more and the more there will be um TypeScript out there.
00:02:03.760 --> 00:02:10.240
Now, the two languages that seem to be really doing well with LLMs, the Python and TypeScript.
00:02:10.400 --> 00:02:14.400
I could see how I think TypeScript is gonna win that race.
00:02:14.560 --> 00:02:17.919
Uh, and that's because TypeScript is faster.
00:02:18.080 --> 00:02:22.800
It has a JIT that makes you know the the execution kind of reasonable.
00:02:23.039 --> 00:02:23.759
I've been briefed, yeah.
00:02:23.919 --> 00:02:24.719
Uh you've been briefed.
00:02:24.960 --> 00:02:25.759
You've heard about this stuff.
00:02:26.159 --> 00:02:27.360
This JIT stuff's gonna be hot.
00:02:27.680 --> 00:02:28.319
You heard it here.
00:02:28.800 --> 00:02:35.759
And the other thing is that it has uh somewhat some tooling around types.
00:02:35.919 --> 00:02:37.919
So there's a culture of typing.
00:02:38.159 --> 00:02:49.919
Um TypeScript is not sound, so that means there are a lot of things that you cannot do, but you can still describe intent in terms of types a lot better than you would in Python.
00:02:50.719 --> 00:03:01.199
And those two will I think give enough uh an edge to TypeScript to beat, you know, to win that race between Python and TypeScript.
00:03:01.520 --> 00:03:03.439
I mean, do you think do you think that's the equilibrium?
00:03:03.520 --> 00:03:08.080
It's just sort of TypeScript eats the world and I think for now, yes.
00:03:08.319 --> 00:03:15.919
As long as LLMs um need a lot of data to be trained, um, I think TypeScript is gonna eat the world.
00:03:16.159 --> 00:03:30.879
But it might not stay this way forever, because what's going to end up happening at some point, I think, is that LLMs are gonna become good enough at programming in general that the programming language probably will not matter as much.
00:03:31.360 --> 00:03:35.919
Uh and when we get to that point, that's when things are going to become really interesting.
00:03:36.240 --> 00:03:48.319
So, one of my dreams here, if I can, if I may, because like I'm impatient to start banging this drum, is like, why hasn't this made a big fluorescence in C and other kinds of um you know machine-friendly languages, let's say, right?
00:03:48.479 --> 00:03:56.479
Because if your story, you know, the sort of age-old push-pull inside of a company was always sort of C versus your programmer productivity languages.
00:03:56.639 --> 00:04:14.000
And the programmer productivity language people, right, the people writing PHP or the people writing Python, the people writing uh TypeScript these days, or whatever your kind of productivity-oriented language is, they do lay a lot more track than the people sitting there kind of, you know, puzzling over 20 pages of template errors and a compile failure, right?
00:04:14.560 --> 00:04:18.079
But if I'm an LM and I love type systems anyway, right?
00:04:18.160 --> 00:04:20.160
Type systems are just sort of awesome feedback for me.
00:04:20.319 --> 00:04:21.839
I eat them for breakfast.
00:04:22.240 --> 00:04:27.839
Why not also do something that's generating amazing machine code the first time, uh, like C.
00:04:28.879 --> 00:04:30.560
It's a little bit of a mystery to me.
00:04:30.639 --> 00:04:35.600
I think it really boils down to data that isn't source code.
00:04:35.839 --> 00:04:52.240
So I think basically, and I'm just speculating here because it's not my job to train these things, but I would imagine that what's very what's going to be very useful is annotated data where you have small snippets with somebody explaining what this thing does.
00:04:52.560 --> 00:05:16.160
And you know, with something like Stack Overflow, um, most of the data was related to TypeScript and Python because these are programming languages where you can actually program this way, where you're trying to put something together quickly, you need feedback, um, you don't know how to do something, you type it in Google, then Stack Overflow.
00:05:16.319 --> 00:05:21.759
Well, now they're you copy a chat GPT, but you know, back, you know, pedal back, you know, 10 years.
00:05:22.240 --> 00:05:23.040
What was Stack Overflow?
00:05:23.360 --> 00:05:23.839
Exactly.
00:05:24.319 --> 00:05:32.000
Um, and so there's probably someone on Stack Overflow who already discussed that, and there's a snippet of code that you can copy paste, right?
00:05:32.319 --> 00:05:40.240
For future listeners, possibly on a lunar base or whatever's gonna happen in the next 18 months, Stack Overflow was a was a seat do web.
00:05:40.480 --> 00:05:52.480
It was a it was a HTTP service where uh people would ask questions, usually about the programming problem they're having, and people would say there was like a social gamified aspect of like trying to answer lots of questions.
00:05:52.560 --> 00:05:56.000
And so people would compete to have highly ranked answers to these questions.
00:05:56.160 --> 00:06:08.879
And so you would Google for you know the problem you're seeing error message you have or the thing you're trying to do or something, and there'd be some little snippet of like, oh, here's how you you know handled this edge case in parsing YAML and blah blah, whatever.
00:06:09.279 --> 00:06:09.759
Exactly.
00:06:10.000 --> 00:06:21.519
So chances are very high that if you're trying to do anything in JavaScript, you basically type it with words, and there will be someone who is going to explain to you how to do it and give you a snippet of code.
00:06:21.600 --> 00:06:21.920
Right.
00:06:22.160 --> 00:06:27.040
And I think that is going to be much harder or impossible in C.
00:06:27.680 --> 00:06:27.839
Right.
00:06:28.000 --> 00:06:32.319
Because if you program this way in C, it's never gonna work.
00:06:33.279 --> 00:06:35.360
I mean, it's it's probably gonna be worse than that.
00:06:35.439 --> 00:06:39.199
It's gonna look like it's working, yeah, and really it isn't.
00:06:39.360 --> 00:06:50.959
Yeah, you know, I've I've seen that a lot where you know, when you you watch somebody experiencing C view C code, and every single line they're like undefined behavior, undefined behavior.
00:06:51.120 --> 00:06:53.360
Oh, I see another undefined behavior right there.
00:06:53.759 --> 00:07:07.600
So I I if I had to speculate, because then again, I I was not involved in the process of training those things, I would imagine that it has a lot to do with data on the internet uh that is available to train the model.
00:07:07.839 --> 00:07:10.800
I mean, this is a I do have a bunch of like a lot of other people, right?
00:07:11.120 --> 00:07:14.800
I spent you know last winter finishing all my side projects in GitHub, basically, right?
00:07:14.879 --> 00:07:17.120
And being blown away that Opus 4.5 could do it.
00:07:17.279 --> 00:07:18.720
It's gotten a lot better since then.
00:07:18.800 --> 00:07:20.160
It wasn't that long ago.
00:07:20.399 --> 00:07:23.600
Um definitely they're pretty good at writing C now.
00:07:23.759 --> 00:07:34.079
And I and I think one of the things that I was surprised at, you know, my shrinking margin, my shrinking little island of like stuff that only Keith can do with computers or whatever over the last couple years.
00:07:34.319 --> 00:07:40.959
Uh a lot of the stuff that was defended there until then was lock-free data structures in C.
00:07:41.439 --> 00:07:42.000
Do you know what I mean?
00:07:42.079 --> 00:07:47.600
Like trying to do correct stuff with STD Atomic where it really is concurrent and a really state machine.
00:07:47.680 --> 00:07:54.399
I think until like about Opus 4.5, it was as if the LLM didn't really think about concurrency and sort of write a serial version that would work.
00:07:54.480 --> 00:07:59.199
And it like just clearly wasn't really thinking about races or anything like that or hazards or anything like that.
00:07:59.439 --> 00:08:01.279
Um it does all that now.
00:08:01.360 --> 00:08:03.680
It does all of it sort of reasonably marvelously.
00:08:03.759 --> 00:08:06.160
Like there will be bugs, you do have to review it.
00:08:06.399 --> 00:08:11.920
Um, but it's more like, you know, a colleague on a pretty senior colleague on your team, honestly, right?
00:08:12.000 --> 00:08:18.160
Because like none of us write defect free code, a sort of defect rate of human written C is estimated at one every 10 lines.
00:08:18.399 --> 00:08:21.839
It feels higher to me, honestly, than than one, you know, one line in 10.
00:08:21.920 --> 00:08:22.720
That's a defect.
00:08:22.879 --> 00:08:35.840
Uh and so even for the data poverty, you know, whatever's going on with sort of abstract reasoning and RL and you know, maybe some of these companies have a big have big C corporate that they don't share that that make this easier.
00:08:36.000 --> 00:08:40.879
Um but I'm pretty impressed for if you'll grant me my premise, basically, right?
00:08:40.960 --> 00:08:46.000
Like grant me my premise that uh that this generation of LMS is also awesome at writing C.
00:08:46.960 --> 00:08:52.879
So it could I think there's a big difference between writing from scratch and modifying something existent, right?
00:08:53.360 --> 00:08:59.759
If we're talking about writing something from scratch, Python and TypeScript are just really a step above.
00:08:59.840 --> 00:09:10.559
Yeah, if you're talking about looking at an existing code base and modify it, LLM is good at a lot of languages, you can use them to do a lot of things.
00:09:10.879 --> 00:09:22.559
But really from scratch, I would go with TypeScript or Python, and also depending on the size of the project that has to be dealt with from scratch.
00:09:22.720 --> 00:09:27.440
So for example, if I had to write one function in C, that would work really well.
00:09:27.600 --> 00:09:27.759
Yeah.
00:09:28.000 --> 00:09:48.320
If I had to write, tell the NLM to write me an entire project with let's say 10,000 lines of code in C, I what I've seen is a big difference between a project, even with other languages like Go, um, it seems to be working a lot better in uh TypeScript or Python.
00:09:48.559 --> 00:09:48.799
Yeah, yeah.
00:09:48.960 --> 00:09:52.159
And uh you may have tried more things than me too, but uh that's interesting.
00:09:52.240 --> 00:09:56.639
I actually thought you were gonna go the other direction with this, that it's like the big existing project that it struggles with.
00:09:56.720 --> 00:10:12.399
Uh but no, I mean it's um it struggles with yeah, it struggles still with understanding what the large project is doing, and uh also it can introduce really subtle bugs because it doesn't understand the environment of the bigger project.
00:10:12.559 --> 00:10:14.240
But this is all a given.
00:10:14.399 --> 00:10:28.240
But I was more saying it's not so much that LLMs have a good time or hard time working on big projects, it's more like the comparison between the the difference in in performance between TypeScript and C.
00:10:29.200 --> 00:10:33.120
You can see it a lot more on a project from scratch.
00:10:33.440 --> 00:10:36.320
So let's let's imagine that you're right of you're right about this.
00:10:36.480 --> 00:10:38.000
I find it's pretty persuasive, to be clear.
00:10:38.080 --> 00:10:43.440
This is like matches my experience as well as it seems just want to write TypeScript and Python to some extent.
00:10:44.080 --> 00:10:45.919
They divide up the pi between them, right?
00:10:46.000 --> 00:10:52.720
Like like most new projects get built in one or the others of these languages, unless you've got some burning need to build it, you know, to the metal for some reason.
00:10:53.120 --> 00:10:59.519
Do you think um do you think TypeScript evolves in ways that are LLM dependent?
00:10:59.679 --> 00:11:02.159
And if so, what are those ways, right?
00:11:02.240 --> 00:11:07.279
Are there are there things the LLMs like wish they had in the language, so to speak?
00:11:07.440 --> 00:11:08.879
Uh I think it will get to that.
00:11:09.279 --> 00:11:12.240
I think it will get to that for many reasons.
00:11:12.480 --> 00:11:17.519
The first one is we're going to run into probably into security problems.
00:11:18.159 --> 00:11:21.360
Um TypeScript already has security problems.
00:11:21.600 --> 00:11:31.440
I mean JavaScript community in general already has security problems because there are a ton of packages out there and you know they're evolving very, very fast.
00:11:31.679 --> 00:11:35.200
And it's a culture of programming that's very different from C or C.
00:11:36.240 --> 00:11:41.519
It's a culture of programming where taking on new dependencies is not frowned upon.
00:11:42.000 --> 00:11:49.200
When you're writing a C project, every single dependency is very carefully thought about, and C as well.
00:11:49.360 --> 00:12:00.000
And that's because um the interaction between those libraries and your own code can lead to very subtle bugs and can lead to things that are very difficult to debug and maintain.
00:12:00.320 --> 00:12:04.080
And so that's why people are very careful about those dependencies.
00:12:04.399 --> 00:12:07.600
In the world of Python and TypeScript, not at all.
00:12:07.840 --> 00:12:13.840
Like whenever you need something, just install the package, and then that package installs another gazillion packages.
00:12:14.000 --> 00:12:19.200
And before you know it, you depend on a ton of code, and some of it is vulnerable.
00:12:19.519 --> 00:12:27.679
And I think what's going to happen is that this problem is going to become exponentially larger for two reasons.
00:12:27.919 --> 00:12:33.600
One, we're going to see a lot more code, and the LLMs will also have that kind of instinct.
00:12:33.679 --> 00:12:36.480
They will install whatever they need to get the stuff.
00:12:36.799 --> 00:12:37.600
That's what they've seen, right?
00:12:37.679 --> 00:12:40.399
That's what they've learned to do to get functional programs.
00:12:40.799 --> 00:12:57.840
Number two, we see models, especially like ones like Mythos now, who are really good at injecting, you know, finding security holes, injecting them, and people are going to use that to find new holes and find new ways to um, you know, hack into the system.
00:12:58.000 --> 00:12:58.240
Yeah.
00:12:58.480 --> 00:13:14.639
So I think just for that already, the system will have to evolve because there will be pressure to maintain that ecosystem in a in a place that is sane, yeah, to bring in tooling and guardrails for for those system.
00:13:15.519 --> 00:13:18.559
So let me try and interrogate this like in so okay.
00:13:20.799 --> 00:13:26.480
I have two completely opposite and incompatible with each other reasons that might not turn out to be the case, right?
00:13:26.559 --> 00:13:37.919
So in one scenario, almost everything about the way the software world is set up uh is is there to maximize code reuse because the prior is software is precious, right?
00:13:38.159 --> 00:13:43.759
It took the sweat of some poor programmer in San Francisco's brow to get this thing to work in the first place.
00:13:43.919 --> 00:13:46.799
You'd be a fool to think you can do better in most instances.
00:13:47.039 --> 00:13:47.679
So don't.
00:13:47.919 --> 00:13:50.559
You just import the thing, whatever.
00:13:50.879 --> 00:14:01.120
Um in a world where software is not cheap, or software is not precious, rather, and where you can throw everything out and start over again whenever you feel like it, why all the code reused?
00:14:01.360 --> 00:14:04.960
Like, like why not, you know, why not make a little JSON parser when you need to parse some JSON?
00:14:05.120 --> 00:14:07.679
And why not make a little YAML parser when you need to parse some YAML and so on?
00:14:08.000 --> 00:14:09.360
Still cheaper to share.
00:14:09.519 --> 00:14:14.720
You know, it's it's it's like uh imagine I made the same argument in programming, right?
00:14:14.879 --> 00:14:17.200
I told you, you know what, RAM is cheap.
00:14:17.440 --> 00:14:24.240
Uh why, you know, reuse some things that you've computed when and not just record so obviously there are trade-offs.
00:14:24.480 --> 00:14:26.879
And output tokens do cost something, it turns out too.
00:14:26.960 --> 00:14:27.440
So yeah, yeah.
00:14:27.759 --> 00:14:29.600
So I think there are trade-offs.
00:14:29.679 --> 00:14:34.799
There are cases where there are these algorithms on you know, SAT solvers and things like that.
00:14:35.039 --> 00:14:40.000
And um, I I used to be really interested in those in the early 2000s.
00:14:40.480 --> 00:14:46.320
And my instinct as a functional programmer was always, well, these things need a lot of backtracking, right?
00:14:46.559 --> 00:14:52.399
So it needs to try something, and then once it is tried, it needs to go back into the state where it was to try something else.
00:14:52.559 --> 00:14:56.480
So it's like obviously you should be using persistent data structure for this stuff.
00:14:56.720 --> 00:15:04.960
And I tried, you know, getting some of those algorithms, like you know, um Davis Putman and these kind of things to work with functional data structures.
00:15:05.679 --> 00:15:11.039
And it was way, way, way slower than undoing the work you'd done.
00:15:11.360 --> 00:15:12.399
Right, and just redoing it.
00:15:12.559 --> 00:15:20.879
And just redoing the other, well, uh minus one basically of your operation and then go back to do to to try something else.
00:15:21.120 --> 00:15:22.480
Super frustrating for me.
00:15:22.639 --> 00:15:31.600
I was like, so really this whole persistent data structure makes you know taking making checkpoints cheap is yeah, it's cool in theory.
00:15:31.679 --> 00:15:38.080
I mean, no, there are some places where it works really well, but there are some others where the trade-off is just not good.
00:15:38.240 --> 00:15:43.600
So I definitely think that for LLMs and code, there will be things like that.
00:15:43.679 --> 00:15:46.720
There will be moments where the LLM thinks, you know what?
00:15:46.879 --> 00:15:50.960
It's going to be too complicated for me to figure out how to set up this or that.
00:15:51.039 --> 00:15:53.600
So let me just regenerate the code that does this.
00:15:53.840 --> 00:16:00.960
But in general, no, I think you still want to share code because tokens are not free.
00:16:01.120 --> 00:16:09.519
They might uh they might be cheaper than a human being, you know, typing things with their little fingers, but they're still not not free.
00:16:09.840 --> 00:16:10.399
Yeah.
00:16:10.720 --> 00:16:14.879
There's a uh and then I I guess the opposite, I think you're right, probably, right?
00:16:15.039 --> 00:16:24.799
Um, I think there's a one of the things that people seem to get confused about when talking about artificial intelligence in general, especially sort of extreme extrapolations of it, right?
00:16:24.879 --> 00:16:35.200
Like way better AIs than we have today, is to assume that uh just because it's very smart and and perhaps also very plentiful and getting more plentiful, that it's literally free.
00:16:35.519 --> 00:16:36.080
You know what I mean?
00:16:36.159 --> 00:16:39.600
Like there will always at least be the energy required to run the model and so on.
00:16:39.759 --> 00:16:44.320
And so you will have to kind of out the other users of those resources to get your thing through.
00:16:44.399 --> 00:16:51.279
So there still is gonna be an economy of effort, even if the intelligence is way, way better and and way more abundant than we can imagine right now.
00:16:51.440 --> 00:17:01.039
Um so I think it's still gonna matter to save effort, and for that reason, there will still be code reuse just because like we're using salt problems as useful to artificial intelligences too.
00:17:01.360 --> 00:17:04.960
At Slack, so you guys uh had a hack conversion.
00:17:05.359 --> 00:17:05.599
We did.
00:17:05.680 --> 00:17:14.720
We went through a we did go through a uh so hack kind of is to is to uh to PHP as TypeScript is to JavaScript, if you'll allow me to say so.
00:17:15.119 --> 00:17:18.960
Although you got there first for the record, because you're probably too modest to say it.
00:17:19.359 --> 00:17:24.480
Um but as a gradual typing system that's that's sort of oriented around how PHP programs tend to be written.
00:17:24.640 --> 00:17:32.240
Um and Slack, a little bit like Facebook at the time I started, was a grown-up Lamp Stack app, was uh you know, Apache MySQL PHP.
00:17:32.640 --> 00:17:43.839
Um the founder, one of the founders, Cal Henderson, had had a bunch of success with that at a previous startup, Flickr, right, when the consumer photo sharing site on the internet um basically saw no reason to change that.
00:17:44.079 --> 00:17:49.359
And around the time I started as as chief architect in 2016, you know, we're starting to see some.
00:17:49.839 --> 00:17:50.880
Wait, chief architect.
00:17:51.039 --> 00:17:51.759
I love these titles.
00:17:51.920 --> 00:17:52.240
Sorry, sorry.
00:17:52.480 --> 00:17:53.359
I love these titles.
00:17:53.599 --> 00:17:56.559
I I'm gonna can I call you chief archite architect for it.
00:17:56.720 --> 00:17:56.960
Yeah.
00:17:57.440 --> 00:17:57.839
Oh well.
00:17:59.119 --> 00:18:00.079
I love those titles.
00:18:00.559 --> 00:18:05.279
It makes me uh think of, you know, um uh what what what religion is this?
00:18:05.359 --> 00:18:06.880
Is it Franc-Maçon who have that?
00:18:07.200 --> 00:18:16.400
Oh, the the the the Han Masons, yeah, the big architect, you know, and so I I it reminds me of that, so that's why I always find those titles funny.
00:18:16.480 --> 00:18:17.759
But please, no, no, no, it's fine.
00:18:18.160 --> 00:18:30.160
I had a weird, I grew up in a Marine Corps household, by the way, and like the modifiers you put in front of like enlisted ranks pile up very similarly to the modifiers you put in front of individual contributor titles, basically, right?
00:18:30.240 --> 00:18:31.839
Like if you're a staff engineer, obviously.
00:18:32.000 --> 00:18:34.240
Ah, yes, there's that true senior this, senior.
00:18:34.640 --> 00:18:36.640
Can I be a public static staff engineer?
00:18:36.880 --> 00:18:38.720
Yeah, can you be a senior staff chief?
00:18:39.200 --> 00:18:40.000
Whatever, right?
00:18:40.319 --> 00:18:50.480
Um, so I mean though the way that we've like I when when that offer came through initially, like when Cal kind of sat down and negotiated like what that meant in the context of Slack, like can mean different things in different places.
00:18:50.640 --> 00:18:57.519
Um and when we kind of like workshop this definition that was like, I'm accountable for Slack's technical strategy, right?
00:18:57.599 --> 00:19:00.720
So I'm accountable in the sense that like I'm prepared to give account, right?
00:19:00.880 --> 00:19:07.440
Like there are other parts of the company that want things, and and when things break, I need to be able to articulate what happened and what we're changing.
00:19:07.599 --> 00:19:22.240
Um and then the technical strategy part of it is like I need to be kind of aware of what the heck we're trying to do, what what you know business challenges we're trying to address in the medium term, and have the tech roadmap at least align with and hopefully, you know, anticipate and improve that course of action.
00:19:22.400 --> 00:19:26.799
I'd say like in hindsight, um I give myself like a B minus or something.
00:19:26.880 --> 00:19:33.279
I like I think like I didn't, you know, I don't think I drove the Ferrari into the ditch completely, but like uh there are definitely things I would do differently.
00:19:33.440 --> 00:19:41.119
But the hack conversion, uh since since you're asking the PL question, I don't know if we've like ever sort of gone blow by blow through much of this, right?
00:19:41.519 --> 00:19:50.319
Um around the time I was starting in 2016, some of the problems that you tend to run into in big PHP code bases were starting to show up, right?
00:19:50.400 --> 00:19:52.319
And so So how big was the code base?
00:19:52.720 --> 00:19:54.480
Oh, uh good question.
00:19:54.799 --> 00:19:56.640
Uh order of a million lines.
00:19:57.200 --> 00:19:57.440
Right.
00:19:57.599 --> 00:20:00.720
So uh in the hundred to million order magnitude, I'm confident.
00:20:00.960 --> 00:20:09.200
And it was running HHVM already or it was running PHP, it was running Zen, PHP 5, I think we were stuck on 5.3 at the time, 5.4 maybe.
00:20:09.440 --> 00:20:14.559
Um and uh and changing PHP versions was a whole thing and so on.
00:20:14.720 --> 00:20:22.319
Um so if you've never worked on, and I guess people these days like might have had their whole careers kind of in a TypeScript code base or something like that.
00:20:22.480 --> 00:20:28.640
If you've never worked on a big, entirely dynamically typed code base, it's really, really daunting, right?
00:20:28.880 --> 00:20:32.000
If if you've never done it, you good for you.
00:20:32.240 --> 00:20:32.720
Yeah, yeah.
00:20:33.119 --> 00:20:34.720
Good for you, you should be happy about it.
00:20:34.960 --> 00:20:35.359
That's right.
00:20:35.440 --> 00:20:35.759
That's right.
00:20:36.079 --> 00:20:37.680
And strive never to have to do it.
00:20:37.839 --> 00:20:38.799
But yes, it's daunting.
00:20:39.119 --> 00:20:52.319
So let's say, for instance, you um, you know, the code base written nine times out of ten does the order of arguments, you know, user ID, then permissions object, then you know, object you're acting on or whatever, right?
00:20:52.559 --> 00:20:55.039
And there's just like some place in the code base that did it some other weird way.
00:20:55.359 --> 00:20:56.240
What about test coverage?
00:20:56.319 --> 00:21:05.599
Because usually that's what determines if it's possible to it, you know, work in a big dynamic language code base.
00:21:05.759 --> 00:21:06.799
So, how was the test coverage?
00:21:07.039 --> 00:21:10.480
Test coverage, I think around the time we were talking mostly in terms of line coverage.
00:21:10.640 --> 00:21:17.599
Um, it was order of 20%, and we had sort of a continual like quarterly OKR around increasing test coverage all the time.
00:21:18.240 --> 00:21:25.759
This ended up sort of uh making me cynical about the sort of Goodhart's law properties of tracking that metric too too closely, right?
00:21:25.839 --> 00:21:30.880
So, Goodhart's law, of course, right, is like when you make something, when you attach incentives to things, it stops being a useful metric.
00:21:30.960 --> 00:21:31.359
Um, right.
00:21:31.440 --> 00:21:40.240
So if we start attaching sort of the proxies of success, including like raises and you know, getting pats on the head from people you respect to test coverage.
00:21:40.400 --> 00:21:45.279
Well, one way to cover the code is to like have all unit tests just be a trace of the implementation, right?
00:21:45.440 --> 00:21:47.680
Like Yeah, people are gonna gain the system.
00:21:47.839 --> 00:21:48.000
Yes.
00:21:48.559 --> 00:21:51.519
So the the the thing is like I'm not I actually that'd be one thing.
00:21:51.599 --> 00:21:57.599
It's one thing if people are cynical and they're just like, ah, I'm just gonna get my bonus and blah blah blah and write these bad tests.
00:21:57.759 --> 00:21:59.039
That's actually not what's going on.
00:21:59.200 --> 00:22:17.680
I think it's actually A more subtle thing where if you sort of drop testing culture on an engineering culture that hasn't had it before, people are like, okay, we're supposed to write tests, and they don't really have like the um I guess the sort of interior model of like what's desirable about like what makes a test good or bad.
00:22:17.839 --> 00:22:25.440
Um, I think there's that, and there's also the tools that kind of shape how you see the world and the programming language as well.
00:22:25.680 --> 00:22:28.559
So this is, you know, I've I've I've sorry to go on a tangent.
00:22:28.640 --> 00:22:31.839
Yes, I've been in in in these debates a million times, right?
00:22:32.079 --> 00:22:46.640
Like you've got a dynamic programmer who tells you, well, I don't need a type system because I don't make typing mistakes, and anyway, I write the tests, I have to write tests, and the tests are gonna catch the typing mistakes that I would be making.
00:22:46.799 --> 00:22:50.880
So, really, you know, the the the type system is kind of wasting my time.
00:22:51.039 --> 00:22:51.279
Right.
00:22:51.440 --> 00:22:59.759
And you know, to somebody who comes from a language like Java or C or whatever, that seems to be incredible to hear.
00:22:59.839 --> 00:23:01.519
You're like literally not credible, right?
00:23:01.920 --> 00:23:03.119
Yeah, literally not credible.
00:23:03.279 --> 00:23:05.599
They're they're thinking, you are lying to me.
00:23:05.759 --> 00:23:10.880
Like you need the type system, I use it all the time, it catches mistakes for me.
00:23:11.119 --> 00:23:18.400
And the thing is, the tool that you use shapes the way you think and also the way you see the world.
00:23:18.720 --> 00:23:35.519
Um, so if you are using you know dynamically typed language all the time, what is going to happen is that you know that things are gonna fail all over the place, and you you will write code with tests that are very defensive.
00:23:35.759 --> 00:23:37.759
So you're gonna really test everything.
00:23:37.920 --> 00:23:44.960
And if that's your style of programming, and I give you Java, you're not gonna change your style.
00:23:45.359 --> 00:24:04.240
So you're gonna write all these unit tests, you're gonna check for all the types of everything, and then you know what you're saying about Java, which is that the type system is a pain in the ass that gets in the way, is true for you because you're actually in the habit of writing all those unit tests, right?
00:24:04.480 --> 00:24:06.240
And the same is true the other way around.
00:24:06.400 --> 00:24:20.799
So if you take somebody like who comes from a statically typed language, they're used to not test the types for the input parameters or things like that, because all of this is checked by the system.
00:24:21.119 --> 00:24:32.960
So they come into Python and they don't change their style of programming, and they realize that you know things break all the time and things don't work the way the way they're supposed to.
00:24:33.200 --> 00:24:46.319
Now, I would argue that in a large code base, there will always be a large enough fraction of people who are not disciplined enough that every dynamically typed code base I've seen is bad.
00:24:46.799 --> 00:24:52.400
Like I I I've seen small projects where this works because they're disciplined enough to make it work.
00:24:52.640 --> 00:24:55.119
But in larger code doesn't scale, right?
00:24:55.200 --> 00:25:00.160
Like just saying like be a good person, care a lot about the code, being right, like that.
00:25:00.240 --> 00:25:02.559
You can't enforce that in a huge organization.
00:25:02.880 --> 00:25:05.359
And I mean, I've I've seen that in other places too.
00:25:05.440 --> 00:25:08.480
You know, there's the functional versus OO stuff.
00:25:08.720 --> 00:25:23.839
So I've seen that a lot where you see people fight because they see the world in different ways, and they don't realize that the tools that they use and the problems that they use to solve has shaped their understanding of programming.
00:25:24.000 --> 00:25:28.640
Another good divide is system programmers versus non-system programmers.
00:25:28.799 --> 00:25:30.720
So let me give you a few examples.
00:25:30.880 --> 00:25:35.759
So if you are a system programmer, you see the world systems programming.
00:25:35.920 --> 00:25:36.799
Yeah, we're growing, yeah.
00:25:37.759 --> 00:25:40.480
Uh you you see the world through memory, right?
00:25:40.559 --> 00:25:47.519
Like what you see in your head when you're programming is memory because the memory layout is basically everything.
00:25:47.680 --> 00:25:58.640
Like, this is what you have to care about because that's how you're gonna write a driver, that's how you're gonna write a garbage collector, that's how you're gonna write all that stuff, you know, or a kernel, all this stuff.
00:25:58.799 --> 00:26:01.200
You need to know exactly how memory is laid out.
00:26:01.359 --> 00:26:05.039
So that is what you see in your hand when you're programming, right?
00:26:05.279 --> 00:26:10.079
And so now somebody comes along and tells you, yeah, we're gonna abstract that all the way.
00:26:10.160 --> 00:26:12.960
You know, this is gonna be called prologue or hassle.
00:26:13.519 --> 00:26:21.759
And and you just, you know, define what you want it to do, and and then the system programmer goes like, that's not programming, like this is light.
00:26:21.839 --> 00:26:22.319
Yeah, exactly.
00:26:22.480 --> 00:26:23.279
Like this is wrong.
00:26:23.599 --> 00:26:28.880
It feels like you're yeah, nobody can tell me like what what what machine instructions correspond to any of this?
00:26:28.960 --> 00:26:39.920
Like, and so the system programmer comes in and you give them a problem, and what they're going to do is think in terms of memory layout, and then they're going to structure things this way, right?
00:26:40.079 --> 00:26:45.200
And so if you give them a garbage collector, it's they're like, This is already too much, right?
00:26:45.279 --> 00:26:46.319
I I want to know more.
00:26:46.480 --> 00:26:51.039
But you see that divide also with uh functional programming versus old programming.
00:26:51.119 --> 00:26:55.359
So, for example, to a functional programmer, everything is about types.
00:26:55.599 --> 00:27:00.319
You need to know the types of things, and that's what's going to shape your understanding of the program.
00:27:00.960 --> 00:27:11.440
Once you know the types, once you have the signatures of the functions, the the functional programmer kind of feels like, yeah, this is this is my understanding, my abstract understanding of the program.
00:27:12.000 --> 00:27:15.759
Except that there are different ways to abstract the same program.
00:27:15.920 --> 00:27:20.480
So in the OO world, they don't care about types, they care about functionalities.
00:27:20.960 --> 00:27:28.160
So what they want is they want to know what is the API, what can I call, what can they do with this thing?
00:27:28.400 --> 00:27:31.839
But what the shape of this thing is doesn't matter to them.
00:27:31.920 --> 00:27:33.200
That's abstracted away.
00:27:33.279 --> 00:27:46.880
That is, and so when you see those two kinds of people trying to um basically um they switch to a different world, you see exactly the same frustration.
00:27:47.279 --> 00:27:53.519
So the functional programmer arrives in an OO code base and is like, what is the type of this?
00:27:53.759 --> 00:27:56.400
You know, what does this program do?
00:27:57.759 --> 00:28:00.480
I'm calling this method, I don't know which method I'm calling.
00:28:00.799 --> 00:28:04.480
That's right, what it's inputs, what it's outputs, it's like all just going on here.
00:28:04.640 --> 00:28:08.079
Like, help me, you know, and the OO program is totally cool about it.
00:28:08.240 --> 00:28:13.759
It's like, no, but you don't need to know what's calling, you just need to know what it does, you know, right?
00:28:13.920 --> 00:28:15.440
And the other way around, right?
00:28:15.519 --> 00:28:24.000
Like you've got the the OO programmer who shows up in a functional code base and who'll who have similar frustrations.
00:28:24.240 --> 00:28:43.920
And what is very interesting with uh all these cognitive biases that come from the tools you're using and the language that you're using, is that I've rarely seen any programmer who understands those biases, who's conscious who consciously aware of those biases.
00:28:44.160 --> 00:28:52.880
Yeah, like you can know this the same way you conceptually know about yourself that you have other cognitive limitations, but then you go out and just like enact in those cognitive limited ways anyway.
00:28:53.200 --> 00:28:53.599
Exactly.
00:28:53.759 --> 00:29:03.200
So what I've seen is you know, the C guy who really thinks that everything should be written in C, and anybody who doesn't use C is an idiot.
00:29:03.359 --> 00:29:03.519
Right.
00:29:03.680 --> 00:29:04.319
And it's just no.
00:29:04.880 --> 00:29:06.160
We call those people correct, right?
00:29:06.319 --> 00:29:11.519
We call those people who can program, people who actually know how to program and know how computers work, right?
00:29:11.839 --> 00:29:19.920
I mean, system programmers are really interesting because there's also some kind of hierarchy of you know how noble the software is.
00:29:20.240 --> 00:29:22.640
Yeah, the less feedback you have, the more.
00:29:23.279 --> 00:29:37.119
But I think it has to do with the execution stack, okay, which is that the if I'm using software that you have written on a daily basis, but not the other way around, in the software hierarchy that makes you more important.
00:29:37.440 --> 00:29:38.799
I'm a linear person, I agree.
00:29:39.119 --> 00:29:41.119
At least that's how system programmers see the world.
00:29:41.359 --> 00:29:41.920
That's right, that's right.
00:29:42.079 --> 00:29:46.079
And so for system programmers, they're thinking, well, I write the kernel, bitch.
00:29:46.160 --> 00:29:49.279
You know, I'm I'm I'm the man, you know.
00:29:49.359 --> 00:29:50.559
Like, what are you doing?
00:29:50.720 --> 00:29:53.279
Everything you do uses my stuff, right?
00:29:53.359 --> 00:30:03.200
I I don't care, but and so it it's it's really interesting, those hierarchies that there's also the um I'm sort of a reformed kernel person.
00:30:03.279 --> 00:30:05.599
Like I expected to end up being doing kernel stuff.
00:30:05.839 --> 00:30:15.279
So, for example, for kernel persons and and system programmers, one thing that drives me absolutely crazy is how shitty their fucking APIs are.
00:30:15.680 --> 00:30:21.759
Sorry for my French, but basically the it all comes back to the it's for good reasons.
00:30:21.839 --> 00:30:31.440
It's basically what happens is they optimize for speed, uh for memory consumption, for basically the hardware.
00:30:32.000 --> 00:30:33.519
They don't optimize for the human.
00:30:33.759 --> 00:30:34.960
Let's say you're in Java.
00:30:35.279 --> 00:30:46.720
If I take a library in Java, I look at the name of the methods, um, and I read the docs quickly or whatever, chances are high I'm gonna get it right without reading all the time.
00:30:47.359 --> 00:30:51.279
This is a game engine or a you know trading AI thing.
00:30:51.759 --> 00:31:07.039
I mean, even going that uh we can go more into details, but let's say you know I have a uh a method that starts a process, I assume that if I just call this method without any of the optional parameters, it's probably gonna do the right thing, right?
00:31:07.519 --> 00:31:13.680
And in in in system programmers' APIs, it's almost certainly never gonna do the right thing.
00:31:14.000 --> 00:31:16.720
So you've got two ways, two annoying ways.
00:31:16.799 --> 00:31:24.559
You've got the one where you're trying to do something very simple, and 99% of the users will try to do these very simple things.
00:31:24.720 --> 00:31:36.640
To get that simple thing to work, you need 25 different calls and setting up, you know, and and and everybody copy-pastes the same code that they found on the internet to get that set up.
00:31:36.799 --> 00:31:40.799
So that's create process in Windows, for example, if you ever use that.
00:31:41.119 --> 00:31:44.480
Um IOctol and like the family of IOctols on Unix, right around.
00:31:44.880 --> 00:31:50.480
And and the other example is the API has just all the wrong defaults.
00:31:50.720 --> 00:32:04.640
You know, so for example, I discovered the last instance of that was I discovered recently, not that recently, maybe 10 years ago, that you could now use locks within shared memory.
00:32:05.279 --> 00:32:10.799
So this used to be something completely S-Rex that nobody was doing was definitely not supported by the code.
00:32:11.200 --> 00:32:12.240
You guys aren't doing this, are you?
00:32:12.400 --> 00:32:13.759
You're not doing this sounds really scary.
00:32:14.000 --> 00:32:15.039
We're not doing that, okay.
00:32:15.279 --> 00:32:16.640
But you can do that, right?
00:32:17.359 --> 00:32:25.759
And so and if this is System 5 shared memory segments that we're talking about, and uh which is like a little bit of a wort on the Unix API to start with, arguably.
00:32:26.160 --> 00:32:27.119
Well, yeah, yeah.
00:32:27.519 --> 00:32:32.480
And now you can tell at the initialization of the mutex that this is actually in shared memory.
00:32:32.640 --> 00:32:39.599
And so if you have different processes that m-mapped that same same part of memory, they can actually use that as a lock.
00:32:39.759 --> 00:32:39.920
Okay.
00:32:40.079 --> 00:32:42.240
And so this is now supported by Linux.
00:32:42.319 --> 00:32:50.160
Um, this is via m map, not via system five shared memory or it's uh it's no, it's uh system uh shared memory.
00:32:50.240 --> 00:32:51.279
So it's an Mmap.
00:32:51.599 --> 00:32:58.319
Mmap map shared, but not the special funny namespace that's not the file system that is the that's the shhmem calls.
00:32:58.400 --> 00:32:58.559
Okay.
00:32:58.640 --> 00:33:00.160
No, I'm I'm showing I'm showing you.
00:33:01.440 --> 00:33:02.480
It's just a regular Mmap.
00:33:02.880 --> 00:33:03.519
So relieved, okay.
00:33:03.839 --> 00:33:06.000
Regular Mmap with uh shared.
00:33:06.079 --> 00:33:07.359
I I misunderstood what you meant.
00:33:07.519 --> 00:33:11.359
So regular Mmap, that is just shared that you can share across different processes.
00:33:11.519 --> 00:33:18.079
And it used to be that you if you try to allocate a lock in that memory, the lock was basically not working, right?
00:33:18.240 --> 00:33:21.279
And so now it's supported and it has been supported for a long time.
00:33:21.440 --> 00:33:24.160
So I was very happy about that because I needed that.
00:33:24.480 --> 00:33:31.759
And I, you know, do the stupid thing, which is that I did not spend my whole day reading the man page.
00:33:32.480 --> 00:33:36.640
And you know, like the the system programs were there to punish me because they don't like that.
00:33:36.799 --> 00:33:37.039
That's right.
00:33:37.359 --> 00:33:41.839
You are supposed to read the whole man page and and read it multiple times.
00:33:43.680 --> 00:33:45.119
And and I didn't do that.
00:33:45.200 --> 00:33:53.200
And and then so what I would expect is if one of the programs crashes, one of the processes crash, it releases the lock.
00:33:53.359 --> 00:33:53.680
Right.
00:33:53.920 --> 00:33:55.680
And it doesn't do that by default.
00:33:55.839 --> 00:33:56.079
Yeah, yeah.
00:33:56.319 --> 00:33:56.799
Of course not.
00:33:56.960 --> 00:33:57.200
Yeah, yeah.
00:33:57.440 --> 00:33:58.000
Why would it?
00:33:58.079 --> 00:33:59.279
Why would it do that by default?
00:33:59.599 --> 00:34:01.200
Well, might that be the safe behavior?
00:34:01.279 --> 00:34:03.839
Maybe it needs to still have exclusive access until it restarts.
00:34:04.240 --> 00:34:16.320
And I mean, when you think about it, what probably happened is that it probably costs something extra to do that extra work when the process is is uh is um killed.
00:34:16.559 --> 00:34:22.480
And so they decided to not make that the default for those who want, but it's fucking stupid.
00:34:22.800 --> 00:34:24.480
Like that's not what people want.
00:34:24.639 --> 00:34:33.280
And so to go back to how system programmers think, they do not think in terms of what do the people who are gonna use my API want.
00:34:34.079 --> 00:34:38.079
They're not thinking those terms, they're thinking in terms of I think that's the library author's problems.
00:34:38.400 --> 00:34:38.639
Exactly.
00:34:38.880 --> 00:34:41.119
That's actually the that's actually how you do think about this.
00:34:41.280 --> 00:34:46.719
Is like you uh because I've been in the position of having to build these horrible systems that you're describing in various ways.
00:34:46.800 --> 00:34:51.280
And like the thing is like you never get performance back if you give it away, right?
00:34:51.360 --> 00:34:57.360
Like there's no there's no lever that lets you like be like, oh whoops, we like this inevitably, and it's like a store, right?
00:34:57.440 --> 00:35:08.800
Maybe that maybe it's literally one instruction, but the semantic of the way we define this means we just absolutely have to do this work in this situation where lots of people don't need us to do that work, and so you want to say, okay, it's all a libraries problem.
00:35:08.960 --> 00:35:12.639
And I mean, I grew I'm um I am one of these people that you're talking about, obviously, first of all, right?
00:35:12.719 --> 00:35:21.039
And I grew up in kind of uh, you know, you look you were educated in O'Camel, I was kind of educated in C, not because because that was how these countries were doing it at the time, right?
00:35:21.119 --> 00:35:25.280
Like you just my first language was C and Pascal.
00:35:25.440 --> 00:35:25.519
Okay.
00:35:25.920 --> 00:35:29.760
And I discovered so I So you but but you so you understand how computers work.
00:35:29.840 --> 00:35:33.440
And yet I understand how computers work confusing opinions.
00:35:33.840 --> 00:35:38.880
Then what when that then what happened is that I learned OCaml much later on.
00:35:39.119 --> 00:35:42.079
Uh and back then, uh the first time I tried it, I didn't like it.
00:35:42.239 --> 00:35:43.360
So first it was Scheme.
00:35:43.760 --> 00:35:51.840
So Scheme, when I learned Scheme, so I I got you know uh that oh god, the name the name escapes me off the Lambda book.
00:35:51.920 --> 00:35:53.599
Um yeah, uh the Sussman book.
00:35:53.920 --> 00:35:54.239
Yeah, exactly.
00:35:54.559 --> 00:36:01.360
Uh interpretation and exactly uh and I structural interpretation computer programs, that's ICP.
00:36:01.519 --> 00:36:02.960
Sorry, it's gonna drive me crazy.
00:36:03.840 --> 00:36:08.719
Uh and so the first time I got my hands on that book, I was maybe in my first year at university.
00:36:08.880 --> 00:36:11.360
Okay, and back then it was still in Scheme.
00:36:11.679 --> 00:36:14.079
I think they have a version in Python now that MIT does.
00:36:14.719 --> 00:36:15.519
Okay, interesting, interesting.
00:36:15.679 --> 00:36:16.480
I I don't know.
00:36:16.639 --> 00:36:20.639
But and I discovered Scheme and I loved it, and it was you know abstracting.
00:36:20.800 --> 00:36:28.320
So what I loved was the simplicity of it, and then I got hooked on functional programming, and after that came O'Caml.
00:36:28.400 --> 00:36:28.719
Yep.
00:36:29.039 --> 00:36:35.599
At first I didn't like OCaml because I didn't like the syntax, so it was a very shallow way, you know, shallow reasons not to like OCaml.
00:36:36.159 --> 00:36:52.480
And then I got hooked on OCaml, and so this is going to be interesting to a C programmer because when you're writing C, what you do basically is you write your program and then question for C programmers out there do you start it when it's the first time you start it?
00:36:52.639 --> 00:36:54.320
Do you start it from within a debugger?
00:36:55.920 --> 00:37:04.480
And to tell the truth, like I just started from within a debugger when it's the first time I tried it because I'm pretty sure it's gonna stay full.
00:37:04.639 --> 00:37:13.039
You know, if I wrote something non-trivial and then you spend a ton of time debugging, and I never like debugging, like I and some people love it.
00:37:13.199 --> 00:37:13.760
Oh, you're right.
00:37:13.840 --> 00:37:20.480
This probably is actually there, this is an interesting cleaver that might sort of separate the world of sort of natural systems programmers from yeah, yeah.
00:37:20.559 --> 00:37:21.920
It's like, would you rather just work?
00:37:22.079 --> 00:37:24.239
Or is it actually kind of fun to investigate?
00:37:24.559 --> 00:37:28.079
I mean, they are people who just love this stuff, they love to debug.
00:37:28.159 --> 00:37:34.559
Uh, they love to you give them, you know, um the secfold file where is that called the core dump.
00:37:34.719 --> 00:37:34.960
Yeah, yeah.
00:37:35.039 --> 00:37:36.800
You give them a core dump, they investigate that.
00:37:37.039 --> 00:37:38.800
I don't know how little languages have core dumps.
00:37:39.119 --> 00:37:39.920
These are useful.
00:37:40.079 --> 00:37:42.320
It's the most core debug things, right?
00:37:42.400 --> 00:37:46.000
You want a user to be able to give you here's the serialization of the program state when it failed.
00:37:46.159 --> 00:37:47.840
Why is it all just a stack trace?
00:37:48.239 --> 00:37:49.119
It's true.
00:37:49.360 --> 00:37:51.599
Uh I it's a good question.
00:37:51.760 --> 00:38:04.719
It's probably too expensive when there's a VM, maybe getting the states of the VM, because I think it when you have a VM, you also need, you know, the core dump or the VM, and that might be too large.
00:38:05.280 --> 00:38:06.079
I I don't know.
00:38:06.239 --> 00:38:16.559
But basically what happened for me is that I was used to programming this way, and with scheme even more so, because in Scheme it's all dynamic, so you have to debug all the time.
00:38:16.719 --> 00:38:26.559
And the first time that got me hooked on OCaml is that I wrote something non-trivial in OCaml and it just worked, you know, the first time.
00:38:26.880 --> 00:38:28.800
For me it was Haskell in 1998, by the way.
00:38:28.960 --> 00:38:30.639
Like it, but yeah, the similar thing of the same thing.
00:38:31.039 --> 00:38:34.159
I wrote all this crap, and like once I got to compile, it just worked.
00:38:34.480 --> 00:38:34.960
Exactly.
00:38:35.119 --> 00:38:42.719
And the first time this happened to me, I was like, oh my god, there's no going to know this is what I need now.
00:38:42.880 --> 00:38:47.360
And then then I got and and true, I think I went a little bit too far with types.
00:38:47.840 --> 00:38:59.760
So I I I like types because they they helped me write correct programs, and then I started trying to encode everything with types, and then I get to got to the point where I did discovered dependent types.
00:38:59.840 --> 00:39:05.679
So I was like, well, wait a minute, I can have any spec I want in the type.
00:39:05.920 --> 00:39:16.239
And then I was like, I remember bursting into the the the office of my my uh my professor back then telling him, we're going to prove everything.
00:39:16.480 --> 00:39:16.800
That's right.
00:39:16.960 --> 00:39:19.760
This is gonna be awesome, there will be no more bands.
00:39:20.000 --> 00:39:22.639
That's right, it's going to be a wonderful world.
00:39:22.800 --> 00:39:25.039
And he was, you know, smirking back.
00:39:25.119 --> 00:39:32.239
He was like, So what it is you're gonna prove exactly like well, everything.
00:39:32.559 --> 00:39:38.639
And this is actually sort of one of the things I do, like circling back a little bit to LLMs, but not in a super hard way.
00:39:38.880 --> 00:39:43.360
You know, types are a kind of lightweight formal system, right?
00:39:43.440 --> 00:39:45.039
Like they're a lightweight formal method, right?
00:39:45.119 --> 00:39:49.519
They're a formal method that's kind of built into your tool chain, and so you're using it kind of by default.
00:39:49.679 --> 00:39:50.000
Yeah.
00:39:50.239 --> 00:39:56.320
But at some level, the type checker, that part of the semantic checking of your program passing is a theorem about your program, right?
00:39:56.400 --> 00:39:57.760
Like is it is a sort of claim about?
00:39:59.119 --> 00:40:00.400
Yeah, there you go, there you go.
00:40:00.719 --> 00:40:02.480
Um yeah, people are smart, man.
00:40:02.719 --> 00:40:04.320
Like lots, lots of smart guys.
00:40:04.719 --> 00:40:20.639
Um one of the things that has been interesting to me about this like last six or nine months is that people that I think of as extremely resistant to or extremely skeptical of you know, both static types, but also other kinds of formal methods, including like building lean forebroofs and stuff like that.
00:40:21.360 --> 00:40:22.960
Some of them are trying to do this, right?
00:40:23.119 --> 00:40:25.280
And and and some of them are having some amount of success, right?
00:40:25.360 --> 00:40:35.199
So like uh like mutual friend Ken Peck, who of course like is one of these unit testing people, like you know, has has a lot to do with the fact that we call him unit tests like built J unit and kind of popularized TDD.
00:40:35.760 --> 00:40:36.159
TDD.
00:40:36.480 --> 00:40:43.599
Yeah, and uh I'm sure it's one of those things with many fathers, but like is certainly one of the people who is out there, you know, influencing people's thoughts about these things.
00:40:43.920 --> 00:41:09.440
And uh, you know, it's like last time I I had a conversation with him, I was actually on his podcast still burning a couple of weeks ago, and you know, he's was building a B tree and and building a little lean for just that the B tree doesn't supposed to do, and kind of feeling like that was working, kind of feeling like this was helping, even though he, you know, didn't know much about lean four especially and didn't know much about formal methods especially, kind of was able to get up to speed along the way.
00:41:10.079 --> 00:41:14.480
And I'm curious like whether that ends up being what we expect, right?
00:41:14.559 --> 00:41:18.639
Because I mean, yeah, like you, when I review a ton of even you know, fable code, right?
00:41:18.719 --> 00:41:22.880
Like there are defects, there's things where it misunderstands your intent or misunderstands what it's supposed to do.
00:41:23.440 --> 00:41:37.679
And if like everything we did was accompanied by some sort of proof that that the things I care about are actually upheld by this program, you know, does that become a thing that either else wants, that their users want?
00:41:38.000 --> 00:41:48.159
So, first of all, for what I'm about to say, I think half of the employees of my company are are gonna want to kill me because they love formal methods and they want more formal methods everywhere.
00:41:48.400 --> 00:41:51.679
The other thing is that I think formal methods is super useful.
00:41:51.760 --> 00:41:57.199
I and I find them even beyond useful, I find them beautiful.
00:41:57.360 --> 00:41:57.599
Right.
00:41:57.760 --> 00:42:10.559
Like I think, you know, the first time I used um uh I used uh cock to prove that you know um some algorithm on an AVL tree was actually correct.
00:42:10.719 --> 00:42:10.880
Right.
00:42:11.039 --> 00:42:17.760
Um so it was actually the uh in the book called The Cock Art by uh Yves Berthault and Pierre Castellan.
00:42:17.920 --> 00:42:20.159
So if you have it's just French for chicken children.
00:42:21.199 --> 00:42:26.239
So if if you have yeah, it's uh French for chicken, it's not the kind of cock, it's the good kind of cock.
00:42:26.480 --> 00:42:27.519
It's the one that's in Q.
00:42:28.079 --> 00:42:28.800
C O Q.
00:42:28.880 --> 00:42:30.880
Yeah, and I think they changed the name recently.
00:42:31.119 --> 00:42:31.440
Yeah.
00:42:31.840 --> 00:42:38.639
And so if you haven't read that book, this is a really, really good, it's my one of my favorite computer science books out there.
00:42:38.800 --> 00:42:49.519
And I think not not very many people know it, but there's something extremely satisfying in getting the proof that whatever you've written is is correct, right?
00:42:49.760 --> 00:43:03.280
Now, the thing is, my experience with those things is um depending on the so there are two two limitations, two things that I think are going to be a problem for LLMs, um, because they are problems for human beings.
00:43:03.920 --> 00:43:14.639
Number one, there are certain classes of problems where uh uh trying to specify them formally is actually harder than solving the problem itself.
00:43:14.960 --> 00:43:22.320
So, of course, if you're trying to sort numbers or writing a B tree, like you just said, the properties are nice mathematical properties, right?
00:43:22.400 --> 00:43:27.440
So if I want to write the spec for a sorted array, that is really easy, right?
00:43:27.599 --> 00:43:32.880
For all i da da less than uh uh i plus one, and you know, done.
00:43:33.039 --> 00:43:36.719
It's and not only is it easy, it's actually really readable.
00:43:36.880 --> 00:43:43.039
Like another human being will be able to look at this spec and go, like, yeah, that's a sorted array, right?
00:43:43.679 --> 00:43:50.079
Now they are in B tree, we all know the properties of B trees or red block trees or whatever it is that you're trying to build.
00:43:50.480 --> 00:43:59.760
So I think for those kind of problems where you have a well identified data structure and well identified properties that you want to prove.
00:44:00.400 --> 00:44:09.519
Those formal methods are going to be bomb and they're going to be used all over the place, probably by LLMs more than humans, because it's a lot of work.
00:44:09.840 --> 00:44:11.199
What is something crazy you believe?
00:44:11.360 --> 00:44:18.239
Like what's what's something where you're mad like on the one hand this, on the other hand, that Julian Verlegay is the only sane man in the world that's gone mad?
00:44:18.800 --> 00:44:20.159
I should have prepped that.
00:44:20.239 --> 00:44:20.960
I don't know.
00:44:21.119 --> 00:44:26.800
Uh I don't know if there is something like that that I believe that is really controversial.
00:44:27.119 --> 00:44:29.440
I think you were trying to get there with local LLMs.
00:44:30.239 --> 00:44:32.320
No, because I don't think that's really crazy.
00:44:32.559 --> 00:44:33.280
You're not maxed out.
00:44:33.599 --> 00:44:34.400
I know, no.
00:44:34.559 --> 00:44:39.679
And when it comes to LLMs and AI, I don't think I have enough context.
00:44:39.760 --> 00:44:44.960
I don't think of myself as enough of an expert to have those kind of crazy ideas.
00:44:45.280 --> 00:44:54.239
But um no, I used to have a crazy idea, which was you know, placade a type system on top of a dynamic language and make it work.
00:44:54.559 --> 00:44:55.119
Yeah, yeah, sure.
00:44:55.360 --> 00:45:01.679
And back then I was, you know, for most people at the company, I was classified as certifiably insane.
00:45:01.920 --> 00:45:03.519
It was a pretty limited project, yeah, yeah.
00:45:04.239 --> 00:45:08.239
You were like the fifth, it was like like uh mathematicians and angle trisectors or whatever, right?
00:45:08.320 --> 00:45:13.519
Like I had you know a stream of people saying I'm gonna put a type system on top of PHP coming to talk to me.
00:45:13.679 --> 00:45:16.000
You were like the fifth one, and like you were the one that had to be.
00:45:17.440 --> 00:45:20.159
You were the one that had like the answers to like the first question, right?
00:45:24.239 --> 00:45:29.840
It was like, yeah, yeah, kiddo, yeah, yeah, we we've had it all before sure.
00:45:32.320 --> 00:45:32.960
Exactly.
00:45:33.199 --> 00:45:35.920
But no, I mean that was definitely a crazy idea.
00:45:36.079 --> 00:45:48.800
I used to have I before that idea, another crazy idea I had was I was obsessed with linear types, and I really thought that linear types were gonna solve the problem of garbage collection.
00:45:48.880 --> 00:45:54.880
Yep, and it was before Rust was popular, so I wrote this compiler called Linear IML.
00:45:55.199 --> 00:46:13.280
And so what's interesting is that um I think history has proven me right, like Rust is a thing, but I was hoping so I was I wasn't hoping to only do system programming with it, and I was hoping that the scope would be much larger.
00:46:13.519 --> 00:46:31.599
And for me, I when when I was working on Linear ML, it as I was using it, it became pretty clear to me that this was never gonna become mainstream to the point that it was going to replace Java because it was just too difficult to use.
00:46:32.320 --> 00:46:35.679
I think that around that time Rust was starting to pick up.
00:46:36.079 --> 00:46:44.800
And then I looked at Rust, which had a scope that was uh um more narrow, which was we're going to replace system programming.
00:46:45.039 --> 00:46:48.159
And I looked at what they're doing and I was like, I think that's right approach.
00:46:48.239 --> 00:46:50.159
That's that's what they should be doing.
00:46:50.400 --> 00:46:57.920
Um, and I think also what Rust had going for it was um C11 was picking up.
00:46:58.079 --> 00:47:10.559
Well, no, C11 was Rust was already had been there a couple of years, but when C11 came out, there was a huge boost for Rust because Rust was basically a formalization of all these principles.
00:47:11.199 --> 00:47:22.320
And so, but I think yeah, so the crazy idea I had in 2004 was hey, we we should use linear types um to manage memory statically.
00:47:22.400 --> 00:47:22.480
Right.
00:47:22.880 --> 00:47:24.320
And then went full in on that.
00:47:24.480 --> 00:47:26.559
I wasn't the only one having that idea.
00:47:26.639 --> 00:47:30.559
In fact, you would you could claim that you know Girard would be the first one.
00:47:30.719 --> 00:47:32.480
Girard was saying that in the 80s.
00:47:32.639 --> 00:47:32.800
Okay.
00:47:32.960 --> 00:47:37.280
So I actually maybe not the philosopher that Peter Thiel's into, but uh Girard.
00:47:39.280 --> 00:47:42.239
He's um French um uh logician.
00:47:42.400 --> 00:47:42.639
Okay.
00:47:43.039 --> 00:47:47.599
Uh and um so he is the one who invented system F.
00:47:47.920 --> 00:47:49.519
So I don't know if you heard of System F.
00:47:49.920 --> 00:47:51.760
I do know of it, but uh that's a deep cut.
00:47:51.840 --> 00:47:53.360
So like why don't you bring me back up the system?
00:47:53.679 --> 00:47:55.599
So yeah, so maybe I can explain to you.
00:47:55.679 --> 00:47:59.039
So basically, when there was first Lambda Calculus, right?
00:47:59.199 --> 00:48:09.519
And Lambda Calculus um Church came up with a very basic uh type system for it, which we now refer to as simply type lambda calculus.
00:48:09.760 --> 00:48:28.400
And that type system for Lambda Calculus was uh very restrictive, like you could barely type any program with it, but it was just checking that the shape of things was correct, and just writing a fixed point operator with it was was already out of the reach of what the type system could do.
00:48:28.559 --> 00:48:36.800
And then so there was a stream, uh there was all sorts of people who worked on building more expressive type system.
00:48:37.199 --> 00:48:40.960
Um, one of the famous ones is the one from Girdle called System T.
00:48:41.440 --> 00:48:44.639
And System T was a good extension.
00:48:44.719 --> 00:48:51.280
Um, and the the main thing that they're trying to do is add looping and well, add a fixed point operator that works.
00:48:51.679 --> 00:48:58.400
Um when Girdle did that, um the people started looking at it and thinking, well, is this expressive enough?
00:48:58.559 --> 00:49:10.480
And that's how they came up with uh Ackerman function, which we now know very well because the Ackerman function was there to show that system T, Girdle's system T was not expressive enough to express all sorts of looping.
00:49:10.639 --> 00:49:17.599
And basically, there was a lot of back and forth, there was a lot of trial, and it all basically finished with system F.
00:49:17.760 --> 00:49:28.159
And system F was basically what's still considered today, I think, the most expressive Thai system for Lambda calculus, where you can pretty much express whatever you want.
00:49:28.320 --> 00:49:31.599
And so that was one of the main contributions of Girard.
00:49:31.920 --> 00:49:48.000
But after that, he worked on linear logic, and linear logic for those who are not familiar with it, is basically something where you it's a different kind of logic because whenever uh so how can I how can I talk about this?
00:49:48.320 --> 00:49:57.360
Basically, what you can notice is that if you use natural logic in certain circumstances, it will lead to incorrect results.
00:49:57.679 --> 00:50:02.960
And usually the way that happens is whenever you talk about resources, right?
00:50:03.280 --> 00:50:07.280
So let's take a resource that everybody knows, let's take money, right?
00:50:07.360 --> 00:50:12.559
And I'm gonna make um statements that involve uh money, right?
00:50:12.960 --> 00:50:16.639
So I'm gonna do the classic thing of A implies B.
00:50:16.719 --> 00:50:22.639
So I'm gonna if A is gonna be a statement, B is gonna be another statement, and I'm gonna say A implies B.
00:50:22.719 --> 00:50:26.480
So let's say A is a statement, I have two dollars.
00:50:26.800 --> 00:50:30.000
So you have two dollars in your pocket, the statement A, right?
00:50:30.480 --> 00:50:34.320
Then B is the statement about what you can buy with it.
00:50:34.800 --> 00:50:38.639
So let's say it can say B is I can buy a chocolate bar, right?
00:50:39.199 --> 00:50:45.440
Then I make another statement, C, which says I can buy you know true in gum.
00:50:45.679 --> 00:50:46.239
Right.
00:50:46.559 --> 00:51:00.559
So if I follow the rules of that natural logic, if I actually say A implies B, which is if I have two dollars, I can buy a chocolate bar, let's say this was true, then I have a second statement that says A implies C.
00:51:00.719 --> 00:51:03.920
If I have a chocolate bar, I can buy true ingum.
00:51:04.159 --> 00:51:05.519
That'd be B implies C though, right?
00:51:05.599 --> 00:51:09.840
Oh, if I have a chocolate bar, no, uh A implies B, A implies C.
00:51:10.079 --> 00:51:10.320
Okay.
00:51:10.719 --> 00:51:18.719
If I have those two things together, in natural logic, I can deduce that A implies B and C.
00:51:19.039 --> 00:51:19.199
Okay.
00:51:19.440 --> 00:51:24.559
So I can deduce that if I have two dollars, I can buy a chocolate bar and I can buy a chewing.
00:51:24.960 --> 00:51:25.760
I'm following.
00:51:26.079 --> 00:51:28.559
And that is obviously incorrect.
00:51:28.639 --> 00:51:37.599
I mean, you can go try out the gas station and tell them, show them your little reasoning, they're gonna say, no, sorry, get out of here.
00:51:37.679 --> 00:51:41.119
You know, we're not we're not giving you um those two things.
00:51:41.280 --> 00:51:45.360
Um, and the reason is two dollars is actually a resource.
00:51:45.760 --> 00:51:50.559
So whenever I write, if I have two dollars, I can buy a chocolate bar.
00:51:50.880 --> 00:52:00.239
Whenever that statement, that uh implication has been used in my proof, it has to consume the two dollars, right?
00:52:00.559 --> 00:52:17.679
And that's what Girard was interested in, and he developed a logic for that, but his secret plan was not to manage resources, well, it was uh he was he was particularly interested in a in a particular kind of resource, and that's memory.
00:52:18.000 --> 00:52:35.039
So basically what happened is he developed a logic where all implications are um changed, um uh modified to have linear implications, and a linear implication consists in taking the statement and consuming it.
00:52:35.440 --> 00:52:43.519
So now when I write A and price B, if I decide to use A, A has not used, I used up A.
00:52:43.599 --> 00:52:46.639
Okay, so now I cannot go back and use it for something else.
00:52:46.880 --> 00:52:57.599
Right now, how do you express it's it's actually strictly more so then you have operators to say when you want to duplicate A or shrink A.
00:52:58.000 --> 00:53:03.679
And because thanks to those operators, you can actually express all of natural logic.
00:53:03.760 --> 00:53:15.360
Okay, because in natural logic, all you have to do is say um duplicate A and R times explicitly, and then you'll get to deduce all the things you would have deduced in natural logic, right?
00:53:15.440 --> 00:53:18.639
So it's strictly more expressive than natural logic.
00:53:18.880 --> 00:53:23.679
Then then at this point, you're probably thinking, what is the relationship with you know memory?
00:53:24.559 --> 00:53:25.440
Where are we going with it?
00:53:25.920 --> 00:53:30.639
And so the thing is, so I have to jump to programming through Curry Howard.
00:53:30.800 --> 00:53:46.719
So for those who are not familiar with it, Curry Howard is a bijection um that was invented in the that was discovered in the 70s, that basically says that a type is to a program what um a theorem is to a proof.
00:53:46.880 --> 00:54:08.800
So what that means concretely is that whenever I tell you, hey, I have a function, it takes an a and it produces b's, um, and you with your little fingers write down that function, you are actually doing exactly the same thing as if I told you, hey, I have a theorem that takes a's and produces b's.
00:54:08.880 --> 00:54:10.079
Can you write the proof?
00:54:10.480 --> 00:54:15.840
Writing that proof is the same job as writing the the function.
00:54:16.159 --> 00:54:23.119
And the the the good example to convince yourself of that is to take basic examples involving implications.
00:54:23.519 --> 00:54:27.760
So uh we we can do one together so that you you get the idea.
00:54:28.000 --> 00:54:33.760
So let's say let's say you know I have a implies b and a implies c, right?
00:54:34.159 --> 00:54:41.039
Um, and I want you to write me something that's uh a proof that a implies c.
00:54:41.840 --> 00:54:49.599
Uh I want you to write the no, wait, you have a implies b, b implies c and well, actually we can do simple.
00:54:49.920 --> 00:54:58.079
We can take you take an a, you take a function that take given an a produces a b, and you have to produce b's.
00:54:58.320 --> 00:55:01.440
So let's say you were writing that function with your little thing, right?
00:55:01.519 --> 00:55:04.800
Your first parameter is an a, your second parameter is a function.
00:55:05.039 --> 00:55:11.280
Well, what you would do is you would apply the a to the function and return that result, right?
00:55:11.440 --> 00:55:11.760
Right?
00:55:11.920 --> 00:55:22.559
It turns out that this exact thing that you're doing has an existing rule in a proof theorem that would be called modus ponens that does exactly that, right?
00:55:22.800 --> 00:55:28.320
So I can take your program and derive uh uh a proof from it.
00:55:28.480 --> 00:55:39.360
It the only thing you have to remember is proving that A implies B is the same thing as writing a function that takes takes the given an A produces a B.
00:55:39.519 --> 00:55:39.679
Right.
00:55:39.920 --> 00:55:45.519
That's a really interesting uh usually like when I ask people for like something crazy they know or or know is true.
00:55:45.679 --> 00:55:47.519
Um it's sort of a hot take, basically, right?
00:55:47.599 --> 00:55:49.039
They're sort of like, well, you know.
00:55:49.360 --> 00:55:50.880
So I didn't have a good hot take.
00:55:50.960 --> 00:55:56.719
So what I did is I took the hot take of somebody else in the past, which I thought was really cool.
00:55:56.800 --> 00:55:58.639
And no, but you did real effort on it, right?
00:55:58.719 --> 00:56:00.719
Like you actually like committed a chunk of your intellectual work.
00:56:01.119 --> 00:56:06.960
That is true, but it wasn't really my hot take in the sense that there were people who were you know already believing in that.
00:56:07.119 --> 00:56:09.760
Uh, but yeah, I I tend to do that though.
00:56:09.840 --> 00:56:14.960
Once I have a hot take, I tend to just go, oh no, I have a hot take.
00:56:15.119 --> 00:56:16.159
So here we go.
00:56:16.719 --> 00:56:27.760
I think um reactive programming is going to be uh a very important part of the success of LLMs, especially in the field of coding.
00:56:28.079 --> 00:56:35.440
And so my hot take, well, and I'm cutting my money where my mouth is, obviously, because I'm building uh all the ecosystem to make that work.
00:56:35.599 --> 00:56:49.840
But basically, my take is LLMs will need to train on larger systems, and to train effectively on larger projects, they will need to be able to iterate on those larger projects faster.
00:56:50.000 --> 00:56:55.199
And so the latency of the tooling is going to become the bottleneck.
00:56:55.840 --> 00:57:00.239
And so right now I think nobody cares yet.
00:57:00.400 --> 00:57:12.880
I mean, a little bit like uh anthropic, but I think that goes in that direction that they start to think uh maybe we should own some of that tooling to help, you know, the feedback loop of the LLM.
00:57:13.119 --> 00:57:25.280
But I think it's going to become apparent, much more apparent once um the we want to train LLMs to do much longer tasks and work on much bigger projects by themselves.
00:57:25.679 --> 00:57:30.880
So right now they perform tasks on large projects that they usually didn't write themselves.
00:57:31.199 --> 00:57:47.519
But once we get to the point where we wanted LLM to write a five million line code uh in a five million uh line code base and maintain it themselves, once we get to that point, then to train those kind of models well, you will need to run experiments.
00:57:47.760 --> 00:57:55.599
And if every time you run an experiment, you have to wait for circle CI for 30 minutes, the training is not going to be very fast.
00:57:56.079 --> 00:58:00.320
So if you take all of that and shrink it.
00:58:00.559 --> 00:58:03.360
So, and what I mean by all of that is really the whole thing.
00:58:03.440 --> 00:58:09.280
So, what I mean is the first step is obviously type checking and linting, all these things that has to come back fast.
00:58:09.760 --> 00:58:19.360
Once this comes back, then you need to do static analysis to figure out which tests you have to run and only run the tests that matter.
00:58:20.000 --> 00:58:39.519
Once the tests passed, you will have to also make the program itself incremental so that you don't have to restart everything, fill up all the caches and everything, invalidate you know just the parts of the code that needs to be invalidated and re-recompute only the parts of the cache that needs to be computed.
00:58:39.679 --> 00:58:47.599
And then if if you can do all of that, which is what we are building at Skip Lab, so you can't have problems in computer science.
00:58:47.840 --> 00:58:48.480
Uh so yeah.
00:58:48.800 --> 00:58:53.280
If you can do all of that, then the change whenever you're doing so.
00:58:53.360 --> 00:58:58.960
Another thing that's going to be important, I think, is the capa the cape, um, the ability to branch.
00:58:59.199 --> 00:59:04.239
So you want your memory to be you want to be able to run like a hundred experiments at the same time.
00:59:04.480 --> 00:59:07.519
And kind of like copy on rip fork the big thing underneath it.
00:59:07.599 --> 00:59:08.000
Yeah, yeah.
00:59:08.239 --> 00:59:23.519
And so once you have all of that, that means that when the the the AI wants to try, you know, let's say a hundred different things that would have taken half an hour each, and all of that takes, you know, step second.
00:59:24.480 --> 00:59:30.159
That the model who has this kind of tooling will beat all the other problems.
00:59:30.559 --> 00:59:35.760
And so, but it's it's the history of optimizing pretty much everything.
00:59:35.920 --> 00:59:48.079
There's always you know a bottleneck, and then all your resources go on that bottleneck, and you only start investing in the next bottleneck once you know things when when you basically get to it.
00:59:48.480 --> 01:00:06.559
And so I think we haven't actually gotten to it yet, but we will get to that point where the bottleneck is the tooling, and when that's the case, then the latency of the tooling is going to be the key, and the only way to build tooling with low latency is with reactive programming.
01:00:07.039 --> 01:00:19.280
If you were uh if you met a 19-year-old version of yourself, except it's 2026, um, and you're yourself and you're brilliant, and uh you're in France somewhere, you're you're you're a brilliant man, seriously.
01:00:19.519 --> 01:00:21.760
Um we'll skip over the false modesty.
01:00:21.920 --> 01:00:26.320
Um and you know, you've got a lot of potential anyway, and and a lot of opportunity.
01:00:26.559 --> 01:00:28.159
And you love all this stuff, right?
01:00:28.239 --> 01:00:37.039
You're really interested in PL, you're really interested in in computers, you're really interested in things like computers, the machines, but also computers like the kind of conceptual frameworks that are unlocked.
01:00:38.559 --> 01:00:42.639
What would you advise that younger version of yourself today to go do?
01:00:42.800 --> 01:00:48.400
Is it still just like do your informatics degree, you know, you're on the right, you're on the right path to keep going?
01:00:48.960 --> 01:00:57.440
Um, or does something about kind of this you know, whole LLM milieu massively change the right way to allocate your energies?
01:00:57.760 --> 01:00:58.719
Yeah, I don't know.
01:00:58.880 --> 01:01:04.559
I mean, we're talking about a younger version of myself who would be young today, who would be starting to.
01:01:04.719 --> 01:01:05.280
That's right, yeah, yeah.
01:01:05.360 --> 01:01:06.400
So you're 2026.
01:01:06.639 --> 01:01:07.280
Okay, Julian.
01:01:07.440 --> 01:01:09.760
And I'm 19 and 2026.
01:01:10.000 --> 01:01:22.400
Yeah, so I would definitely add more time, you know, working on data science and spending more time figuring out how all this stuff works.
01:01:22.800 --> 01:01:24.639
All this stuff being like kind of machine learning.
01:01:24.880 --> 01:01:25.760
Yeah, machine learning.
01:01:25.840 --> 01:01:31.440
Yeah, figuring out how machine because I I came pretty late in the game for these kind of things.
01:01:31.519 --> 01:01:36.239
Like I learned seriously about them in 2016, 17, something like that.
01:01:36.559 --> 01:01:40.079
Um and you waited for me to leave Fair before you started working on it.
01:01:40.159 --> 01:01:40.559
Thanks, man.
01:01:40.719 --> 01:01:43.360
Yeah, I appreciate you joined Fair after you left.
01:01:43.519 --> 01:01:44.800
So that was interesting.
01:01:44.960 --> 01:01:57.039
There was a time where I was thinking joining Fair in 2013, uh, right after Hack, where you were telling me all about these uh this cool stuff happening at Fair, and I I was very curious about it.
01:01:57.199 --> 01:02:06.880
So I would definitely tell the younger version of myself go learn what Transformer is, how this stuff works, go build an intuition, because that's the stuff that takes the longest time.
01:02:07.039 --> 01:02:13.039
Because for example, when I joined Fair, the way I like to learn things is usually by using IT.
01:02:13.679 --> 01:02:25.840
So the first thing I did is I rebuilt a framework for you know AI and rebuilt um uh an autograd and all that stuff because I wanted to understand how all the formulas were working.
01:02:26.079 --> 01:02:33.519
I had to make autograph in C, or I had to make a template system where it's like these things are, you know, and then you just I don't and I don't think anybody ever used it for anything.
01:02:33.599 --> 01:02:34.880
I hope they didn't use it for anything.
01:02:35.039 --> 01:02:39.440
But point is like we're engineers, we learn with our hands, you kind of have to make stuff to know.
01:02:39.760 --> 01:02:56.800
Same instinct, but basically what ended up happening is that I do know how these things work and I don't, in the sense that I do know what steps need to be taken to get a transformer to work, but I don't have a good intuition.
01:02:56.960 --> 01:03:09.840
I have a little bit of an intuition of why it works and why not, but I think what takes a very long time in it is to build an intuition where you know what kind of things are going to encourage the model, for example.
01:03:10.559 --> 01:03:12.559
And that that takes time.
01:03:12.639 --> 01:03:19.199
So tell my younger self, go educate yourself on that and become better at what along those lines.
01:03:19.280 --> 01:03:25.119
Like when I started doing this stuff at Fair, I think that part of it felt unlike any other kind of computing to me.
01:03:25.360 --> 01:03:26.000
Yeah, that's true.
01:03:26.320 --> 01:03:33.599
Like when I was I I remember I was working on a word embedding model that um was sort of inspired by WordDev, but was like, we're gonna use hashtags as supervision.
01:03:33.679 --> 01:03:35.760
So it's sort of a bilinear embedding system.
01:03:36.000 --> 01:03:40.559
We're trying to minimize the distance between a bag of words and the hashtags it co-occurs with or whatever.
01:03:40.800 --> 01:03:43.760
And like random CPUs and stuff, like very simple thing, right?
01:03:44.079 --> 01:03:45.920
And I kind of had something kind of working.
01:03:46.000 --> 01:03:46.480
You know what I mean?
01:03:46.559 --> 01:03:50.880
It was kind of like you'd put it up the tweet and it would kind of like get a hashtag that sounded plausible some of the time.
01:03:51.119 --> 01:03:52.880
And then we hired Jason Weston, right?
01:03:52.960 --> 01:04:00.320
We brought Jason Weston in, who's like a great M N great NLP guy, already very experienced at the time, now even more so, has done a bunch of amazing stuff since then, of course.
01:04:00.480 --> 01:04:03.519
But he kind of like looked at this and was like, yeah, this is a cool project.
01:04:03.840 --> 01:04:04.960
This should be better.
01:04:05.119 --> 01:04:10.159
Like for all the things you're looking at and all the parameters this thing has, this just like should be more impressive.
01:04:10.480 --> 01:04:12.159
Try like these seven things.
01:04:12.400 --> 01:04:22.480
And it was kind of like one of them was like, you know, clip the vectors to the unit hypersphere and you know, do uh use this learning rate schedule and well, and a couple of other things, right?
01:04:22.719 --> 01:04:25.519
And like all and you could do it in one step.
01:04:25.599 --> 01:04:29.119
It wasn't sort of like sit there and like retrain for the first one, refrain for the second one.
01:04:29.199 --> 01:04:44.000
It was all just like an improvement, and it was almost and it reminded me, I kept using the analogy of like, it's like a chef like tastes your chocolate chip cookie recipe or something, and it's just like you need more butter, you need more salt, oven needs to be hotter, but then at the end you should add some water or something, whatever like crazy stuff.
01:04:44.079 --> 01:04:47.039
And you do all the weird stuff they say, and it comes out way better.
01:04:47.280 --> 01:04:55.920
And it's not a digital kind of way better, it's this weird analog kind of way better, where just like these results are subjectively much more impressive, and it's still kind of like this.
01:04:56.000 --> 01:04:58.239
I mean, that we know more about the underlying math.
01:04:58.400 --> 01:05:01.119
Like we know, you know, you wouldn't just like clip to the hypersphere now.
01:05:01.199 --> 01:05:03.199
There's like smarter ways to do it and everything.
01:05:03.360 --> 01:05:05.760
Um distributed systems are a bit like that too.
01:05:05.920 --> 01:05:06.639
Actually, that's a good point.
01:05:06.719 --> 01:05:07.360
That's a good example.
01:05:07.519 --> 01:05:08.159
That's a good example.
01:05:08.400 --> 01:05:17.840
People who are good at that stuff, they will be able to architect something from the get-go that's just going to be much, much better than somebody who hasn't been bitten by those problems before.
01:05:18.079 --> 01:05:19.760
We can still build like a TLA plus model though.
01:05:19.840 --> 01:05:20.239
Do you know what I mean?
01:05:20.320 --> 01:05:26.079
Like there's there's still things in distributed systems where there's like a digital notion of correctness that saves your butt, you know.
01:05:26.239 --> 01:05:27.519
The performance part of it is true.
01:05:27.599 --> 01:05:31.280
There's a lot of taste and a lot of a lot of intuition around the around getting it to run well.
01:05:31.679 --> 01:05:38.400
And it also feels the same that they're when they want to optimize the performance, oftentimes they run experiments.
01:05:38.639 --> 01:05:40.960
So it feels like experimental science, right?
01:05:41.119 --> 01:05:51.039
Where they run experiments and in fact, performance work in general feels a little bit like that because the hardware is so complex now that you know nobody really knows what's going on in there.
01:05:51.199 --> 01:05:56.480
So you have to treat it like you know a black box and run experiments, and including the people who made the hardware, right?
01:05:56.639 --> 01:05:59.760
The hardware is so complex it has weather, the hardware is so complex that it you know gets.
01:06:00.159 --> 01:06:00.960
In the states, you know.
01:06:01.280 --> 01:06:07.760
But there's still people with good intuition, good taste who will know this kind of stuff is more likely to work.
01:06:07.840 --> 01:06:10.239
So it's not going to be a hit 100% of the time.
01:06:10.559 --> 01:06:16.239
But they will tend to have better ideas than you know the layman who just doesn't know anything about it.
01:06:16.639 --> 01:06:32.079
So it feels to me that I I would have liked if I was starting software today, I would try to build that taste A ASAP for you know these kind of systems or um anything related to AI.
01:06:32.239 --> 01:06:32.559
Yeah, yeah.
01:06:32.880 --> 01:06:35.280
Um and then I don't know.
01:06:35.360 --> 01:06:40.960
I mean, there are some people who say that you shouldn't learn how you shouldn't, you know, learn programming, pick up programming.
01:06:41.280 --> 01:06:43.760
Um, but I don't know about that, you know.
01:06:43.840 --> 01:06:51.280
Like uh I think pretty prominent people in that space have said that you know learning how to program is useless now.
01:06:51.360 --> 01:06:51.679
Yeah.
01:06:52.079 --> 01:07:02.880
I then again I'm super biased because I like to program, so maybe maybe uh I'm wrong and I'm just not acknowledging my bias, but I find it hard to believe.
01:07:03.280 --> 01:07:10.800
I I still think that we're pretty far from you know having the LLN just completely takes the jobs of all the programs out there.
01:07:11.039 --> 01:07:33.360
What I think is much more likely is that the program is actually going to become much, much more interesting, where you have an army of LLNs, an army of agents that are working for you, doing all the grunt work that nobody likes doing, like refactoring and you know, cleaning up some some stupid stuff that anybody could do, it's just a time consuming.
01:07:33.599 --> 01:07:46.800
Uh it will also help, you know, try out ideas, but I think that at least for the foreseeable future, I don't think it's going to replace the the engineers.
01:07:47.039 --> 01:07:55.360
Uh but you know, uh again, I've underestimated LM before, so I'm not because in my book I think we need a breakthrough.
01:07:55.679 --> 01:07:59.360
I think I said it in in another in another I've I've said it many times.
01:07:59.519 --> 01:08:11.360
I think we need another breakthrough like Transformers where like you know, uh where to like this kind of breakthrough to something that really changes the game to get to that last step.
01:08:11.599 --> 01:08:15.840
And many people think no, uh if we just keep on pushing LMs, we'll get there.
01:08:16.000 --> 01:08:16.319
Yeah.
01:08:16.560 --> 01:08:21.359
So I don't know, is it good advice to tell someone today to learn how to program?
01:08:21.760 --> 01:08:23.600
My best guess is still yes, to be honest with you.
01:08:23.680 --> 01:08:47.359
My best guess is that like even if uh so one of the one of the things I've observed with people using coding agents and LMs and these other kinds of you know massively overpowered, super high-leverage software construction tools is the more I think of somebody as having already attained real mastery of of engineering as a craft and science, the better they're able to use these tools, which is not a priori what I'd have expected, right?
01:08:47.520 --> 01:08:50.880
Like you might have expected that it kind of lifts the floor, but the ceiling is wherever it was.
01:08:51.199 --> 01:08:52.079
It goes both ways.
01:08:52.239 --> 01:08:52.800
Yeah, yeah.
01:08:53.039 --> 01:09:05.680
Those who were mediocre engineers are much, much more mediocre now because the LLMs are basically it's LLMs are like steroids, so they'll supercharge whatever whatever you are, right?
01:09:05.760 --> 01:09:12.960
And so if you were mediocre and you were pushing out shitty code, well now you're pushing out a ton of shitty code.
01:09:13.439 --> 01:09:15.520
That really sucks for everyone.
01:09:15.760 --> 01:09:21.359
And if you were a good engineer pushing out good code, then you know now it's it's like leverage, right?
01:09:21.439 --> 01:09:22.720
That's that's what the LLM does.
01:09:22.880 --> 01:09:24.880
It's an amplifier, it's an amplifier.
01:09:24.960 --> 01:09:36.079
And so it's true that I would tell people to go learn how to program, but I would put a caveat in it, I would say, if you're truly passionate about it.
01:09:36.399 --> 01:10:12.720
Because what I mean, I uh you probably heard this month before when I joined uh uh I was at Facebook and I was frustrated, so it was late state Facebook, and I was frustrated because there was a lot of people who I was interacting with who didn't care about um something relatively geeky, but you know, some programming stuff, and who like do you know, and and I was telling you, I'm frustrated because you know, imagine you're a kid and you like to play basketball, and your whole life you dreamed you dreamt of playing basketball, and you finally made it make it to the NBA, right?
01:10:13.119 --> 01:10:22.000
You've been in the big leagues, and you know, and and you have a plan, you want to talk about things, you want to talk about new ways about training, about new ways of this, new ways of that.
01:10:22.159 --> 01:10:28.800
And the rest of the team, bar maybe one, is like, dude, we're just gonna throw the ball in the basket, all right?
01:10:28.880 --> 01:10:30.880
Nobody gives a nobody gives a fuck.
01:10:31.039 --> 01:10:33.359
This is just a game, we're just going to throw the ball.
01:10:33.520 --> 01:10:36.640
You're annoying us with your new techniques, nobody cares, right?
01:10:37.279 --> 01:10:49.279
And there was a lot of that happening in software in Silicon Valley, in you know, because there was a lot of people who chose the job because it was a well-paid job, right?
01:10:49.840 --> 01:10:58.159
And so I think now with LLMs, having this kind of attitude is going to be very, very risky.
01:10:58.319 --> 01:11:10.560
So if you go into software because you think that it will be a well-paying job, but you're not really passionate about what you're doing, I would say be very, very careful.
01:11:10.880 --> 01:11:22.880
Because what's going to happen is that if you're looking, so typically if you're not passionate about something, what you're gonna do is you're gonna try to find a niche where you get to do always the same thing.
01:11:23.439 --> 01:11:25.600
And that's all gonna get automated away.
01:11:26.079 --> 01:11:31.359
Yeah, that's all in the in the day and age of it used to be you could have a whole career that was like I'm the build system guy.
01:11:31.439 --> 01:11:31.680
Exactly.
01:11:31.920 --> 01:11:33.920
I know everything about make, I know everything about CMake.
01:11:34.239 --> 01:11:34.479
Exactly.
01:11:34.640 --> 01:11:37.760
Uh, but maybe there's some new Fangal build systems, do they even work anyway?
01:11:37.920 --> 01:11:38.399
Who knows?
01:11:38.560 --> 01:11:40.079
I'm the make and C Make guy.
01:11:40.239 --> 01:11:40.479
Exactly.
01:11:40.640 --> 01:11:43.119
And that's what I do for this company, and it's enough, right?
01:11:43.439 --> 01:11:47.279
And I think that kind of is is it is structurally never gonna happen again.
01:11:47.600 --> 01:11:48.159
Exactly.
01:11:48.319 --> 01:11:58.720
So my advice would be if you are truly passionate about it, and in which case, my advice is completely useless because if you're truly passionate about it, you're gonna do it anyway.
01:11:59.199 --> 01:12:02.000
And I mean, that's why I started computers.
01:12:02.079 --> 01:12:04.000
I started programming because I loved it.
01:12:04.159 --> 01:12:04.560
Yeah, yeah, yeah.
01:12:04.560 --> 01:12:07.840
And then back so in the 90s, like computing was a subculture, right?
01:12:07.920 --> 01:12:14.960
Like and back then, you know, nobody would think that you would make big money out of it, especially in France.
01:12:15.039 --> 01:12:23.199
You know, it was more seen like a subpar engineering field where you would, you know, nobody saw this as a as a smart career choice.
01:12:23.520 --> 01:12:27.439
Uh at least not when I I I started like in the 80s when I was a kid.
01:12:27.600 --> 01:12:28.000
Yeah, yeah.
01:12:28.239 --> 01:12:32.880
And and I think, yeah, so the same should apply here.
01:12:32.960 --> 01:12:37.199
Like you the the right question to ask yourself is would you do it anyway?
01:12:39.520 --> 01:12:42.640
Then then you are you know you know you are you're in a great place.
01:12:42.800 --> 01:12:45.199
Uh the same is probably true with music.
01:12:45.439 --> 01:12:50.640
There are other things like that that I I I think you should only start if you're passionate about that.
01:12:50.800 --> 01:12:51.039
Yeah, yeah.
01:12:52.000 --> 01:12:59.760
So yeah, so one one of the differences with music though is if you start it for the money, well, that's probably a bad idea.
01:13:00.000 --> 01:13:10.800
The people who are passionate about it are probably gonna be so much better than than than you, Ali, that you're not gonna find you know a good job and you won't be able to make a living off it.
01:13:10.960 --> 01:13:14.960
And I think now computer science is probably gonna become like that.
01:13:15.119 --> 01:13:16.159
Computer engineering.
01:13:16.640 --> 01:13:26.479
It used to be that we needed so many little hands to write these little shitty things that there was space for a bunch of people who didn't care.
01:13:26.800 --> 01:13:33.680
Yeah, you'd have your 10 million line code base at your gigantic company, and there'd always be some project to move from this data store to that data store.
01:13:34.079 --> 01:13:41.439
And it was like some director somewhere has an army of people who are you know doing nothing but doing these relatively mechanical migrations.
01:13:41.680 --> 01:13:47.039
And if if that project was going slowly, that director could ask for more people and probably get them and so on.
01:13:47.119 --> 01:13:47.359
Yeah.
01:13:47.600 --> 01:13:47.920
Yeah.
01:13:48.000 --> 01:13:54.239
And and actually my theory is that directors love to have um useless projects.
01:13:54.640 --> 01:14:00.960
So I I've I've worked in enough big companies now to know that you know the director wants a big hat count.
01:14:01.199 --> 01:14:01.439
Yep.
01:14:01.520 --> 01:14:03.920
That's I mean, that is actually, and in fairness, that's the measure of impact.
01:14:04.239 --> 01:14:06.159
That's the measure of man's control if you're a manager.
01:14:06.800 --> 01:14:13.760
And and ideally, the problem is when you have people and those people are doing real work, it creates friction.
01:14:14.239 --> 01:14:25.920
Because real work means change and people are reluctant to change, and so it will create human friction that you then have to oil, put oil on, which is basically the job of a man the marginal chain, right?
01:14:26.399 --> 01:14:43.600
And so ideally, if you're a director and you want to grow your org, the best thing to do is to uh hire as many people as possible to work on something useless that you've convinced upper management that somehow was useful.
01:14:43.920 --> 01:14:44.319
That's right.
01:14:44.640 --> 01:14:46.800
That is the idea, that's where you want to be.
01:14:47.199 --> 01:14:49.840
We have this class of bugs that we've had in the past that really hurt us.
01:14:50.000 --> 01:14:58.640
It's because we're on data store fribble, and we need to get to data store boggle to get it, and once we get to boggle, this whole class of bugs goes away.
01:14:58.880 --> 01:15:00.880
Or we could take millions of lines of code.
01:15:02.960 --> 01:15:04.560
Rewrite is is is also a good one.
01:15:04.720 --> 01:15:06.880
And the Rust rewrite that's happening in the case.
01:15:08.399 --> 01:15:10.159
We can have a Rust rewrite that makes sense.
01:15:10.239 --> 01:15:16.079
So for example, um the one that Bun did recently, they had a bunch of uh memory bugs.
01:15:16.239 --> 01:15:18.479
You switch to Rust, you don't have them anymore.
01:15:18.640 --> 01:15:19.359
That makes sense.
01:15:19.520 --> 01:15:30.880
But when you go from a safe language to Rust, and that safe language didn't have performance issues to begin with, you're like, why are we doing this again?
01:15:30.960 --> 01:15:32.319
And and it's not gonna be done.
01:15:32.399 --> 01:15:34.880
It wasn't the plan, it was not to do it with an LLM.
01:15:36.960 --> 01:15:38.399
An army of people doing that.
01:15:38.640 --> 01:15:46.239
For those that have not followed the rip from the headlines bun reference here is it used to be in Zig, they landed a Rust rewrite at the PR.
01:15:46.560 --> 01:15:47.439
That's what you're gonna say.
01:15:48.640 --> 01:15:56.880
Um, so you know, probably uh probably to the chagrin of some open source contributors, I imagine, that might have had a PR open LLM author or otherwise.
01:15:57.119 --> 01:16:01.199
I mean, rebasing after rebasing after that one is gonna be it.
01:16:01.520 --> 01:16:02.880
I guess you'll let the LLM rebase it.
01:16:02.960 --> 01:16:06.000
You let the LLM figure out how to how to express your intent in Rust.
01:16:06.239 --> 01:16:20.079
So, you know, I mean, I mean, and and so for directors, I mean, talking about bad incentives, they have those those really weird incentives that number one, they need to get as many people as possible because that's how you know that's how they make more money.
01:16:20.239 --> 01:16:24.880
And so they have an incentive to uh hire more people than they need.
01:16:25.520 --> 01:16:41.840
And then they have an incentive for those people to do things that are gonna cause as little friction as possible, which usually means to work on things that are as you know not disruptive and that are useless.
01:16:42.479 --> 01:16:45.039
The more something is useless, the less people argue about it.
01:16:45.520 --> 01:16:50.000
And for those listening at home, by the way, Julian and I have worked at some big companies before.
01:16:50.159 --> 01:16:56.720
Uh, the that does tend to be a little corrosive to your uh idealism about what a job and technology can be like over time.
01:16:56.880 --> 01:17:01.680
I do want to say like those are also there is good work that happens there, and there's good teams that do awesome stuff.
01:17:02.319 --> 01:17:04.159
Well, thank you once again for joining us, Julian.
01:17:04.319 --> 01:17:05.680
It's been an honor and a privilege.
01:17:05.840 --> 01:17:08.960
Next time we'll probably be virtual, unfortunately, but we'll have to do this again.
01:17:09.199 --> 01:17:12.880
Um, you know, for those of us that that for those of you that listen this far, thanks for listening.
01:17:12.960 --> 01:17:17.760
And remember to smash, like, and subscribe into stratosphere, like Jose can say, go write it out of his mind.
01:17:18.079 --> 01:17:18.479
Of course.
01:17:18.800 --> 01:17:22.079
Um, and uh hopefully we'll see you next time on Computers Coffee and Beer.
01:17:22.239 --> 01:17:23.039
Thank you.