Why Bad Programming Languages Win (and the Good Ones Don't)

WEBVTT
00:00:00.080 --> 00:00:01.679
Welcome to Computer's Coffee and Beer.
00:00:01.760 --> 00:00:05.679
Uh I'm here in conversation with the inimitable Julian Verligay, as always.
00:00:05.919 --> 00:00:09.439
And this week we're going to be talking about hits in programming languages.
00:00:09.759 --> 00:00:24.160
Yeah, I mean one thing that I'd like to talk about is how, you know, some of the programming languages that we use end up becoming mainstream while they're clearly not the best programming languages.
00:00:24.399 --> 00:00:24.640
Of course.
00:00:25.120 --> 00:00:31.440
And you know how frustrating that can be for people building tools.
00:00:31.760 --> 00:00:35.920
I'm not a PL person, as like I have a very like deep nihilism about it, I guess.
00:00:37.280 --> 00:00:46.399
I'm like, hey, it's all kind of notation around Turing machines, and like correctness is all on the wrong side of the holding problem, no matter what notation you use.
00:00:46.560 --> 00:00:47.679
So who cares?
00:00:47.920 --> 00:00:53.520
And I think that's like and and even sometimes like bad, you know, worse is better in certain ways, right?
00:00:53.600 --> 00:01:04.079
So I think uh I thought of PHP as being kind of you know kryptonite for programming language bigots when we were hiring technically at Facebook a little bit, was it's like if you just take program languages too seriously.
00:01:04.560 --> 00:01:05.120
Yeah, yeah, yeah.
00:01:05.359 --> 00:01:07.120
I heard this argument a lot.
00:01:07.439 --> 00:01:10.560
And it's I don't think that's uh to be clear, it's a rationalization, right?
00:01:10.640 --> 00:01:16.079
It's not it's not a real thing that was planned out beforehand, but no, don't get me wrong.
00:01:16.239 --> 00:01:23.599
There's definitely some people who get carried away with programming languages and language nodes can can be a little bit annoying.
00:01:23.680 --> 00:01:45.120
Like they they can spend, you know, I I you can have somebody who's gonna spend a week designing, you know, some very crafted library to do something trivial, like logging something, and then they come back to you and they say, but it's an algebra, or the graph computes, you know, or it's a dag that.
00:01:46.560 --> 00:01:47.599
Logs.
00:01:47.840 --> 00:01:48.640
Yeah, yeah, exactly.
00:01:48.959 --> 00:01:50.480
You're you're like, nobody cares.
00:01:50.640 --> 00:01:52.319
We're trying to log something here.
00:01:52.560 --> 00:01:55.760
We didn't ask you to design an algebra, like this algorithm.
00:01:57.200 --> 00:02:02.239
This algorithm is probably aged out for about 10 years, though, from like 2010 to 2020.
00:02:02.480 --> 00:02:15.840
You could like take any domain, if you worked at Facebook, Google, probably a few other companies, basically take any domain, observe that it can be described as a DAG because graphs are very general objects, and then be like, it's graph log.
00:02:16.240 --> 00:02:17.439
Logs are nothing but graphs.
00:02:17.599 --> 00:02:21.280
And then then, you know, the the staff engineer promotion would follow days later.
00:02:21.599 --> 00:02:21.919
Yeah.
00:02:22.080 --> 00:02:22.319
Yeah.
00:02:22.479 --> 00:02:27.439
And slap on that some category theory jargon and and you're good to go.
00:02:27.520 --> 00:02:28.479
You're you'll be able to do that.
00:02:28.639 --> 00:02:30.319
That's that's the senior staff promotion.
00:02:30.400 --> 00:02:33.280
That's the category theory is how you get to, yeah.
00:02:34.159 --> 00:02:36.000
No, but I I hear what you're saying.
00:02:36.080 --> 00:02:50.319
And and it it is so there's some truth to what you so you know, I I had something similar going on when I was in California, where, you know, I moved to California and everybody was going on and on about treks and nature.
00:02:50.479 --> 00:03:10.000
And and you know, I I like going on treks and I think nature is beautiful, but there was some of it that was a little bit over the top, that was really like, you know, uh a little bit uh kind of kind of a way to uh to you know show show that they cared about these things and whatnot.
00:03:10.240 --> 00:03:13.520
And just to be an asshole, I was like, yeah, I don't care about nature.
00:03:13.680 --> 00:03:14.319
I I don't care.
00:03:14.479 --> 00:03:18.479
No, I stay in the city, you know, good old dud of the city, I love it.
00:03:18.719 --> 00:03:35.599
And you know, I I think that there's some of that going on with programming languages, especially on your end, where you do know that notations are important, you do know that all of this stuff matters, but you know you have a language node coming on uh coming around and you're like, yeah, well, I don't care.
00:03:36.000 --> 00:03:38.400
It's all you know, notation around the chewing machine.
00:03:46.000 --> 00:03:47.919
Uh if I may, kind of briefly too, right?
00:03:48.560 --> 00:03:49.199
Go ahead, go ahead.
00:03:49.599 --> 00:03:57.360
There's this like I I'm joking around about assembly a little bit, but that was kind of a little bit of a continuity of like VMware to uh to Facebook for me.
00:03:57.439 --> 00:04:19.680
It was like VMware was kind of based on like holding our nose and dealing with a really messy, path-dependent, stupid architecture, which is like the x86 and the PC, which is which like already in the 2000s was this like palimpsist of like unplanned layers built upon unplanned layers, like an archaeological expedition or whatever, how you go through, you know, and different times the city burned to the ground and got rebuilt or whatever.
00:04:19.920 --> 00:04:22.480
x86 was always already that in 2001.
00:04:22.720 --> 00:04:28.639
And all the sort of smart people and academics and and people with PhDs like wanted to deal with these nice, pretty machines, right?
00:04:28.720 --> 00:04:36.959
They wanted to deal with like risk machines where there's like a uniform register file, and it's really easy to target your compiler to it, and it's like really easy to encode instructions, the decode instructions.
00:04:37.040 --> 00:04:40.800
And x86 is this like ridiculous behemoth with a weird encoding for instructions.
00:04:40.959 --> 00:04:46.000
If you look at the chip, a shocking amount of resources is there to like figure out what the instructions are even saying.
00:04:46.319 --> 00:04:49.680
And uh, but it was commercially really important.
00:04:49.759 --> 00:04:58.000
And if you're willing to sort of hold your nose and make awesome tools for it, you could make the world a lot better because for whatever reasons, even if they're bad reasons, there were a lot of people on it.
00:04:58.079 --> 00:05:01.439
And I think like PHP kind of you know rhymes with that in certain ways, right?
00:05:01.519 --> 00:05:04.879
PHP wasn't really designed to kind of go.
00:05:05.199 --> 00:05:09.040
JavaScript is the poster boy is this situation, right?
00:05:09.199 --> 00:05:12.160
The whole web is running on it, so what are you gonna do, right?
00:05:13.199 --> 00:05:14.000
No, I think that's a good thing.
00:05:14.160 --> 00:05:19.040
I think one of the one of the things too is just like you you have this outlet too, which is like, let's say I'm a Haskell person.
00:05:19.519 --> 00:05:22.319
Uh-uh, and and I I agree notation matters, by the way.
00:05:22.399 --> 00:05:26.879
We'll get into like why I agree and why I'm not gonna carry this argument too too far in a hot second.
00:05:26.959 --> 00:05:28.959
But like, say you're a Haskell person, right?
00:05:29.120 --> 00:05:32.959
Sort of prototypical, my language is better than your language, weenie.
00:05:33.199 --> 00:05:33.360
Right.
00:05:33.439 --> 00:05:41.680
Like I think I think the Haskell community probably will admit that they take the cake for like, we will mog on you about how your language is inferior from most other communities.
00:05:41.759 --> 00:05:43.680
Maybe, maybe lean for is creeping up there these days.
00:05:43.759 --> 00:05:44.399
I don't know.
00:05:44.639 --> 00:05:49.040
Um you still like can just write a 6502 interpreter in Haskell, right?
00:05:49.199 --> 00:06:00.000
Like you still can just like write an an accidentally or intentionally Turing complete thing for some crummy language in the beautiful, lovely, you know, easy to reason about world that you're in.
00:06:00.160 --> 00:06:03.600
And people accidentally find themselves doing that without having intended to do it.
00:06:03.759 --> 00:06:04.240
You know what I mean?
00:06:04.319 --> 00:06:09.759
Like you start out by parsing YAML and then you have like a way to refer to another block of YAML and blah, blah, blah, blah, blah.
00:06:10.000 --> 00:06:13.920
Next thing you know, you've actually like got a little language in there and you've got a crummy interpreter for it.
00:06:14.079 --> 00:06:20.879
And, you know, and all your beautiful Haskell is doing is creating another one of these accidental messes of a language.
00:06:22.160 --> 00:06:31.600
Okay, so what you're saying is it you you just need to go one layer deep, uh, one one layer above, and and you can recreate the mess that you were trying to avoid.
00:06:31.759 --> 00:06:38.160
I think there's a lot of truth to that, especially in Haskell when you find yourself using a lot of monads.
00:06:38.319 --> 00:06:47.279
Um for me, when when you write monadic code, you're basically writing in a in a procedural programming language without realizing it.
00:06:47.360 --> 00:06:49.680
So there's definitely do, right?
00:06:51.439 --> 00:06:51.920
Yeah, exactly.
00:06:52.000 --> 00:06:55.759
They they also have well that's that's syntactical sugar, but yeah.
00:06:55.920 --> 00:07:00.720
Yeah, it's true that even the the syntax feels like a procedural language.
00:07:00.959 --> 00:07:04.480
But I mean for JavaScript in PHP, they went pretty far.
00:07:04.720 --> 00:07:14.319
It wasn't just, you know, not you know some, you know, it it was so it w went pretty far in the other direction, right?
00:07:14.399 --> 00:07:30.480
When when equality is not transitive things or reflexive or any of the basic parties you would expect from equality, that's well then again, if you're going to compare floats, then I don't think you can be reflexive, right?
00:07:30.720 --> 00:07:34.079
Because um nan is not equal to nan.
00:07:34.240 --> 00:07:35.360
And and I think so.
00:07:35.439 --> 00:07:36.800
I don't know if this story is true.
00:07:36.879 --> 00:07:43.279
We should have we should look it up after after, but I think let us up in the comments if Julian gets this part wrong.
00:07:43.600 --> 00:07:44.560
All right, cool, cool.
00:07:44.879 --> 00:07:46.319
Leave a comment of this is wrong.
00:07:46.560 --> 00:07:57.120
But I think the whole nan is not equal to nan was because they did not want to add uh an instruction to check if uh a value was nan.
00:07:57.439 --> 00:08:03.439
And so they decided to encode it this way because they didn't want to have an extra instruction.
00:08:03.680 --> 00:08:08.959
And in retrospect, you're like, ah, why would you do that?
00:08:09.360 --> 00:08:18.000
As a person who like came to scientific compute relatively late in my life, by the way, like a bunch of these nuances about how you represent floats were like a little hazy for me for a long time.
00:08:18.160 --> 00:08:30.560
So like briefly, right, you need to you need you instead of embracing the kind of exception on things like division by zero and stuff approach uh in in most scientific compute, live, you know, runtime environments, right?
00:08:30.639 --> 00:08:36.879
Combinations of libraries and hardware, there's instead this this bit pattern that represents not a number.
00:08:37.200 --> 00:08:38.960
Where do you find bit patterns that aren't floats?
00:08:39.519 --> 00:08:40.240
There's more than one, no?
00:08:40.559 --> 00:08:40.799
Exactly.
00:08:40.879 --> 00:08:43.840
Yeah, you can have there's a there's a bunch of different things that are all NANDs.
00:08:44.000 --> 00:08:50.559
Um they all would be denorms, I guess, if they were like if they were, if you're trying to interpret them as a maticid and exponent.
00:08:50.720 --> 00:09:06.559
Um and the way that people uh one of the things, one of the sort of characteristic things you fall into when you first start messing around with PyTorch, and maybe some of you are already down this road, but PyTorch and these other numeric libraries is like you run your big thing that like doesn't work or whatever.
00:09:06.720 --> 00:09:10.720
And it turns out somewhere in the middle it was like propagating nans throughout everything.
00:09:10.799 --> 00:09:15.600
And basically nan, you know, when arithmetically combined with any other number, tends to be nan.
00:09:15.759 --> 00:09:19.840
So that also, like, by the way, is going to violate and some of this is down the IEEE standard, right?
00:09:19.919 --> 00:09:28.480
So some of this is not all down to the language, but like, you know, if you have an idea that sort of zero times anything else is zero, well, zero times nan is nan, it turns out.
00:09:28.559 --> 00:09:31.120
And one point zero times nan is nan, and so on.
00:09:31.279 --> 00:09:39.759
Um, so there are there are other kinds of uh you know, identities you might expect to hold that end up not holding, like nan minus nan is is nan instead of zero, and blah, blah, blah.
00:09:40.000 --> 00:09:45.759
Um but yeah, it's like this little poison value that kind of like taints all the computes, the all the computations that go forward.
00:09:46.000 --> 00:09:54.559
Well, I mean, no, and and and you're just pointing at one of them, which is like I mean the the the crazy thing is when it comes to equality, right?
00:09:54.639 --> 00:10:02.320
Because you want uh many languages want to define structural equality that does something reasonable by default.
00:10:02.399 --> 00:10:11.519
So you want to be able to take any value in the language and be able to compare things just to be able to put them in some kind of sorted collection.
00:10:11.840 --> 00:10:17.440
And the moment there's a floating point in there, well, you have a dilemma, right?
00:10:17.600 --> 00:10:28.720
Like either you use uh the comparison functions that are implemented by the floating point unit, or you write your own custom thing.
00:10:28.960 --> 00:10:38.159
If you use the one that's by the FPU, that means that the moment there's uh a NAN in there, well, all everything everything is broken.
00:10:38.480 --> 00:10:45.440
So that was a very, very unfortunate choice that you know NAN is not compatible with NAN.
00:10:45.759 --> 00:10:51.519
Not not having a total order on these kind of things is uh is very unfortunate.
00:10:51.759 --> 00:11:00.080
But when it comes to JavaScript, so to go back to JavaScript, there's also and PHP, both PHP and JavaScript do that.
00:11:00.320 --> 00:11:03.039
They insist on converting things all the time, right?
00:11:03.440 --> 00:11:04.879
They want to beef the program logic.
00:11:05.200 --> 00:11:13.120
If there's some crazy interpretation of like what you might have meant by comparing these two things, it's gonna guess that must have been what you meant, yeah.
00:11:13.840 --> 00:11:16.879
And you know, sometimes the guessing is really strange.
00:11:17.039 --> 00:11:33.679
So for example, when it comes to uh you know using um a key, so uh when you do two strings on an array in JavaScript, it produces uh the word array with uh capital A, a paquase A.
00:11:33.919 --> 00:11:44.320
And so that conversion is is super unfortunate because imagine you end up passing an array into something where that wasn't expected.
00:11:44.559 --> 00:11:56.399
Now it's going to be converted to this string, and then if it was compared to uh, you know, another string that will it will look like it's working, but it doesn't do at all what you think it should be doing.
00:11:56.720 --> 00:11:59.519
And PHP does a lot of that too.
00:11:59.600 --> 00:12:10.879
These are the two languages that you know, like PHP also has an obsession with converting um strings that look like integers to integers.
00:12:11.039 --> 00:12:17.840
So whenever you have a hash table and you use a string as a key, it's going to try very hard to look.
00:12:17.919 --> 00:12:20.320
Like, does this thing look like an integer?
00:12:20.399 --> 00:12:22.080
Like, does it start with a digit?
00:12:22.240 --> 00:12:29.840
And it's what's interesting too is that the the conversion is not just converting, you know, string to it.
00:12:30.480 --> 00:12:32.000
It's not the same function, right?
00:12:32.080 --> 00:12:41.759
It's it's really this special thing that's like, okay, this is used as a key, so I need to figure out if this thing is really an integer before I'm gonna before doing something with it.
00:12:42.639 --> 00:12:53.120
And yeah, I must admit that sometimes I just wish that it had been done differently, or that, you know, somebody had actually sat down.
00:12:53.279 --> 00:13:12.960
And and to it's true that, you know, I think the programming language community has some, you know, has some share some responsibility there because back in the 90s we needed a language to do web development, and you know, they were too busy developing things that nobody cared about.
00:13:13.039 --> 00:13:15.919
And the ergonomics of them were something like PHP.
00:13:16.480 --> 00:13:21.279
Yeah, and like the Lamp Stack was like worked great, basically, was easy to get started with.
00:13:21.440 --> 00:13:27.279
Um, and and you could sort of as a non-programmer understand it and get something done some of the time.
00:13:27.519 --> 00:13:36.639
And that was like enough that that was enough product market fit for PHP to I think it's arguably still the largest language if you sort of survey web backends.
00:13:36.720 --> 00:13:39.279
Um that like correct me if I'm wrong, I think that's still the case.
00:13:39.440 --> 00:13:41.279
I'm sure JS is coming up.
00:13:41.519 --> 00:13:44.080
Um I imagine the prevalence of coding.
00:13:44.240 --> 00:13:46.399
Yeah, I imagine the prevalence of coding agents will eventually start.
00:13:46.720 --> 00:13:49.279
Careful because O'Camel is certainly around the corner.
00:13:49.919 --> 00:13:56.720
Yeah, I think maybe in the corner of the south of France you're in, OCaml's doing numbers, but like I'm not and like and I'm like in say too much.
00:13:58.720 --> 00:14:04.240
And obviously, like I'm sitting here talking with you, and like we've worked on PL implementations and designs for a chunk of our careers.
00:14:04.480 --> 00:14:10.799
Like I when I make fun of programming language nerds, I do so from you know, in a way that includes myself and yourself.
00:14:11.120 --> 00:14:16.559
And, you know, arguing that notation matters, like think about calculus, right?
00:14:16.639 --> 00:14:22.000
There was sort of a long, there were centuries where basically a handful of people had a chance to understand the calculus.
00:14:22.159 --> 00:14:26.720
We teach it to high school seniors now, and arguably the difference is that the notation got better.
00:14:26.879 --> 00:14:28.799
Like maybe we got a little better at teaching it or whatever.
00:14:28.960 --> 00:14:31.120
Certainly kids didn't get any smarter or more diligent.
00:14:31.200 --> 00:14:33.600
Um, at least I'd be really surprised if that was the answer.
00:14:33.759 --> 00:14:41.120
I think we just like have a way of writing down calculus that like fits in the human brain a little more ergonomically than like the crazy Leibniz notation or whatever, right?
00:14:41.360 --> 00:14:43.360
Um I don't even know what Newton's notation was like.
00:14:43.519 --> 00:14:48.559
I've seen like examples of Leibniz's notation, and it is really gobsmackingly hard to understand what he's talking about.
00:14:48.879 --> 00:15:02.639
So uh and and arguably notation in that sort of computer, in that sense of computer programmers, like being able to figure out what the meaning of what they're writing has made, like has moved forward, right?
00:15:02.720 --> 00:15:09.519
Like if you look at sort of PLI or some of these crazy like mid-20th century high-level languages, like they're garbage.
00:15:09.600 --> 00:15:12.960
You would not want to spend a single afternoon trying to get software working in these things.
00:15:13.360 --> 00:15:16.240
Excuse my French, I know everybody's doing the best they could with what they had.
00:15:16.399 --> 00:15:22.639
Um, but like until you get to Pascal or so, it's sort of like I'm not even sure what life would be like, if that makes any sense.
00:15:22.879 --> 00:15:25.200
Um and I was always surprised.
00:15:25.360 --> 00:15:34.000
This is like some of this is like a hobby horse I used to ride, and I'm not sure that like it, I need to ride it anymore now that it's like now that the coding agents are gonna do all the actual character level work for us.
00:15:34.159 --> 00:15:43.039
But like it surprises me that there's a perspective on programming language quality that that never got like borrowed stuff from experimental psychology, right?
00:15:43.120 --> 00:15:46.559
Because at some level, this is like a claim about human intellects, right?
00:15:46.639 --> 00:15:52.960
That like human minds kind of grok modern-day calculus notation better and they grok perhaps JavaScript better.
00:15:53.039 --> 00:15:55.120
I'd say definitely O'Camel better.
00:15:55.440 --> 00:16:01.440
Open question about whether Haskell's better or not, but like, you know, once it compiles, it works in many cases for Haskell, and that's interesting.
00:16:01.600 --> 00:16:02.879
Um, if that's what you can do.
00:16:03.120 --> 00:16:08.080
I mean, it's it's different because it needs a lot of um, you know, algebraic properties too, right?
00:16:08.159 --> 00:16:15.200
It's it's it's like, you know, asking if you should design mathematical notations using psychology.
00:16:15.440 --> 00:16:21.039
I don't know if it's going to be enough because you need, yes, you need something that's appealing to a human eye.
00:16:21.360 --> 00:16:27.679
But I don't know that psychology, I mean, so there's some psychology to it, if you want to call it that.
00:16:28.080 --> 00:16:37.759
But the the part that is going to matter more is going to be um, you know, does it compose well things, things, things of the sort, right?
00:16:37.840 --> 00:16:41.200
Is it it does it have does it have good good properties?
00:16:41.440 --> 00:16:44.480
And I don't know that psychology can help you answer those questions.
00:16:44.639 --> 00:16:47.360
But I mean there are different ways a language can win though.
00:16:47.440 --> 00:16:51.600
So there are some languages that win just by the sheer number of people who are using them.
00:16:51.840 --> 00:16:55.440
And then there are others who win by influence, so it'd say.
00:16:55.600 --> 00:17:02.879
So, you know, if you look at today's languages, the number of them that have that has pattern matching in them is huge.
00:17:03.039 --> 00:17:06.240
It it didn't used to be the case like 30 years ago, right?
00:17:06.480 --> 00:17:07.920
Yeah, that's a win and now pattern matching.
00:17:14.000 --> 00:17:16.880
This function means this, on that kind of datum, it means that.
00:17:17.039 --> 00:17:18.079
Um, as opposed to the data.
00:17:18.240 --> 00:17:23.680
Yeah, I mean what's interesting is that it does it does both a switch and uh binding.
00:17:23.839 --> 00:17:25.519
So that's the interesting aspect of it.
00:17:25.680 --> 00:17:36.079
Well, when you just do a switch, um you you you're not doing the binding at the same time, and things get even more interesting when you go multiple layers deep.
00:17:36.240 --> 00:17:43.039
So you can say it's something that looks like this, then that talks at that about more than one layer.
00:17:43.279 --> 00:17:47.920
So when whenever you're working with trees, so imagine you're trying to balance a tree, for example.
00:17:48.240 --> 00:18:00.319
Trying to balance a tree, you want to pattern match more than one layer deep, and you want to go get the you know two layers deep leaf and move it around and do things of the sort.
00:18:00.880 --> 00:18:04.640
With pattern matching, you read the code, it makes perfect sense, right?
00:18:04.880 --> 00:18:07.680
If you have to write that by hand, you'll you'll get to it.
00:18:07.839 --> 00:18:10.960
Of course, it's true and complete, so you can do whatever you want.
00:18:11.279 --> 00:18:13.279
But it's it's definitely less readable.
00:18:13.440 --> 00:18:15.039
And another big one are closures.
00:18:15.200 --> 00:18:18.079
Uh closures are now everywhere, including in Java.
00:18:18.480 --> 00:18:19.920
Reversal consensus that you want that.
00:18:20.079 --> 00:18:20.400
Yeah.
00:18:21.359 --> 00:18:21.599
Exactly.
00:18:21.680 --> 00:18:26.240
Which inverse closures in Python are a comp or a complaint everybody has about Python these days.
00:18:26.319 --> 00:18:27.440
It's no longer a weird perspective.
00:18:27.839 --> 00:18:30.720
Yeah, because they're they are one line long or something like that, right?
00:18:30.960 --> 00:18:31.759
Yeah, expression.
00:18:32.160 --> 00:18:32.640
Another one.
00:18:33.279 --> 00:18:33.599
Yeah.
00:18:33.759 --> 00:18:40.319
And so the the the thing that's interesting is that I think the functional world is kind of winning this way.
00:18:40.480 --> 00:18:47.599
So we don't see a big functional language that is taking over, um, with the exception of Rusts, if you want to call it functional.
00:18:47.680 --> 00:18:49.519
I don't know, that's a bit of a stretch.
00:18:49.759 --> 00:18:59.359
Um, but what's interesting with functional the functional paradigm is that it seems to be influencing the rest of the rest of the ecosystem.
00:18:59.519 --> 00:19:04.799
All the other languages seem to be adopting and and so there are some languages like that.
00:19:04.960 --> 00:19:07.839
So, for example, small talk is definitely like that.
00:19:08.240 --> 00:19:17.440
It was never very popular, in the sense that I don't I don't know, you know, a big industry that has been using small talk um seriously.
00:19:17.599 --> 00:19:20.400
Uh, but it has been incredibly influential.
00:19:20.640 --> 00:19:32.240
And you can see a lot of its macro systems and the way it's uh the the way its objects are built and all that clearly had a ton of influence with the languages that came after.
00:19:32.480 --> 00:19:44.880
So I think there are different ways you can have an impact on languages, but I'm still baffled at how horrible the mainstream languages are.
00:19:45.039 --> 00:19:55.119
I'm still baffled by the fact that those who are the most popular and uh are so incredibly wrong in so many ways.
00:19:55.359 --> 00:19:58.400
So like when you love to take the outside perspective on this, right?
00:19:58.480 --> 00:20:00.079
Like what would an economist say or something?
00:20:00.240 --> 00:20:02.000
Like, like let's take the outside perspective on this.
00:20:02.079 --> 00:20:03.599
Because like I share this with you too, right?
00:20:03.680 --> 00:20:08.880
It's like there's a like pop programming language culture is garbage culture, if that if that makes any sense, right?
00:20:08.960 --> 00:20:13.200
It's kind of a it's a little bit like So let me clarify something before we get into that.
00:20:13.519 --> 00:20:19.279
The fact that it's garbage doesn't really matter in the sense that I'm not talking about taste here.
00:20:19.599 --> 00:20:27.119
What I'm talking about is all that garbage makes it incredibly hard for people like me to write good tools for these things.
00:20:27.279 --> 00:20:28.799
And that's what's frustrating me.
00:20:29.039 --> 00:20:46.000
So if I take something like JavaScript, I want to write a tool that does any kind of analysis, it's basically impossible, not impossible, but I've I'll be spending 95% of my time fighting JavaScript idiosyncrasies instead of doing my job.
00:20:46.240 --> 00:20:50.240
So it it feels like you know, HP, as we recall well, right?
00:20:50.400 --> 00:20:52.400
Like yeah, it's not it's not just JavaScript.
00:20:52.480 --> 00:20:56.000
It's imagine writing a jit for PHP.
00:20:56.480 --> 00:20:56.960
I know, right?
00:20:57.119 --> 00:20:57.759
How crazy would you have to do that?
00:20:59.279 --> 00:21:00.240
Would you go to the page?
00:21:00.400 --> 00:21:03.519
You'd probably never say anything smart ever again in your life if you did that.
00:21:04.000 --> 00:21:04.960
Exactly.
00:21:05.200 --> 00:21:21.119
Um and so no, what was interesting too is that I think people underestimate that when when they so I think most people think that I'm just being a douchebag who is being pedantic about programming languages and don't realize that it's not that.
00:21:21.279 --> 00:21:30.559
It's just that whenever I see these things, I'm thinking, oh my god, that whoever wants to write a tool on this is gonna take forever.
00:21:30.720 --> 00:21:34.799
And and another example is so let's say you're trying to build something efficient, right?
00:21:34.880 --> 00:21:37.759
Like let's say you try to compile JavaScript efficiently, right?
00:21:37.920 --> 00:21:39.119
Or PHP.
00:21:39.440 --> 00:21:52.640
Well, if it wasn't JavaScript, or if it wasn't PHP, imagine somebody, you know, in the 80s, 90s, they all figured out that really what they needed was some kind of syntax around Lisp.
00:21:52.960 --> 00:21:58.720
You know, imagine the creator of JavaScript had figured out well, wait a minute, Lisp has all the features I need.
00:21:58.880 --> 00:22:00.720
I just don't like to Syntax.
00:22:00.799 --> 00:22:04.960
So I'm just going to write an alternative syntax for Lisp, right?
00:22:05.279 --> 00:22:08.079
And then the same with PHP, and then the same with Ruby.
00:22:08.160 --> 00:22:12.160
And the same with all these, you know, um dynamic languages.
00:22:12.319 --> 00:22:14.480
There's no you don't need Ruby.
00:22:14.559 --> 00:22:15.839
You don't need Python.
00:22:15.920 --> 00:22:17.359
You don't need JavaScript.
00:22:17.519 --> 00:22:22.640
You could have written all these things as an alternative syntax on top of Lisp, right?
00:22:22.799 --> 00:22:23.440
So just take the page.
00:22:30.640 --> 00:22:33.680
No, but they they could have just taken the syntax they wanted, right?
00:22:33.759 --> 00:22:43.839
But then compile transpile them basically, even on the fly, you wouldn't need tooling for that, but just something that just basically runs it on a Lisp machine, right?
00:22:44.079 --> 00:22:46.480
Where it's the functions are properly.
00:22:46.640 --> 00:22:57.200
And then had they done that, we wouldn't have all the problems that we had with PHP, because you wouldn't have, you know, the Frankenstein of a data structure that is the PHP array.
00:22:57.359 --> 00:23:06.480
We wouldn't have, you know, the the way the locals work, where the the binding of the locals is just and the references don't get me started on that.
00:23:06.720 --> 00:23:13.440
And now we would have basically all those languages compatible with each other, all of them super efficient.
00:23:13.759 --> 00:23:19.920
All the people working on JIPS, they could focus on this one thing and make that really fast.
00:23:20.319 --> 00:23:27.680
But no, what we have is very talented people spending a long time, you know, optimizing something like JavaScript.
00:23:27.759 --> 00:23:31.519
And if you try to do it generically, it doesn't work.
00:23:31.680 --> 00:23:38.799
If you take something like JavaScript, you try to compile it to the JVM, thinking, hey, the JVM is super efficient, so let's do that.
00:23:38.880 --> 00:23:39.920
It's going to be fast.
00:23:40.160 --> 00:23:42.240
No, it's going to be horribly slow.
00:23:42.319 --> 00:23:43.200
It's not going to work.
00:23:43.359 --> 00:23:46.160
Because the JVM assumes some patterns.
00:23:46.240 --> 00:23:51.039
And that's a mistake that a lot even at Facebook, I think I remember some people tried that.
00:23:51.440 --> 00:23:51.839
There was an effort.
00:23:53.200 --> 00:24:02.720
And I think there's a there was a was an IBM research paper from like 2013 or 14 that I ended up just like kind of pointing people at uh that with the title The Repurposed JIT Phenomenon.
00:24:02.799 --> 00:24:04.079
And it was by a guy named Castaños.
00:24:04.319 --> 00:24:06.480
I think Jose Castaño said IBM Research.
00:24:06.640 --> 00:24:07.920
And he was looking at a different system.
00:24:08.000 --> 00:24:09.839
He was looking at, I think, Iron Ruby.
00:24:10.000 --> 00:24:15.599
So Iron Ruby was uh a Ruby front end that targeted the CLR, so Microsoft's.net runtime.
00:24:15.920 --> 00:24:20.480
Kind of, you know, from 100,000 feet comparable to the JVM in many ways, CLR implementation.
00:24:20.559 --> 00:24:23.119
I'm sure CLR people would rip my head off for saying that.
00:24:23.279 --> 00:24:26.319
There's lots of nice things about the CLR and lots of nice things about the JVM.
00:24:26.480 --> 00:24:32.240
But from 100,000 feet at similar sort of application domains, and C sharp is, you know, has a similar audience to Java and so on.
00:24:32.400 --> 00:24:36.880
And the problem that they run into, and then I'm not sure you wouldn't run into in this world too, by the way, Julian.
00:24:37.039 --> 00:24:43.680
The problem they run into is like just the pro the people who are writing Ruby programs are not the people who are writing C sharp programs.
00:24:43.839 --> 00:24:46.400
And the CLR engineers have to make trade-offs everywhere, right?
00:24:46.480 --> 00:24:58.480
They have to make decisions about how the garbage collector is going to work, like what to inline, what not to inline, you know, when a method is sort of, you know, is cacheable and not cat when a when a binding of the method is cacheable and it's not cacheable and so on.
00:24:58.720 --> 00:25:04.160
And broadly, Ruby programmers just like write really different stuff with really different statistics than CLR programmers.
00:25:04.319 --> 00:25:09.119
And so you try and sort of make all these trade-offs for one population of programs and you kind of overfit to it.
00:25:09.200 --> 00:25:16.480
It's like literally, I mean, they don't use the terminology from statistics and machine learning, but it is literally a kind of overfitting to that domain.
00:25:16.640 --> 00:25:20.400
And then you bring along this other sort of out-of-distribution population of programs.
00:25:20.480 --> 00:25:23.200
And it's hard to run that out-of-distribution population of programs well.
00:25:23.359 --> 00:25:29.519
And actually, this is one of the things, by the way, since we're, you know, we live in these sort of AI-based times.
00:25:29.680 --> 00:25:50.160
Um, one of the things that I've really liked from the ML world that I wish I could bring, you know, for a while when I was still sort of doing reviews for compiler and VM journals and things like that, is just the concept of overfitting is not present all the time in the research community around compilers and and code generation and other people that are sort of doing performance running PL work.
00:25:50.319 --> 00:25:52.319
And so, you know, we train on the test set all the time.
00:25:52.400 --> 00:25:54.720
That's what benchmarks are in some ways, right?
00:25:55.119 --> 00:25:59.839
And you've got enough knobs on the side of your compiler that you can overfit to the benchmark, right?
00:25:59.920 --> 00:26:10.559
There's enough sort of parameters around what do I inline, what do I out of line, you know, what do I, when do I throw away a translation, when do I bring it back, and so on, um, that you really can overfit to a population of benchmarks quite easily.
00:26:10.640 --> 00:26:12.880
And that's like a robust thing that's actually happened.
00:26:12.960 --> 00:26:16.000
Um, and there's a guy um at King's College London.
00:26:16.160 --> 00:26:18.240
Um we'll have his name in the comments.
00:26:18.319 --> 00:26:19.759
I'm gonna start this embarrassing.
00:26:20.000 --> 00:26:20.319
Thank you.
00:26:20.640 --> 00:26:21.599
Lawrence Tratt.
00:26:22.079 --> 00:26:23.519
Yeah, Lori, Lori Tratt.
00:26:23.680 --> 00:26:32.079
Uh, goes by Laurie, but Lawrence Tratt in the in the citations, uh, which is, you know, uh your JIT blows hot and cold, basically, right?
00:26:32.160 --> 00:26:43.039
That like you have this intuition that you're just in time compiler, doesn't know anything about your program, learns all about it while it runs for a short little bit, and then reaches an equilibrium where it's like got all this nice efficient code that's optimized for your program.
00:26:43.200 --> 00:26:47.440
But in reality, like deoptimization is an arrow that points backwards in that graph, right?
00:26:47.519 --> 00:26:51.119
Like the your JIT runtime eventually decides, like, oops, I got this one wrong.
00:26:51.200 --> 00:26:55.119
I'm gonna actually undo it and you know, unroll this loop or inline this method or whatever.
00:26:55.359 --> 00:27:00.640
And that space of possible programs it can reach is actually really big, and there's no guarantee that it ever really settles down.
00:27:00.880 --> 00:27:12.240
Um, so I think this is actually the the world you're pointing at, by the way, Julian, with a Lisp core that sort of everything syntactically retargets is very much like the racket scheme ecosystem, right?
00:27:12.319 --> 00:27:25.279
This is sort of the vision for racket at some level, was that you'd have this like language-oriented environment where almost everything's a DSL, but at its core it would boil down to this little uh to this particular scheme dialect that they call Rackets.
00:27:25.920 --> 00:27:29.279
So I think the the CLR wanted to do that too, right?
00:27:29.359 --> 00:27:34.240
They wanted to be at some point they were advertising.NET as the universal language.
00:27:34.400 --> 00:27:38.400
I think you have two broad categories to split up here.
00:27:38.559 --> 00:27:45.359
So I would say a Lisp engine would be perfect to run any dynamic language, right?
00:27:45.519 --> 00:27:47.279
I'm not talking about static language.
00:27:47.440 --> 00:27:54.880
Static languages are a completely different beast, and if you want to compile them efficiently, then you have to consider a bunch of things.
00:27:55.039 --> 00:28:06.640
But the reason why I'm I I'm I was you know talking about Python, Ruby, um JavaScript, those languages, there's nothing special about them.
00:28:06.799 --> 00:28:18.160
The only thing that's special about them is that they added a bunch of idiosyncrasies that are specific to this language that don't bring that much to the language.
00:28:18.400 --> 00:28:29.039
I could I could have basically built a version of PHP that looks exactly like PHP, target targets Lisp, and no developer would have ever known the difference.
00:28:29.359 --> 00:28:39.680
They would have looked at it as like, yeah, it looks like PHP to me, and would have accept that of course that wouldn't work in practice because a ton of libraries are going to depend on some behavior that you wouldn't want.
00:28:39.839 --> 00:28:54.240
So that's why I'm pointing at those languages specifically, because I think that had they been written on top of a Lisp engine, the result would have been much, much better because we would have had what you wanted from PHP.
00:28:54.319 --> 00:29:05.440
So typically PHP, when it came out, what you wanted is the web was basically a gigantic HTML page with a little bit with a few things that were dynamic in it, right?
00:29:05.519 --> 00:29:11.039
Like it was HTML and then in the middle some SQL query and stuff like that, right?
00:29:11.279 --> 00:29:14.079
And PHP just made that super easy, right?
00:29:14.160 --> 00:29:19.759
Like you just wrote your HTML and in the middle of it you just had the special tag and did whatever you wanted, right?
00:29:20.160 --> 00:29:34.400
So had that thing been built on top of a Lisp engine, well, all the constructions would have been, you know, properly crafted, like you would have had proper closures and proper everything, right?
00:29:34.480 --> 00:29:44.319
The the lexical uh uh scoping would have been good and and all the things that they they fucked up, basically, excuse my French, would have worked correctly.
00:29:44.400 --> 00:29:46.240
And the same is true with JavaScript, right?
00:29:46.400 --> 00:29:56.799
Like imagine when you look at JavaScript, you're not thinking, oh yeah, if I had to encode this in in Lisp, it would look look nothing like normal Lisp.
00:29:57.039 --> 00:30:04.960
No, it would be just like a normal Lisp program, but they wouldn't be all these quirks that were there that are there for no good reason.
00:30:05.039 --> 00:30:10.720
Like the the fact that this is bound dynamically in JavaScript just doesn't make any sense.
00:30:10.880 --> 00:30:12.480
Like, why would you do that?
00:30:12.720 --> 00:30:15.440
Um and and and all sorts of things of the sort.
00:30:15.519 --> 00:30:16.480
So but that's the same.
00:30:16.640 --> 00:30:18.079
Can I interrupt this theory for a second?
00:30:18.400 --> 00:30:23.759
Can I interrupt this perspective for like uh with an inconvenient set of facts about this perspective?
00:30:24.079 --> 00:30:25.440
Um and I'm with you by the way.
00:30:25.519 --> 00:30:28.319
Like I like I think like I agree that there's something kind of totalizing about Lisp.
00:30:28.400 --> 00:30:35.440
It's like a just to finish, I I don't mean that all languages should target a Lisp engine that just wouldn't work.
00:30:35.839 --> 00:30:37.200
Just the dynamic languages.
00:30:37.599 --> 00:30:39.920
Yeah, I don't take this to be a prescription per se.
00:30:40.160 --> 00:30:56.079
But so what I'm curious about here, because you're you're making this sort of dynamic versus static dichotomy, and I agree with you, like that most of these dynamic languages are just kind of some kind of notation around a dynamic tree, and yeah, could we could have represented it with parentheses and nobody'd been the wiser.
00:30:56.400 --> 00:31:06.480
Um we have the fact in the ecosystems out there that all of the Lisp, including Common Lisp and Scheme implementations, are ahead of time oriented, right?
00:31:06.559 --> 00:31:16.240
Most of them like if they're interested in compiling stuff, they do so, you know, method by method in the case of Common Lisp, I guess, but like you still are producing machine code without seeing the darn thing run.
00:31:16.480 --> 00:31:32.000
And it was actually small talk uh mentioned earlier that sort of blaze the trail of JIT implementations of languages, like that blaze the trail of like what if you have a deeply dynamic language where you really can't know where this method leads and you barely even have if statements and and almost everything is binding, right?
00:31:32.400 --> 00:31:44.400
Um and what I think there's a for those of you that haven't played around a small talk, by the way, you can it takes a while to kind of get your head around because the notation looks a little different or whatever.
00:31:44.640 --> 00:31:56.640
But it's like you can sort of squint and it's a little bit like JavaScript in like that everything's an object and you can rebind the methods on things, and people actually do so, and it sort of matters for performance that you handle that in a semi-efficient way, and so on.
00:31:56.880 --> 00:31:59.519
Um, I guess like what do you make of that pattern of facts, right?
00:31:59.680 --> 00:32:08.000
Because I think the the common list people are share with Smalltalk that they would often deliver applications as like a system image, right?
00:32:08.079 --> 00:32:17.599
You'd sort of like get your common list thing into a state where it's ready to interact with, and then you'd like snapshot it, and the way that I run your application is resuming that snapshot, um, which also how small talk works.
00:32:18.000 --> 00:32:24.799
Yeah, so I I I I've I've realized why while listening to you that I just didn't exp express myself very clearly.
00:32:24.960 --> 00:32:32.000
So basically what I'm trying to say is not that common list should be the basis for all dynamic languages.
00:32:32.160 --> 00:32:44.240
What I was trying to say is that there are some of those dynamic languages that were created back in the days that would have been much, much better off just ending ending up being just some kind of syntax on top of an existing language.
00:32:44.480 --> 00:32:47.839
So for example, for JavaScript, that might have been yeah, small talk.
00:32:47.920 --> 00:32:55.279
Now that now that you're you're mentioning it, I think it might have made more sense to you know make that an alternative syntax.
00:32:55.519 --> 00:32:57.119
And and there's nothing wrong with that.
00:32:57.200 --> 00:32:59.200
Like I think it was what what was it?
00:32:59.359 --> 00:33:01.519
What was the name of the alternative syntax for JavaScript?
00:33:01.599 --> 00:33:02.319
Was it Clojure?
00:33:02.480 --> 00:33:02.640
No.
00:33:03.119 --> 00:33:05.680
A CoffeeScript or CoffeeScript, on a couple of things.
00:33:05.920 --> 00:33:07.599
Coffee script, yeah, Clojure is something else.
00:33:07.920 --> 00:33:30.720
Uh there was um so they so had Python and PHP been some kind of alternative syntax for Lisp and JavaScript some kind of alternative syntax for Smalltalk, I think we would be so much better off today because we would have to deal with systems that are much saner, if you will.
00:33:31.119 --> 00:33:48.319
And and the amount of energy that has been spent on those quirks, I j if if we had to put a number on it, like how much money has been wasted because of the way a quality works in JavaScript, right?
00:33:48.400 --> 00:33:50.720
Like how is it it's in billions.
00:33:51.839 --> 00:33:52.000
Exactly.
00:33:52.079 --> 00:33:54.079
The question is how many billions and whether it's tens of billions.
00:33:54.160 --> 00:33:54.400
I agree.
00:33:54.480 --> 00:33:55.359
Like it's gotta be billions.
00:33:55.759 --> 00:33:56.720
It's it's ridiculous.
00:33:56.880 --> 00:34:06.319
The amount of money we're spent on that stuff and how long it takes to and so to all those who tell me, yeah, but it doesn't matter, I'm like, hmm, it kind of does.
00:34:06.400 --> 00:34:12.800
I mean, we not so I'm not uh among those who say, well, let's get rid of it and let's do something new.
00:34:12.880 --> 00:34:14.320
No, I'm pretty realistic.
00:34:14.480 --> 00:34:16.159
Like I know we cannot get rid of it.
00:34:16.239 --> 00:34:34.239
That's that's what we have, and these are the systems we we have in place, but there's a part of me that's like, ah, if only had we done things differently 20 or 30 years ago, and things and so another big mystery is why did didn't ML pick up?
00:34:34.400 --> 00:34:36.480
Um, so especially O'Caml.
00:34:36.719 --> 00:34:40.480
So I mean look look back in imagine you're in the 90s, right?
00:34:40.639 --> 00:34:52.639
You're in the 90s, O'Caml basically has everything that you know Go is praised for, plus everything, you know, otherling, like Scala was praised for.
00:34:52.800 --> 00:34:58.960
Like, you know, you look at OCaml, so you have pattern matching, you it's it's actually really efficient.
00:34:59.119 --> 00:35:06.960
Um when you compile it, you are maybe 2x away from C C, but not that much more than that.
00:35:07.119 --> 00:35:09.280
It's actually pretty fast to compile.
00:35:09.519 --> 00:35:14.320
It uh runs pretty much everywhere, has a very small footprint, a bit like Go, right?
00:35:14.400 --> 00:35:17.840
Where you can take the runtime, it's not a gigantic runtime.
00:35:18.000 --> 00:35:25.760
Um, it has you know a bunch of really good features, exceptions, pattern matching, a good module, a strong module system.
00:35:26.159 --> 00:35:29.360
Um, so that has been a mystery for me.
00:35:29.519 --> 00:35:29.920
Like why?
00:35:30.719 --> 00:35:36.159
As somebody who was running around in the 90s, I guess like I'm a tiny bit older than you, you probably can or can identify a little bit.
00:35:36.239 --> 00:35:38.400
So I'm I'm I'm I'm doing the mental exercise, right?
00:35:38.480 --> 00:35:39.760
It's 1995.
00:35:40.000 --> 00:35:47.039
I'm wearing my flannel, I'm like listening to a Nirvana album, I hate my dad, and and I'm excited about computers, right?
00:35:47.440 --> 00:35:52.800
And and I was excited about computers in a way that was channeled almost entirely through the C programming language.
00:35:53.039 --> 00:35:58.400
And and I knew that ML existed and I could like find these like Usenet news groups or whatever where people were talking about it.
00:35:58.559 --> 00:36:02.000
Um, but it didn't grab me and and and why didn't it, right?
00:36:02.239 --> 00:36:12.159
And I think and some of this is like path-dependent stuff about the subculture that computing happened to be, by the way, uh like and then the paths that it happened to take in different communities and stuff.
00:36:12.400 --> 00:36:15.679
But for me, I was uh interested in being a systems programmer, right?
00:36:15.840 --> 00:36:18.320
I was interested in sort of the hardware software interface.
00:36:18.639 --> 00:36:28.159
And the quality and at this time, the quality of language implementation was such that you occasionally ran into bugs in your compilers and interpreters and so on, right?
00:36:28.320 --> 00:36:29.760
And this is this has changed a little bit.
00:36:29.920 --> 00:36:35.360
Like now that's kind of like a career-defining thing if you find a real bug in your compiler, if it's like a C compiler or something.
00:36:35.599 --> 00:36:37.360
But it used to be just like you'd run into them.
00:36:37.440 --> 00:36:42.800
Like the compiler would crash, like whatever, or generate bad code, or generate stuff that was sort of slightly wrong.
00:36:43.119 --> 00:36:50.400
And rightly or wrongly, and it might have been wrongly in hindsight, but rightly or wrongly, I had the illusion that I understood what C compilers did.
00:36:50.559 --> 00:36:51.440
Does that make any sense?
00:36:51.599 --> 00:37:03.840
Like I felt like I kind of, you know, knew my way around an instruction set or two, um, you know, was able to kind of get my head around the first few chapters of like an undergrad textbook about compilers.
00:37:04.000 --> 00:37:17.760
And so I was like, oh, okay, what it's doing when I say x equals 12 is it's like got some bit pattern 12 encoded in an instruction somewhere, it puts that immediate in a register, it stores that register in the memory location associated with X, and so on.
00:37:17.920 --> 00:37:21.679
Whereas if I say X equals 12 in my OCaml program, who the hell knows what happens, right?
00:37:21.760 --> 00:37:23.599
Like like all kinds of crazy fine, fine.
00:37:23.679 --> 00:37:24.639
I buy that theory.
00:37:24.800 --> 00:37:25.039
Fine.
00:37:25.440 --> 00:37:27.840
But what what what when Go comes out?
00:37:28.000 --> 00:37:36.639
You know, Go comes out and you're like, I can't help but think oh Swift comes out, and I can't help but think, why the hell do I need Go?
00:37:37.039 --> 00:37:38.320
Why do I need Swift?
00:37:38.480 --> 00:37:40.800
Like I have OCaml, it's perfect.
00:37:40.960 --> 00:37:44.800
Like and and and and the only thing I can think of is a syntax.
00:37:44.880 --> 00:37:47.039
It's that's the only thing that I can think of.
00:37:47.280 --> 00:37:51.840
And with Swift, actually, it's not that far away from an ML syntax.
00:37:52.000 --> 00:38:03.199
So I it's still a mystery to me that you know they they say that um when you are investing in the market, if you are if you're too soon or too late, you're still wrong.
00:38:03.360 --> 00:38:07.039
Like being being too soon is also being wrong, right?
00:38:07.199 --> 00:38:09.519
And so I wonder if OCaml was just too soon.
00:38:09.599 --> 00:38:15.039
Like they they just came up with all these concepts and they were not mature.
00:38:15.199 --> 00:38:18.639
I mean, the audience was not mature enough to grasp them.
00:38:18.800 --> 00:38:19.920
I I don't know.
00:38:20.159 --> 00:38:25.360
But when Go comes out, I cannot help but think, why do I need this?
00:38:25.599 --> 00:38:26.800
I have OCaml.
00:38:27.199 --> 00:38:31.199
What does Go bring me that that that OCaml doesn't give me, right?
00:38:31.280 --> 00:38:39.280
Like it has a relatively small runtime, it has I mean, it lacks a bunch of features that OCaml does have.
00:38:39.440 --> 00:38:49.599
Um maybe it's that they wanted to simplify, I don't know, but in which case, just release a version of OCaml where you ban some features, if that's what you want to do.
00:38:49.760 --> 00:38:54.239
Or maybe they wanted more control over the the language in the compiler, I don't know.
00:38:54.480 --> 00:38:58.400
But I mean I've probably written more Go than OCaml in my life for what it's worth.
00:38:58.480 --> 00:39:03.679
And I'll uh and this is I'll I'll talk like the way when when Go first kind of exploded on the scene.
00:39:03.920 --> 00:39:06.960
Um, when was it?
00:39:07.199 --> 00:39:07.920
It sounds right around that?
00:39:08.079 --> 00:39:09.360
Yeah, eight, nine, something like that.
00:39:09.519 --> 00:39:10.880
It's been around a bit now.
00:39:11.199 --> 00:39:17.360
Um bunch of controversial choices in there, probably the most famous of which is like no generics, just it's too hard to think about.
00:39:17.440 --> 00:39:20.400
We're not gonna now they're in now.
00:39:20.559 --> 00:39:24.559
Yeah, so you can't sort of slander go as like we don't have generics anymore.
00:39:24.639 --> 00:39:25.760
They did finally fix this part.
00:39:25.920 --> 00:39:31.440
But like a pretty big omission in terms of like its peer systems languages that were coming out around that time.
00:39:31.760 --> 00:39:38.800
And I remember thinking two things, like I saw that a lot of my peers were taking it seriously basically because it had the Google stamp on it.
00:39:38.960 --> 00:39:39.039
Right.
00:39:39.199 --> 00:39:41.920
And Google was sort of more or less, you know, the IBM of that day, right?
00:39:42.000 --> 00:39:54.079
Like Google kind of stood alone atop the pyramid of of serious technology companies that really could shove a new programming, a systems programming language out there and people would take it seriously, or back end if you don't consider it systems if it's not in the kernel or whatever.
00:39:54.480 --> 00:39:58.320
I remember taking it sort of seriously specifically because of Ken Thompson.
00:39:58.559 --> 00:40:02.800
Like I remember specifically kind of being like, you know, this guy's earned the benefit of my doubt.
00:40:02.960 --> 00:40:08.800
Um, you know, had a lot to do with the genesis of Unix, like is probably, you know, more than anybody else the author of Plan 9.
00:40:08.880 --> 00:40:15.119
And I think Plan 9 in many ways is like a, you know, beautiful inheritor of Unix's TikTok involved as well, right?
00:40:15.280 --> 00:40:18.079
Rob Pike's involved as well, but Ken Thompson was in the mix, right?
00:40:18.159 --> 00:40:24.320
Uh, you know, sort of the getting the band back together of people that were still practicing of early Unix days.
00:40:24.480 --> 00:40:27.519
And Ken Thompson also like has done other crazy computing stuff, right?
00:40:27.599 --> 00:40:35.840
For like nine years running, he was the computer chess champion in like the late 70s to the early 80s, when it was like all running on PDP 7s or 11s, rather.
00:40:36.000 --> 00:40:46.320
Um, so I mean, I was just sort of like, okay, if if you're fired up about this, I will learn a little bit about it and write a little bit of it and try and figure out whether I'm excited about this or not.
00:40:46.480 --> 00:40:51.920
Um, and like, you know, the little gophers were cute and like go routine was a good name for go routine and blah, blah, blah.
00:40:52.239 --> 00:40:58.480
And some of it was just kind of like, it is a you know, credible persona that's excited about this thing.
00:40:58.639 --> 00:41:03.519
And if it was literally just him saying, like, hey, have you checked out standard ML or have you checked out OCaml?
00:41:03.599 --> 00:41:05.039
Um, I'd have been like, yeah, check them out.
00:41:05.119 --> 00:41:05.280
I don't know.
00:41:06.079 --> 00:41:06.800
They're those things.
00:41:06.880 --> 00:41:07.119
Sure.
00:41:07.199 --> 00:41:08.559
They're those functional things.
00:41:08.800 --> 00:41:20.480
Um and and I and I think Go also did, especially in those pre-generic days, like have the whether it's true or illusion, like have the ergonomics of one of those languages that maps to the machine directly.
00:41:20.559 --> 00:41:21.360
Do you know what I mean?
00:41:21.519 --> 00:41:28.719
That's not completely true because there's a GC under the hood, but like, but it felt like you could imagine the machine instructions that were going to be poured out from the stuff you're writing.
00:41:28.960 --> 00:41:31.039
Um and the GC is very, very simple.
00:41:31.119 --> 00:41:35.840
It's really a mock and sweep that goes in circles, so it's not even generational.
00:41:36.079 --> 00:41:40.719
So that's uh that makes the performance more predictive, more pr predictable as well.
00:41:40.880 --> 00:41:46.880
So no, that I mean I I buy the simplicity argument, and maybe I I didn't want to pick on Go specifically.
00:41:47.039 --> 00:41:49.519
Uh so maybe that's what came off as well.
00:41:49.840 --> 00:41:51.199
Julian Cold.
00:41:52.239 --> 00:41:59.679
It's more like I see the excitement that Go had, and I never understood why Ocamel never had a moment like that, right?
00:41:59.760 --> 00:42:01.039
Like why it never really picked up.
00:42:01.360 --> 00:42:02.480
Well, like Scala, right?
00:42:02.559 --> 00:42:03.840
Like take Scala.
00:42:04.159 --> 00:42:04.480
Right?
00:42:04.559 --> 00:42:12.960
I mean, it seems like at some level that the the energy there is partly like, hey, we took all of these things from all of these languages, and now it's on the JVM.
00:42:13.199 --> 00:42:14.880
Yeah, but Scala was a bit much.
00:42:14.960 --> 00:42:27.519
I mean, uh as much as I like uh, you know, Darski, I think he's uh he's a good eye, but I think it was a bit like the C of and I say was because the community I I it doesn't have as much influence today.
00:42:27.599 --> 00:42:33.920
It's not that the thing is dead or anything, but it clearly had a moment I would say in 2010, something like that.
00:42:34.159 --> 00:42:37.039
That's when 2000 early 2010s.
00:42:37.360 --> 00:42:42.239
And I think it was a bit the C of functional programming languages, if you will.
00:42:42.480 --> 00:42:43.440
Yeah, just put the functionality.
00:42:45.679 --> 00:42:48.800
So exactly they just put everything in there.
00:42:49.039 --> 00:42:58.880
And then uh what invariably happens when you do that is that it becomes very difficult to reason about what the semantics, what does this thing actually do?
00:42:59.119 --> 00:43:08.320
Because when those features interact with some features interact with with each other, you you lose track of what what's actually happening.
00:43:08.559 --> 00:43:18.079
And then the compilation itself was slow and so both slow at compile time, so it was actually took a while to compile Scala.
00:43:18.559 --> 00:43:22.639
Yeah, and the generated code was also not very efficient.
00:43:22.800 --> 00:43:32.239
Um, so I heard some you know wizards were able to get that thing to run efficiently and good for them, but I never got Scala.
00:43:32.559 --> 00:43:35.039
And so again, you know, Scala comes out.
00:43:35.199 --> 00:43:39.679
I could see how it's different from O'Camel because it actually targets the JVM.
00:43:40.000 --> 00:43:53.360
Uh, but again, I'm like, well, we have a perfectly functional, uh well, uh no pun intended, uh functional language that is already very efficient and nobody is using it.
00:43:54.639 --> 00:44:03.840
And now there's this thing that comes out that in my book is not as good as OCaml on any dimension.
00:44:04.800 --> 00:44:08.639
I think OCaml is a better overall language in Scala.
00:44:09.519 --> 00:44:14.880
The only thing that Scada has going for it is that it's actually targeting the it it's targeting the JVM.
00:44:15.119 --> 00:44:18.639
But then again you could have compiled OCaml to the JVM.
00:44:18.719 --> 00:44:24.960
Like they they built F sharp and I would claim that F sharp is a little bit of that.
00:44:25.039 --> 00:44:34.239
It's a l it's basically ML targeting.NET although I'm not super familiar with F sharp but I heard very good things about downsign.
00:44:34.559 --> 00:44:41.519
So I I think it's it's one of those where it's programming languages are still a mystery to me.
00:44:41.599 --> 00:45:17.679
I've been studying them for a very long time and sometimes it feels a bit like you know um like uh like how a a song becomes a hit you know like yeah for a kids driven yeah the adoption of it is literally yeah yeah and there's some other similar feeling right like artists sometimes they make a song it's their favorite song they really think it's going to be a hit and it flops completely and then there's this thing they've written one day in five minutes and it becomes this phenomenon and they're like oh wow I I never even really liked that song.
00:45:17.840 --> 00:45:18.559
That's cool.
00:45:18.639 --> 00:45:26.960
You know yeah I mean Rasmus Lordars did not think PHP was his career defining contribution when he was like slapping it together in 93 or whatever, right?
00:45:27.519 --> 00:45:34.639
Yeah he thought he was um but I I think this session would not be complete without bad mouthing C a little bit.
00:45:34.719 --> 00:45:37.119
Because you Yeah we've been really easy on C so far.
00:45:37.280 --> 00:45:38.559
Can we just shit on this for a second?
00:45:38.800 --> 00:45:39.760
Like my God.
00:45:40.320 --> 00:45:45.519
So and I say this as like a person that like earned my living writing a lot of C for quite a while.
00:45:45.760 --> 00:46:00.719
Gosh in terms of things that like so um the two cra like like there are three crazy decisions about C that made perfect sense at the time right one was complete C backward compatibility right and which by the way is not true.
00:46:00.880 --> 00:46:04.079
But I think it's not true anymore.
00:46:04.159 --> 00:46:09.119
But but it was like the first decade and a half or something of C was along this trajectory.
00:46:09.280 --> 00:46:21.440
And if and if you're curious about why this was by the way the expectation was that people's software investment and the investment of like programmer expertise in learning C was too high to overcome.
00:46:21.519 --> 00:46:31.440
So you had to like let people pound include first of all you had to have like interop with C pro C projects and be able to have a small number of C files in your C project.
00:46:31.679 --> 00:46:37.599
Those C files need to be able to use every little bit of the C project down to the syntax, right?
00:46:37.679 --> 00:46:39.760
It was not enough that there'd be some IDL or some blah blah blah.
00:46:39.840 --> 00:46:44.400
No, none of that's got to be pound include foo.h has to work.
00:46:45.760 --> 00:46:51.199
The other was the the next was no abstr uh no but cost free abstractions, right?
00:46:51.280 --> 00:46:55.039
We will only allow features into the code by the way is not true either because of exceptions.
00:46:55.679 --> 00:46:58.559
Exactly it's not true either because of exceptions, but okay.
00:46:59.280 --> 00:47:07.199
Exceptions violates it you pay for it, I mean also just like the fact that there's an economy of effort in implementing a compiler in runtime is how you pay for it too, right?
00:47:07.280 --> 00:47:26.400
Like if if if you add surface area to the language effort and correctness goes into implementing that surface area instead of making it faster or whatever else right so there's I always thought this was sort of a false economy but like there but there needs to be in God's mind right like it there needs to be some possible implementation of the thing that is overhead free.
00:47:26.719 --> 00:47:41.280
And and by the way like one of the one of the sort of excuses that the thing that got C going initially was actually like the object oriented mania of the 1990s right when people were sort of convinced that well I think what they meant was overhead free if you don't use it.
00:47:42.239 --> 00:47:49.280
They were okay with having features costing you something but what they didn't want was a feature to cost to everybody using C.
00:47:49.920 --> 00:47:50.079
Right.
00:47:50.320 --> 00:47:52.159
That was that was my understanding of the thing.
00:47:52.719 --> 00:47:54.800
That's a more precise phrasing of it you're right.
00:47:55.199 --> 00:48:08.639
And the thing that exceptions cost you if you don't use them if you're curious about this is that like you're not sure you can run time link against libraries that do have exceptions and then you need to be able to unwind stack frames and so like there are things about the way that that the binary gets produced that need to anticipate the presence of exceptions.
00:48:08.880 --> 00:48:09.360
Right.
00:48:10.880 --> 00:48:13.360
And uh so uh I'm sorry.
00:48:13.519 --> 00:48:19.920
And then number three was just sort of um like the kind of absolutism around finding a way to say yes to new features, right?
00:48:20.000 --> 00:48:26.480
That like that there was that there was never kind of something they didn't want to lose a check mark comparison with any language.
00:48:26.800 --> 00:48:32.320
If you were if you were down there chicking boxes for like well does it have you know does it have pattern matching?
00:48:32.400 --> 00:48:37.199
I don't know let's find a way to put pattern matching in this freaking thing I guess right even though it's this very awkward fit.
00:48:37.360 --> 00:48:45.199
Or does it have you know type classes when I compare to Haskell and and I can well okay so let's spend 10 years freaking out how to get type classes into this thing.
00:48:46.239 --> 00:48:52.079
Then call them by a different name because that's refuse to pretend and pretend that it's like not borrowed from Haskell incidentally.
00:48:53.440 --> 00:49:14.159
And and if you don't have that by and so let's see the experience like where where type classes come from for the working C programmer was this sort of decade and a half where you'd use templates because like you bought your C book, your C book tells you generic programming is this amazing new paradigm that C enables you get kind of into it, you see that like it does do some nice things for you and so on.
00:49:14.320 --> 00:49:30.079
But now your life is kind of dominated by these bizarre like multi-page error messages that are all in headers you'd never heard of before, referencing identifiers you didn't type, saying, oh I can't unify this type because under under TP is not under under PQ and I don't have a method named foo defined on it.
00:49:30.159 --> 00:49:53.599
And you're like, well yeah of course you don't what does that have to do with what I'm trying to do and type class is like at least let you give a name to whatever the presence of a method named foo on a type is and let the error message be hopefully a little compressed and a little more human readable, which turns out to be important and turns out to be like something that you need to give users control over because they're gonna have their own levels of abstraction and then make their own mess of like crazy, crazy template header files.
00:49:53.920 --> 00:50:05.519
And I will say like a weird thing about and and maybe this is just like a customization to this when I was when I started my program career as a like you know making money in 2000, I went to VMware and basically wrote assembly and C.
00:50:05.599 --> 00:50:15.519
Like VMware was a C pro was a C product um there were some parts of C eventually at user level but we understood C really well and we wanted to do a good job writing a big C program.
00:50:15.599 --> 00:50:19.760
And there's a little bit of assembly at the margins because you're doing stuff on the machine but we tried to keep that as small as possible.
00:50:19.920 --> 00:50:21.920
And so C was kind of my workaday thing.
00:50:22.079 --> 00:50:28.079
And in 2004 when my initial option grant vested out at VMware I interviewed around a little bit like you do.
00:50:28.639 --> 00:50:33.679
And one of the places I interviewed was a promising search startup by the name of Google.
00:50:33.920 --> 00:50:37.199
And they made it really clear that they were a C shop.
00:50:37.280 --> 00:50:42.480
And I was like cool I'm the you know this sort of talking to the recruiter I'm like great I get it y'all are into C.
00:50:43.360 --> 00:50:53.760
I could if you hump a few bars I'll fake it or whatever, but I don't know all the methods on STD string and I don't know all the you know like whether it's size or length on the standard containers and blah blah blah it's size it turns out.
00:50:54.880 --> 00:50:56.000
Are we cool here?
00:50:56.159 --> 00:51:08.000
And and sort of the day went pretty well for about four or five interviews and there's one guy who's like the only thing he wrote on the board was like implement this function it was like you know reverse or Zapoundrum or something on const std string ref.
00:51:08.639 --> 00:51:15.119
And I was like well okay let me say you know I I'm gonna be faking the methods of of string.
00:51:15.199 --> 00:51:16.719
I I understand how you implement a string.
00:51:16.880 --> 00:51:19.360
I understand like that it's not terminated.
00:51:19.440 --> 00:51:28.000
I understand it's a little bit different than like the C strings I'm used to and I'm gonna move forward with this but like please have a sense of humor about like if I'm getting something wrong here somewhere.
00:51:28.239 --> 00:51:30.960
And he totally did not like I don't understand this.
00:51:31.039 --> 00:51:35.360
You're a computer programmer this is STD string why is this not like what you eat for breakfast?
00:51:35.519 --> 00:51:48.159
And it left this like awful taste in my mouth I have to admit like for a couple of years after that I just had my big a big chip on my shoulder about like this stupid freaking language and like it's huge standard library and all of the surface area it seems to occupy people's minds.
00:51:48.400 --> 00:51:57.599
And I I gotta admit it was really like C11, which is like when you got closures, um when you got auto or thereabouts you got auto.
00:51:57.679 --> 00:52:00.960
So you're no longer having to type out all these crazy verbose types everywhere.
00:52:01.679 --> 00:52:11.760
And a couple of other like little things that came with C11 where I I had this weird feeling where it was like it was as if you built a bridge from Spain to Boston by like putting garbage into the Atlantic ocean.
00:52:14.559 --> 00:52:19.119
But they finally added enough stuff to C that I'd much rather use it than C.
00:52:20.320 --> 00:52:21.840
Yeah no I I agree.
00:52:22.000 --> 00:52:46.800
I think that and especially uh C11 was really the the the moment where I think they brought it all together and C became but I mean I would I would argue that all the old school C the one I was writing in early 2000s is basically dead like nobody writes new anymore nobody likes or uses virtual methods.
00:52:47.199 --> 00:52:54.800
So it's it's not so much that they added enough features that it's now enough to be better than C.
00:52:54.960 --> 00:53:22.400
I think what happened is that they added a bunch of shit that nobody wanted and then people has basically forgotten about all that and they added new stuff that people actually wanted because it helps them reason about you know the the memory and whatnot and Rust came around around too and and so I think C11 was really the the the moment where C took a new a new dimension.
00:53:22.639 --> 00:54:05.519
But there was dark ages I would say early 2000s writing C plus plus in early 2000s was was um interesting so especially I was I was in a company back then where they embraced the whole oh oh until the end until the you know everything is is an object everything is virtual um encapsulation mixed with yes mixed with interface implementation Julia come on let's go yeah yeah it was uh it was painful but uh no I think what's uh I think C I wish there had been a version of C that had not tried to be compatible with C.
00:54:05.840 --> 00:54:32.000
I mean made all these choices that you described earlier and I wish somebody had just built a version of C with generics basically would have been enough for me uh for you know until C11 basically between 1985 and you know 2011 it would have been nice to have something like C with generics and that would have been plenty for me.
00:54:32.159 --> 00:54:47.920
You know and if you're curious by the way what like C codebases in that era looked like right because like the sort of criticism of C at that time from C programs was like oh you're gonna you just have to like write a different tree every time you want to have a tree of strings instead of a tree of integers or something like that.
00:54:48.079 --> 00:55:06.880
No we didn't like come on we're not idiots like there was a C preprocessor but yeah there's a C preprocessor you write a macro that defines a new type and you'd you know you'd say tree of strings tree of whatever but the macro macros are much easier to read debug and they're kind of ugly and they have their own quirks.
00:55:07.039 --> 00:56:15.519
I mean I I get it when it's the crazy error message problem right but so do templates for 20 years you know a cleaner version than C macros is basically what I would have um liked using something that's understood by the compiler can give me good error messages um because that's that was uh and and yeah another problem with C is typically because there was no such uh generics the standard library basically comes with almost nothing right like the the moment you need a data structure that that works with some other kind of type you cannot express it cleanly in the language and so that's that's why basically to a C programmer everything is an array right like and whenever you actually make people take it support I guess yeah exactly it's really funny when when you see people um you know uh do lead code interview on a whiteboard you see how they think how the language shapes how they think about things because the C programmer sees arrays everywhere right they just write arrays and why would they need anything other than an array?
00:56:15.599 --> 00:56:28.880
And even when they do need a hash table they tend to re-encode their own hash tables so they will uh you know for years this was actually an interview question I'd ask people is like just build me a hash table and as a C programmer that's like a thing you do sometimes.
00:56:28.960 --> 00:56:31.360
You know you've got to roll your own hash tables and so you do it sometimes.
00:56:31.519 --> 00:56:33.840
And asking somebody that in Python is just really weird.
00:56:33.920 --> 00:56:34.480
Do you know what I mean?
00:56:34.639 --> 00:56:41.920
Like they're just kind of like okay I guess I guess there's a thing that's an array and you project a hash onto it or something but I don't know why I'm doing this.
00:56:42.559 --> 00:56:42.880
Yeah.
00:56:43.199 --> 00:57:06.559
I actually did that with uh with quite a few um younger uh people who I were interviewing who were out of very prestigious Ivy League schools and some of them just couldn't tell me like you know they they had been fed Python for years at school and I was like okay well what is a dictionary how does it work you know and like like I guess I don't know.
00:57:07.119 --> 00:57:18.960
It's just it associates keys you know values to keys and yeah but what happens in the computer when you call this thing it doesn't have association it doesn't yeah it doesn't magically know this thing goes with that thing.
00:57:19.039 --> 00:57:20.480
Like how does that actually work right?
00:57:20.719 --> 00:57:46.079
Yeah exactly there's a uh there's a side project I'm working on that like I'm close enough to dropping something that I feel okay about plugging for a second which is like I feel like I've talked a lot with you know undergrads and and early career technical people these days where after sort of seven minutes of conversation about whatever it is they're doing, I come to feel that from my perspective, they just don't know how computers work, that these machines are just totally mysterious to them.
00:57:46.400 --> 00:58:01.599
And even though like their job requires them to sort of squeeze more blood out of this pile of GPUs they've got or whatever, that they that the idea of sort of there being an instruction set and there being a clock and there being like a kind of, you know, that this all is physically realized in some way, right?
00:58:01.679 --> 00:58:08.880
That the memory hierarchy is so many levels deep and narrows at this rate and so on, um, is just sort of totally alien.
00:58:09.119 --> 00:58:13.119
There's a time when I was like the systems bigot you were making fun of a couple episodes back, right?
00:58:13.199 --> 00:58:19.360
Where I was the guy who's like, oh like you're my code runs under your code, therefore I know more and more important and more fundamental than you.
00:58:19.519 --> 00:58:20.480
That's false, right?
00:58:20.559 --> 00:58:26.320
Of course like like a lot of the stuff that happens in operating systems is very routine and was solved like 50 years ago or whatever.
00:58:27.440 --> 00:58:44.800
However, it is nice to at least be able to like reason a layer or two below where you're actually operating just to kind of understand idiosyncrasies, understand when the abstractions break, you know, understand why there might be a discontinuity around 64 kilobytes or something when you whatever the number is these days.
00:58:45.760 --> 00:59:09.440
And I think that part has been sort of placed outside of the mental reach of a lot of people and and uh so I'm gonna make a bunch of videos that sort of starts with like relays and switches and tries to sort of get to like if you were on a desert island, you know, how would you make a skill of act how would you make sort of the the Mark I computer with little coconut shells and electrolyte fluid in them that you know lets you compute something with a machine instead of with your hands.
00:59:10.079 --> 00:59:24.159
Anyway, marks the space that's pretty funny looking forward uh to uh to watching this um and yes I guess hardware is another one where I think most for me hardware is really the boundary where I haven't got gotten much deeper.
00:59:24.320 --> 00:59:26.400
I I know very little about hardware.
00:59:26.559 --> 00:59:33.440
So the the deepest I've gotten to has been hardware but then I've always treated hardware as a black box.
00:59:33.679 --> 00:59:44.400
Not entirely like you know I educated myself a little bit on how these things work but um definitely not at at the same level as all the other layers in the stack.
00:59:44.639 --> 00:59:55.840
And I feel like if you manage if you understand those layers of abstraction you just better and you just understand better what the computer does.
00:59:56.079 --> 01:00:13.199
I think there's also some truth to to that if you're going to do maths and proofs, I really highly recommend to do formal proofs like take one of those uh proof assistants and actually write down the proof formally entirely with your little fingers.
01:00:13.519 --> 01:00:29.280
Because even if you read a white paper and you think you understand the proof for me it has always been an eye-opening um moment to take one of those proofs and re-encode them from scratch in one of those systems because your level of understanding is just not the same.
01:00:29.519 --> 01:00:47.280
Once you've you've you you had to go all the way down to the the little terms, even proving the most basic things that you needed, I feel like your level of understanding is very different than if you did it from you know from uh a hundred feet above the ground.
01:00:47.920 --> 01:00:54.079
So that's And by the way I don't claim to sort of understand electronics or physics, right?
01:00:54.159 --> 01:00:59.039
Like if you ask me sort of how transistors work, I'm like, I don't know man, they're magical solid state switches and they're really small.
01:00:59.199 --> 01:01:00.800
And I'm glad there's lots of them on a die.
01:01:01.119 --> 01:01:01.679
Thank God.
01:01:01.920 --> 01:01:10.880
But I don't you know I don't claim to sort of get like the subtleties of like when you dope it this way or that way and blah blah blah and is it this flavor of of fat or that flavor of FET?
01:01:11.039 --> 01:01:11.280
Cool.
01:01:11.360 --> 01:01:13.519
Like there are real professionals that handle that stuff.
01:01:13.920 --> 01:01:35.039
And that's on the other side of the abstract what what's really interesting about our job is the job of programming I think a lot of it is knowing when to use an abstraction and knowing when to break the abstraction that is it is frankly 80% of the job maybe not 80% but the it's it's a big heart of the job.
01:01:35.119 --> 01:01:43.199
And I think that might be for the C community let's get back to them because I feel like we we haven't we haven't badmouthed them enough.
01:01:43.519 --> 01:01:49.920
My B for the C community is that they have done a little bit of false advertisement.
01:01:50.079 --> 01:02:04.079
So let me tell you a story of what where C typically becomes toxic right so you have a company this company is writing C and you so you are naive about it.
01:02:04.159 --> 01:02:14.159
Like you you you have learnt about C and you think C is like any other language right you think it's going to be like Java or C sharp or whatnot excuse me.
01:02:14.960 --> 01:02:53.360
But it's not it's a very very different beast and it's a black hole of complex complexity which means that if you think you're going to just write one library in C you're not once once the C cancer entered your your company then that's it it will spread everywhere you'll have C everywhere and you had a company with five people or ten people they started using C you come back one year later they're no longer talking about the problems of your company or what kind of software they should write no no no these are C people talking about C problems.
01:02:53.599 --> 01:03:05.840
That's basically what you're gonna get and that's fine in in a sense if if you know if you think C is the right way of solving the problems you have then maybe you're willing to pay that cost.
01:03:06.000 --> 01:03:08.639
But I think there's some kind of false advertisement there.
01:03:08.800 --> 01:03:23.840
But I think the moment where C becomes really bad is the moment people join your company don't understand don't know C and start using it for like something it isn't.
01:03:24.159 --> 01:03:39.440
So you have a C shop, you have people who come from the Java world and they say oh it's exactly like the kind of thing I write in Java then there's some library here that implements blah blah blah all new blah blah blah factory.
01:03:39.519 --> 01:03:45.920
Let's go new exactly new factory this and there's a method oh it looks like it's doing what I want it to do.
01:03:46.159 --> 01:03:50.320
And those people what they're doing is completely reasonable.
01:03:50.480 --> 01:04:04.719
They're taking assumptions that they had from another language which is Java and in Java that's how you're supposed to do things because in Java an object and all the abstractions that Java provides to you are actually abstractions.
01:04:04.960 --> 01:04:10.159
So what they want is they don't want you to think about how this thing is implemented.
01:04:10.320 --> 01:04:16.800
They just want you to think about the API right in C are no abstractions.
01:04:17.039 --> 01:04:20.400
There are no abstractions it's just macros.
01:04:20.880 --> 01:04:29.519
These things are there to save you some typing that is it but it it does not mean you shouldn't care about how it's implemented under the hood.
01:04:29.679 --> 01:04:32.480
You should know exactly what it does.
01:04:32.880 --> 01:04:36.800
And if you don't if you handl a pointer God knows what happens with that pointer.
01:04:36.880 --> 01:04:41.440
And if you don't know exactly what the contract is yeah yeah you just wrote a bug.
01:04:41.679 --> 01:04:57.760
Yeah and I think that's a and so I and so I feel like this is the hardest thing in C is that it's really unique to this culture where they present things that look like abstractions but are not.
01:04:58.079 --> 01:05:08.960
And to anybody who comes from any other culture whatever programming language you come from somebody gives you an API you don't have to think about the implementation you just have to use this API.
01:05:09.119 --> 01:05:10.320
That is not C.
01:05:10.960 --> 01:05:12.639
Yeah you better not do that.
01:05:12.800 --> 01:05:23.039
So that's why I call it false advertisement in the sense that it comes with those things these things that look up look they look like abstractions but they're not and they're gonna bite you.
01:05:23.599 --> 01:05:29.280
So end of round no no wonderful no I like I I would I'll go to this TED talk.
01:05:29.360 --> 01:05:41.840
And I think like one of the questions I asked you when we were in person together in San Francisco that I've come up with a slightly different answer to that I want to run by you because it is you know because we're C adjacent here is like, okay if we've got all these genies writing the code for us now, right?
01:05:41.920 --> 01:05:49.519
We've got we've got I I don't have to sit there in Vim and figure out exactly how to stroke C to make it generate the right the right semantics.
01:05:49.679 --> 01:05:55.280
I can describe those semantics and then have it be lowered by some you know intelligence that's not my own.
01:05:55.599 --> 01:06:07.440
Why are we doing all that lowering to something like JavaScript and Python where it's so much harder to get A machine efficient implementation instead of lowering it to C, which is sort of natively a little bit easier to map onto the hardware in the limit.
01:06:08.400 --> 01:06:11.679
I think it's more people disagree with me, but like what's it?
01:06:11.760 --> 01:06:13.039
It has to do with data.
01:06:13.360 --> 01:06:14.960
I think it's just we have multiple.
01:06:15.679 --> 01:06:16.960
This is one thing we came up with that time.
01:06:17.119 --> 01:06:18.000
Was first of all, there's more.
01:06:18.079 --> 01:06:21.280
If you if you troll GitHub, there's a ton more Python, a ton more JavaScript.
01:06:21.519 --> 01:06:23.760
More of it's getting written every day because of coding agents too.
01:06:23.840 --> 01:06:25.440
So it's sort of starting to be a flywheel.
01:06:25.599 --> 01:06:33.039
But even putting that to the side, because my because my experience of like the genies writing C in the small is that they're good at it now.
01:06:33.199 --> 01:06:46.800
Like from Opus 4.5 forward, if I ask it to write me like a lock-free hash table in C that does X, Y, and Z and uses tombstones this way and handles the ABA problem that way and blah, blah, blah, it'll spit it out, lickety split, and the thing will work.
01:06:46.880 --> 01:06:50.159
And it'll sort of explain all the trade-offs and all the like it's it's real, you know.
01:06:50.239 --> 01:06:59.360
This is, and that would have been, you know, kind of staff engineer type of work or whatever in the heyday of C before that sort of got a little more commoditized.
01:06:59.760 --> 01:07:07.039
What it struggles with, and I think to some extent, what humans struggle with too, learning or adapting to building new things is a different.
01:07:07.199 --> 01:07:09.920
So I learned that the that the genies are great at C.
01:07:10.880 --> 01:07:18.559
And so I thought, oh, what if I like rescue HHVM from its bizarre shackles as Facebook's house language?
01:07:18.639 --> 01:07:31.280
What if I take HHVM, it's a project I know really well, um, and I wanted to like work as well on Mac OS as Python does and work as well on Mac OS as as Node does or something, and just make it like a full-fledged member of the open source language ecosystem.
01:07:31.440 --> 01:07:36.960
Because they kind of turned off, you know, it was it wasn't worth people's time at inside of Meta to do macOS support, for instance.
01:07:37.039 --> 01:07:38.960
So I stopped doing that and and so on.
01:07:39.519 --> 01:07:46.559
And uh so I sat so I kind of sat up a little, you know, my genies and set up a little harness and made a loop, as we call it these days, and so on.
01:07:46.719 --> 01:07:48.639
I was like, let's go, let's figure this out.
01:07:48.800 --> 01:07:50.880
And I spent a bunch of money and a bunch of time.
01:07:51.360 --> 01:07:56.559
And it basically was like stuck every single time going through the loop of being like, oh, that didn't work.
01:07:56.800 --> 01:08:03.039
Let me blow away the CMake directory, compile for three hours and get to the problem again, and then oh, that didn't work.
01:08:03.199 --> 01:08:08.719
And so it was actually just that the that the iteration loop went around compile this big ass program.
01:08:09.199 --> 01:08:13.599
Whereas the iteration loop for Python and JavaScript and PHP and blah, blah, blah, doesn't.
01:08:13.760 --> 01:08:27.119
And I think like that's one of the reasons that people learn these languages a little bit faster, too, is that like if I have to, you know, learn C by getting the Unity engine or something and having my machine chew for five hours, and then like I change one thing and it chews for five more hours.
01:08:27.359 --> 01:08:33.279
Um and and there's all these tricks that like working C programmers have, by the way, to like get compile time down, right?
01:08:33.359 --> 01:08:41.279
There's like ways to kind of torture the way that what you put in a header file, what you put in a C file, and so on to make it so that you're less likely to fall into this problem.
01:08:41.359 --> 01:08:43.119
But in an existing project, you don't get to change that.
01:08:43.199 --> 01:08:45.359
And it seems like the tools are still a little naive of it.
01:08:45.520 --> 01:08:51.520
And I'm curious whether this part is actually sort of there's a there's a an element of mystery in our whole discussion here, right?
01:08:51.600 --> 01:08:55.680
Which is that like we're talking about these extremely popular languages that we view as bad.
01:08:55.920 --> 01:08:57.359
And like I don't think we're wrong.
01:08:57.439 --> 01:09:03.920
Like I think that the nits we're picking here are real problems that lead to bugs and lead to people who struggle with using the language correctly and everything.
01:09:04.079 --> 01:09:09.600
And yet those ones are succeeding, and you know, uh, OCaml or whatever your favorite one isn't, Lisp.
01:09:10.079 --> 01:09:17.760
Um, and I'm curious whether a chunk of it is like just some of this, some of these weird ergonomic things that like I can get the iteration cycle going faster.
01:09:17.920 --> 01:09:20.319
And I have more examples available to me.
01:09:20.560 --> 01:09:27.920
Even before the uh, you know, the sort of data, you know, this all became sort of reinforcement learning problems that are fed by GitHub.
01:09:28.159 --> 01:09:34.000
Even before that, I had more people around me who could help me learn JavaScript and help me learn Python.
01:09:34.159 --> 01:09:44.000
Um, there was this on-ramp to programming for a lot of people in the kind of 90s and aughts, which was like looking at the JavaScript from other people's websites and sort of trying to understand what the heck it was doing.
01:09:44.079 --> 01:09:48.319
And you know, for a lot of people, that led to a real career writing its own JavaScript.
01:09:49.039 --> 01:09:53.439
No, I I think there's there's some truth to that, but I I think it's not the end of the story.
01:09:53.600 --> 01:09:58.000
We are just at the very beginning of this story of what LLMs can do.
01:09:58.079 --> 01:10:05.760
And I think invariably we are going to get to the point where we will build custom tools and custom languages for LLMs.
01:10:06.000 --> 01:10:07.680
I think we will get to that.
01:10:07.920 --> 01:10:10.079
When is it now the time?
01:10:10.239 --> 01:10:11.359
I don't think so.
01:10:11.680 --> 01:10:21.039
And if we get to that point, I don't think teaching an LLM all the idiosyncrasies of C ⁇ is is the way to go, right?
01:10:21.359 --> 01:10:29.520
Like C was built the way it was built because I mean a lot of it is backwards compatibility, right?
01:10:29.600 --> 01:10:31.520
And the history of the language.
01:10:31.840 --> 01:10:40.000
And now with LLMs and the the speed at which they can produce new code, and I think that's uh that's a very important metric anymore.
01:10:40.159 --> 01:10:49.520
Like, you know, it it used to be one of those things where you know, whenever you wanted to use a certain programming language, um, people would come to you and say, oh no, don't do that.
01:10:50.079 --> 01:10:53.680
Your ecosystem is not mature enough, we won't have all the libraries we need.
01:10:53.840 --> 01:11:01.119
I feel like today, after what happened with a zig to Rustry writes, I think these days are over, right?
01:11:01.359 --> 01:11:18.239
Like you want to say, you know, you can take any pro any any library you want, you you you pay enough tokens and you let an LLM loosen it, and you can say, all right, take this JVM library and port it to Go or whatever, whatever it is that that you're using.
01:11:18.560 --> 01:11:26.000
So it's I'm exaggerating, obviously, because I'm sure you know it takes time to test and and it also costs tokens.
01:11:26.159 --> 01:11:27.279
But you get what I mean.
01:11:27.359 --> 01:11:34.000
I think that the whole there is an ecosystem argument has been weakened a lot.
01:11:34.560 --> 01:11:40.479
And since that argument has been weakened, that was the main argument against new languages, right?
01:11:40.640 --> 01:11:47.439
So there's obviously the cognitive burden from going from one language to the next, but I think LLMs are not going to care much about that.
01:11:47.600 --> 01:11:53.119
Um, and the other big argument was, well, you don't have the libraries, you don't have the ecosystem.
01:11:53.520 --> 01:12:04.239
But once LLMs can really translate pretty much anything like that, yeah, there will be no need for them to use JavaScript.
01:12:04.319 --> 01:12:06.560
They can use whatever they want internally.
01:12:06.720 --> 01:12:26.720
And I would be so I think if I had to bet they're probably already using in the deeper layers something that looks like, you know, what the language of the future will look like for them, meaning they they are probably abstracting away a lot of the because that's what LLMs always do, right?
01:12:26.800 --> 01:12:35.600
Like the tokens go in, and then in the inside the deeper layers they have a representation that has nothing to do with with the actual syntax.
01:12:35.680 --> 01:12:38.079
So I I my guess is they're already doing that.
01:12:38.159 --> 01:12:50.399
When you're feeding them Python or JavaScript or Ruby or whatever it is, or C, it's after a few layers, this all gets boiled down to similar symbols and similar representations.
01:12:50.960 --> 01:12:57.920
I was talking with a founder that are that have to pre-train their own foundation model for reasons uh last week or week before last.
01:12:58.079 --> 01:13:02.960
And it hit me to this paper that really surprised me, which uh describes pre-pre-training.
01:13:03.199 --> 01:13:07.199
And this was uh the you know, sort of imagine the usual LLM regime.
01:13:07.359 --> 01:13:11.680
You've got some fine-tuning task downstream you'd like to learn, maybe it's coding, maybe it's something else.
01:13:11.920 --> 01:13:22.800
There's going to be this pre-training step where you just kind of like boil the ocean and read everything on the web and somehow learn from you know every Redditor in the world the structure of languages and eventually formal languages and so on.
01:13:22.960 --> 01:13:30.560
And this pre-pre-training step that they propose was actually a bunch of artificial grammars that they kind of randomly generated and then fuzzed.
01:13:30.640 --> 01:13:35.119
So it might be a grammar that's like parentheses and curly braces that are matched, right?
01:13:35.279 --> 01:13:40.239
Where it's like you don't stagger them or whatever, and there's as many lefts as rights and they they nest.
01:13:40.479 --> 01:13:42.640
Or it might be, you know, other similar languages like that.
01:13:42.720 --> 01:13:47.039
And you pre-pre-train on this artificial subset and it makes the rest of it go faster.
01:13:47.199 --> 01:13:58.800
Like somehow having the ability to kind of do some simple, you know, okay, I can match the left to the right, um, or predict that a right's coming up because like there's there's a left that I haven't closed yet.
01:13:59.039 --> 01:14:04.319
Somehow the ability to do that like helps you learn the rest of both natural and artificial languages better.
01:14:04.399 --> 01:14:06.079
I think that's really, really interesting.
01:14:06.239 --> 01:14:14.079
Um, and it seems like this pragmatically holds outside of the context of that paper, which is like one of the things you have to take with a grain of salt with deep learning research for the last 10 years.
01:14:14.159 --> 01:14:17.039
It's like sometimes people do it and you just try it and can't make it work.
01:14:17.199 --> 01:14:19.760
Um, it seems like this actually works, and that's really fascinating.
01:14:20.000 --> 01:14:29.760
And then uh another another idea from this that I think I got, I went down a rabbit hole earlier this week around how LLMs utilize syntax in natural languages, right?
01:14:29.920 --> 01:14:34.640
Because I had been basically I just like hit a breaking point with like reading AI slot text.
01:14:34.720 --> 01:14:37.520
Like someone gave me another long ass email that I had to read.
01:14:37.600 --> 01:14:39.760
And it was like definitely clawed output.
01:14:40.000 --> 01:14:41.920
And and I was like, well, what am I doing?
01:14:42.000 --> 01:14:43.119
Like, why do I know this?
01:14:43.359 --> 01:14:47.760
Like, why is it so manifestly obvious to me that this is was not written by a human being?
01:14:47.920 --> 01:14:55.439
Why do I resent having to pay attention to every character in this thing when I know that the person who sent it to me didn't generate every character with that much energy?
01:14:55.760 --> 01:15:02.560
Um and and one of the things that I came up with was like, well, it seems like it overuses certain constructions for sure, right?
01:15:02.640 --> 01:15:03.840
Like we all joke about this.
01:15:04.000 --> 01:15:05.680
It's not X, it's Y, right?
01:15:05.840 --> 01:15:09.279
Like it's, you know, it's you know that's load bearing and blah, blah, blah.
01:15:09.359 --> 01:15:09.439
Right.
01:15:09.520 --> 01:15:12.319
There's there's certainly like little key phrases and m-dashes and everything else.
01:15:12.399 --> 01:15:19.600
But there's also like medium scale stuff, like the sort of shape of sentences and the shape of paragraphs and the whole kind of flavor of some arguments that it writes and things like that.
01:15:19.760 --> 01:15:23.199
And I kind of dug into this a little bit, and this has been something that people have looked into.
01:15:23.279 --> 01:15:40.560
Uh, there's like a good 2024 paper that's kind of the butt the trailhead of this, that is um so there's no uh a priori, just because you made a model that generates a distribution of the next token and you sample from it greedily, you don't necessarily get natural sounding text from that.
01:15:40.640 --> 01:15:40.720
Right?
01:15:40.800 --> 01:15:47.199
If you do a completely naive thing and you and you do it with a state-of-the-art amazing model, like do huge model, tons of data.
01:15:47.359 --> 01:15:54.159
Um if you just mindlessly go through and like greedily sample, it'll end up in some loop of tokens that are locally plausible.
01:15:54.239 --> 01:16:01.680
It'll say, like, I don't know, period, I don't know, period, I don't know, period, all of which like are high probability next to each other, but generates an incoherent text.
01:16:01.840 --> 01:16:06.000
But it's trying to generate, you know, one token at a time, and that's what it's doing, right?
01:16:06.560 --> 01:16:11.439
That same sampling methodology kind of has fingerprints at larger scale, right?
01:16:11.600 --> 01:16:20.800
So there's nothing kind of in the in the hacked up sampling stuff that people do to make the output of these things better that necessarily makes it sound more naturalistic in this way.
01:16:20.960 --> 01:16:23.199
And by the way, it's not even clear that like you want that.
01:16:23.359 --> 01:16:25.119
Um, it turns out post-training makes this worse.
01:16:25.279 --> 01:16:31.119
Like when we when we instruction fine-tune these things, they get even more stuck into their ruts of the little grammar that they like.
01:16:31.279 --> 01:16:44.960
And it's because like we want them to have clear answers and we want them to sort of explain things in certain ways, and we don't want them to use all these funky constructions that like some rando on Reddit writing run-on sentences and blah, blah, blah, is gonna use and you're gonna find hard to follow if you try to read it, right?
01:16:45.199 --> 01:16:47.680
So there's a reason that it hits this narrow manifold.
01:16:47.760 --> 01:16:56.479
And imagine all this in the context of programming languages because a lot of what I want from as a human, of a lot of what I want for programming languages, some sense of expressivity, right?
01:16:56.640 --> 01:17:03.600
Like the reason you're pounding the table for O'Camel and not these Lisp dialects is just like, I don't know, man, this parentheses kind of get me down.
01:17:03.680 --> 01:17:04.960
Like I don't want, I don't want to be a tree.
01:17:05.039 --> 01:17:12.640
I want to be able to say A plus B, not you know, plus A B and like mentally undo all the kind of treefication when I try to read this code.
01:17:12.720 --> 01:17:15.600
I want it to like look like notation I might write in a notebook.
01:17:15.840 --> 01:17:21.600
Um and it might be that like that requirement is is almost reversed for these guys.
01:17:21.680 --> 01:17:29.680
That like, even though they can parse all of C and understand all of C, I'd expect that the C they generate, and when I review the C they generate, this seems to be true.
01:17:29.840 --> 01:17:32.479
I'd expect the C they generate to like look a lot like Java.
01:17:32.560 --> 01:17:33.119
Do you know what I mean?
01:17:33.199 --> 01:17:42.000
To be kind of flat and kind of iffy and you know, not using a lot of the richness of the language, not doing a lot of the sort of compressions I would sort of mindlessly do or ask for in a review.
01:17:42.159 --> 01:17:43.199
Um, and it's fine.
01:17:43.359 --> 01:17:47.039
Like there's a lot of code in the world that's not exactly the way I would write it, and it's not really a problem.
01:17:47.199 --> 01:17:50.720
And so I don't I don't bother trying to reinforce learn it negatively.
01:17:50.800 --> 01:17:55.680
Like the freaking code works, it's fine, it's readable, but it's doing that same thing that it does in English.
01:17:55.760 --> 01:18:00.399
It's like hitting this little tiny narrow manifold in the bigger space of C programs.
01:18:00.640 --> 01:18:07.199
And so, you know, do they even expressivity might be a bug, not a feature, basically, for for these authors, is my thought.
01:18:07.359 --> 01:18:16.159
It's like maybe they really do just want to spit out Lisp or whatever and not be distracted by is this a switch or an if, or is this you know a while or a for or whatever?
01:18:16.479 --> 01:18:19.359
No, and and we agree on this totally, on both things, right?
01:18:19.439 --> 01:18:27.279
Like I agree with you when you were saying earlier that I think one of the keys for LLMs is going to be the latency of the tooling.
01:18:27.439 --> 01:18:29.279
I've said that before, I said it again.
01:18:29.600 --> 01:18:35.439
Very biased about that because you know I like reactive programming and I think that's gonna play a key role.
01:18:35.680 --> 01:18:54.640
But yeah, if we want um LLMs or AIs that are not necessarily LLMs to pick up uh programming at a larger scale, they will need to be trained, and to be trained, they will need to get feedback, and that feedback, the slower that feedback is, the more difficult it's going to be.
01:18:54.880 --> 01:19:06.880
And then yeah, when it comes to yeah, the the kind of code that they are that they are producing Yeah, I agree with you that there's some there are some similarities with text, right?
01:19:06.960 --> 01:19:24.239
So if I were to, you know, try to if I was to to try to summarize what you were saying, so basically what you're saying is when an LLM generates text, you can immediately tell because it it tends to um f uh fall into always the same traps, into the same thing.
01:19:24.479 --> 01:19:27.920
It keeps making the same tree shapes, yeah, the same part of the shapes that it likes.
01:19:28.319 --> 01:19:28.479
Right.
01:19:29.119 --> 01:19:43.680
And there's probably something similar going on with with programming where when you look at the code, it has certain structure structuring that it likes that doesn't feel natural to somebody who is proficient in the language.
01:19:43.840 --> 01:19:45.600
I think there's some truth to that.
01:19:45.840 --> 01:19:53.199
Um but uh it's uh I mean an LLM also for me, there's another dimension that's missing there.
01:19:53.520 --> 01:20:00.079
For me, I get excellent results when you let them choose pretty much everything, right?
01:20:00.159 --> 01:20:07.760
So you tell them, hey, I want you to build me this, and you're really not specific about the thing that you want.
01:20:08.079 --> 01:20:11.600
And then when you do that, it will produce an amazing result.
01:20:11.680 --> 01:20:13.439
It can get you a video game, right?
01:20:13.680 --> 01:20:25.199
You can say, hey, build me a video game of somebody snowboarding on you know a hill made of trees or whatever, whatever, you know, whatever rocks your boat.
01:20:25.359 --> 01:20:28.399
And it will come up with something that is really impressive.
01:20:28.640 --> 01:20:33.439
And the moment where things fall apart is when you're trying to be more specific, right?
01:20:33.600 --> 01:20:50.479
And when you say, Oh yeah, yeah, so I want to snowboarding, but I want, you know, I want it to be this kind of colour and this kind of shape and this kind of vibe and this kind of and then you start with the iteration where and that kind of that kind of fits what what you're saying, right?
01:20:50.560 --> 01:21:07.279
Like where they have a certain path, the RLM has a certain path that it wants to it it always wants to go back to, whether it's the shape of the text, whether it's you know, a certain kind of features that it likes or the the the the way it likes to to build something.
01:21:07.520 --> 01:21:12.319
And as long as you you go in that direction, everything goes smoothly.
01:21:12.880 --> 01:21:22.960
But the moment you try to say, well, no, I don't want you to write things this way, I want you to do them this other way, um yeah, things things fall apart.
01:21:23.119 --> 01:21:29.039
So I think the solution will probably we'll probably have to train things differently.
01:21:29.439 --> 01:21:54.399
If we want those those um patterns to change or you know, to be so yeah, we'll we'll probably have to train the LLMs with um with some kind of uh you know feedback that tells it like this kind of this kind of diff is not idiomatic, that's not the way you you would do things, and then things are probably gonna get a lot better.
01:21:54.960 --> 01:21:55.119
Yeah.
01:21:55.279 --> 01:22:08.079
But constraining the LLM pre and post-training is probably going to be um where the next big leap forward is going to be in in programming, I think.
01:22:08.239 --> 01:22:12.079
Because I feel like those models they're already really smart.
01:22:12.239 --> 01:22:16.000
Like they can do really good stuff, really complex stuff.
01:22:16.560 --> 01:22:21.680
It's just that they yeah, as you said, I would say that the there are a couple of things missing.
01:22:21.840 --> 01:22:29.359
The first one is they tend to do things that are not idiomatic and they tend to over-engineer things.
01:22:29.520 --> 01:22:31.760
So they they lack of taste, I would say.
01:22:31.920 --> 01:22:36.000
They they don't they they don't have they don't make tasteful decisions.
01:22:36.319 --> 01:22:36.560
Right.
01:22:36.880 --> 01:23:13.680
And to make tasteful decisions, I feel like you have to work on much larger systems by yourself and own those decisions, which I think LLMs have never been trained on, and that's going to be where the next big thing is going to be, where you have an LLM, you let it program something that's like one million lines of code, and you let it iterate on that thing for a really long time, and you let it own the bad decisions in the experience of living with a big code base for a decade or whatever, which is sort of at some level where this taste comes from in humans right now.
01:23:14.000 --> 01:23:14.399
Exactly.
01:23:14.560 --> 01:23:19.840
And so it will develop taste once we do that, because that's what taste is.
01:23:20.079 --> 01:23:30.079
Taste is this feeling that if you don't do things this way, or more importantly, if you do things in that particular way, things are gonna go badly.
01:23:30.720 --> 01:23:33.840
You don't exactly know why, you just know that it doesn't smell good.
01:23:34.239 --> 01:23:34.720
Yeah, exactly.
01:23:34.880 --> 01:23:35.119
Exactly.
01:23:35.279 --> 01:23:37.840
It's like I I associate pain with this thing you're doing.
01:23:38.000 --> 01:23:38.159
Yeah.
01:23:39.279 --> 01:23:43.680
Um and that intuition, I think, is gonna take a while to develop.
01:23:44.319 --> 01:23:52.399
And then the way we you and I developed this, by the way, was, or you know, maybe I should speak for myself, but my impression from talking to you is this is true for you too.
01:23:52.640 --> 01:24:06.640
The way we developed this intuition was like a decade or two of working with people who knew more than us, uh, were more accountable for the thing I was working on, um, you know, better paid, scarier, more well-known.
01:24:06.880 --> 01:24:19.600
And like me scraping together a diff all week to show them, and trying to do everything I can behind the scenes to make sure it's going to be right the first time, and showing to them, and then being like, no, this is all garbage.
01:24:19.760 --> 01:24:22.000
I'm tearing this to a million little pieces, go back.
01:24:22.159 --> 01:24:27.840
And you go back to your desk with a million little pieces and you try and put them back together with the feedback and then and so on, right?
01:24:28.239 --> 01:24:31.279
And the sort of emotional, the imprinting, right?
01:24:31.359 --> 01:24:39.680
The sort of emotional weight of like that feedback partly comes from the proof of work of the fact that you're moving those pieces around in friggin' Vim.
01:24:39.840 --> 01:24:40.399
Do you know what I mean?
01:24:40.479 --> 01:24:43.039
Like you're moving those pieces around like character by character.
01:24:43.279 --> 01:24:49.119
It take you, it would take you hours and hours to like go get through that feedback and put it back together and be like, okay, thanks for the feedback.
01:24:49.199 --> 01:24:50.159
Is this what you mean?
01:24:50.399 --> 01:24:53.840
Um, and that's kind of one of the places where that taste seems to be forged.
01:24:53.920 --> 01:25:06.319
And if you've never had that experience before, at least as I've as I know human computer programmers, if you've never had kind of some of some apprenticeship like that somewhere, it's really, really hard to get to that level of taste.
01:25:06.560 --> 01:25:08.560
Um and that's at least part of the.
01:25:54.239 --> 01:25:56.880
I don't think it's going to be enough one or the other.
01:25:57.039 --> 01:25:57.279
Right?
01:25:57.439 --> 01:26:10.479
Like if all what you've done is show your work to other people and and look for approval, that's never going to give you, you know, the kind of intuition you need to build a larger system by yourself.
01:26:10.720 --> 01:26:12.560
You really have to put an institution.
01:26:13.359 --> 01:26:13.760
Yeah.
01:26:14.399 --> 01:26:18.560
This has to happen, and you have to own the decisions that you made.
01:26:18.880 --> 01:26:24.880
And that will be um life lessons that you know you you cannot get anywhere else.
01:26:25.039 --> 01:26:27.119
Uh, but the contrary is true too.
01:26:27.279 --> 01:26:32.000
Like if the only thing you've done is work by yourself and and start really ambitious.
01:26:32.239 --> 01:26:42.560
So, for example, I've worked with programmers that I find really, really talented, like very, very good and and extremely proficient, you know, very fast and everything.
01:26:42.720 --> 01:26:49.920
But who spends you know most of their lives working alone, the kind of code they write is very difficult to follow.
01:26:50.079 --> 01:26:54.880
A typical example is going to be they have their own standard library, right?
01:26:55.039 --> 01:27:15.039
Because then so they built a bunch of primitives, and you know, if this is C, they have their own header file, and in that header file, there's like a million lines of different things and different primitives that they needed at some point where and and they crafted it for themselves because that's what they needed, right?
01:27:15.439 --> 01:27:22.960
And you show up in a code base like that, and you're like, Well, wait a minute, what what is this this three-character thing?
01:27:23.039 --> 01:27:24.479
And what is this type over there?
01:27:24.560 --> 01:27:27.680
And then you're oh, okay, so that's STD, blah blah blah.
01:27:27.760 --> 01:27:28.800
That's STD this.
01:27:29.279 --> 01:27:35.359
I even came across someone one time who reinvented down smart pointers in C.
01:27:36.479 --> 01:27:37.680
And you're looking at this.
01:27:38.800 --> 01:27:43.920
You're like, you're like, is it really like and then you look at the implementation?
01:27:44.239 --> 01:27:45.359
Okay, this is wrong.
01:27:45.439 --> 01:27:51.199
This is wrong, this is a really bad you know, reference game.
01:27:51.520 --> 01:27:52.000
But why?
01:27:52.079 --> 01:27:55.199
I don't use threads, I don't care about all these race conditions.
01:27:55.680 --> 01:28:05.840
Well, you know, sometimes you're gonna end up with a library and maybe the Library is going to spawn a thread and and things are not going to be you know the way you imagine them to be.
01:28:06.079 --> 01:28:12.560
But anyway, so uh I would say to forge uh to forge an intuition, uh you need both.
01:28:12.720 --> 01:28:25.920
You need to go out there, build your own things, but you also need uh feedback, at least if you have the intention of working with other human beings, which might not be the case for a lot of people.
01:28:26.079 --> 01:28:31.760
I was gonna say, like, this is one of the questions I want to ask you too, is like we're we're both talking like multiple decades into a software career.
01:28:31.920 --> 01:28:38.960
Do you think in 2050, let's say, um, this kind of human taste is still a factor?
01:28:39.199 --> 01:28:42.479
Like, is is this kind of human judgment still a thing?
01:28:42.720 --> 01:28:45.359
Or is this fable 28 handling it for you?
01:28:45.680 --> 01:28:46.800
I'm extremely biased.
01:28:46.880 --> 01:28:58.960
You know, I feel like you know, the people in the 80s who are telling you, oh, you need to know how to program things with four kilobytes of memory and all these kids with all that rum, you know, that that's not real programming.
01:28:59.039 --> 01:29:06.640
You know, I I feel a little bit like that when I say that, but you know, I feel that I I don't think LMs are the answer.
01:29:06.720 --> 01:29:09.760
So, all right, super unpopular take.
01:29:09.920 --> 01:29:14.479
Uh, everybody who I know who lives in San Francisco is gonna tell me I'm crazy.
01:29:14.720 --> 01:29:16.960
To me, AGI is not around the corner.
01:29:17.119 --> 01:29:21.760
I don't think LLMs are going to replace programmers anytime soon.
01:29:22.000 --> 01:29:24.239
Um, I think they're super useful.
01:29:24.560 --> 01:29:30.479
I think they are going to do amazing things, but what I see so far is they cannot be trusted.
01:29:30.560 --> 01:29:35.520
That's basically what what you know uh what I took what my take on all this is.
01:29:35.760 --> 01:29:45.600
And since you cannot trust them, that means that you will need human beings you can trust, and that the job, at least for now, is going to stay.
01:29:46.079 --> 01:29:50.560
Um, I think for the best, we'll be able to use LLMs to do all sorts of things.
01:29:50.800 --> 01:29:55.279
All the grunt work, all the the shitty refactoring that nobody wanted to do.
01:29:55.520 --> 01:29:56.560
They can do all that.
01:29:56.640 --> 01:29:59.760
And then the database migrations, yeah.
01:30:00.399 --> 01:30:11.039
Yeah, all those migrations that nobody wanted to do, especially uh except for you know, um managers who wanted to manage an org where people don't do anything useful.
01:30:11.359 --> 01:30:13.119
Uh you know, no names.
01:30:13.520 --> 01:30:15.439
Give me 500 more people, this will go faster.
01:30:15.520 --> 01:30:15.840
Yeah.
01:30:16.079 --> 01:30:16.479
Exactly.
01:30:16.880 --> 01:30:17.840
Nice code base you got there.
01:30:18.000 --> 01:30:20.000
But managers migration lasted any longer.
01:30:20.079 --> 01:30:20.560
Yeah, yeah.
01:30:20.800 --> 01:30:21.199
Exactly.
01:30:21.439 --> 01:30:32.479
Bad managers, they always want uh to be in charge of projects that are completely useless, but somehow recognized as useful by upper management.
01:30:32.560 --> 01:30:34.319
This is this is their sweet spot.
01:30:34.479 --> 01:30:47.680
Because then all these people are doing things that nobody cares about, like a rewrite that wasn't necessary, and everybody is busy, and there's no friction, nobody complains about anything, and then they're they're on top of the world.
01:30:47.840 --> 01:30:52.079
They get to do they they don't get to do any work, basically.
01:30:52.239 --> 01:30:53.760
Uh but with that, yeah.
01:30:54.000 --> 01:31:02.079
So going back on the yeah, I don't know what the what uh coding life will look like in 2050.
01:31:03.279 --> 01:31:14.239
I think that one aspect of the job that's probably gonna die is all the you know the code editing wizardry, right?
01:31:14.399 --> 01:31:21.840
You know, the people in Emacs who can type with fingers like this, yeah, they get their picky too on the Medicate.
01:31:23.920 --> 01:31:33.680
And and it was really funny that you know there was some kind of subculture where people were actually really proud of this stuff, where you know they would do control, blah, blah, blah, blah.
01:31:34.319 --> 01:31:35.279
It's an esport, right?
01:31:35.359 --> 01:31:38.640
It's like I think I think it's like you know, having actions per minute in Warcraft or whatever.
01:31:38.800 --> 01:31:45.039
We're actually like thinking about doing an esports Vim World Championship, uh, you know, at uh just as raw entertainment, right?
01:31:45.119 --> 01:31:46.720
Like get like a sportscaster, right?
01:31:46.960 --> 01:31:47.760
Get flashy lights.
01:31:48.159 --> 01:31:48.560
Cool.
01:31:48.960 --> 01:31:49.199
Cool.
01:31:49.359 --> 01:31:50.399
Anyway, sorry, keep going.
01:31:50.560 --> 01:31:51.039
Yeah.
01:31:51.359 --> 01:31:54.319
No, I mean, and so I think that part of the job is gone.
01:31:54.479 --> 01:32:06.159
I think the part of the job where you know you you really know your code editor really well and you do some kind of dark magic, that that part of the job I think will will go away.
01:32:06.319 --> 01:32:15.840
Um, but then the rest, I think the fundamentals it I think we are going to end up with a bunch of different AIs who are good at different things.
01:32:16.159 --> 01:32:23.359
And um I don't know that we'll get to AGI in the next decade.
01:32:23.520 --> 01:32:24.079
We'll see.
01:32:24.239 --> 01:32:25.359
I've been wrong before.
01:32:25.920 --> 01:32:26.479
Interesting.
01:32:26.720 --> 01:32:27.279
I'm not sure.
01:32:27.359 --> 01:32:31.680
Like, like I think I my take on this is honestly that intelligence doesn't have a technical definition.
01:32:31.840 --> 01:32:33.680
It kind of eludes technical definition.
01:32:33.760 --> 01:32:36.239
It's one of the reasons we can disagree about this this way.
01:32:36.399 --> 01:32:44.399
So I think like a lot of strange stuff is gonna happen, and some people are gonna call it AGI, and some people are gonna beg to differ, but certainly we'll have computers doing a lot of weird stuff.
01:32:44.720 --> 01:32:56.399
Um, I think on the question of like where humans slot in here, I think it's interesting to look at some domains like translators or radiologists, where you know, arguably we've been living with superhuman performance by machines for a while.
01:32:56.560 --> 01:33:03.199
If you narrowly think of the job of a radiologist as classify this malady given this medical imaging, right?
01:33:03.279 --> 01:33:06.479
Like if it's here's a CT scan, why does this guy have a headache?
01:33:06.720 --> 01:33:13.119
We do have machine models that are sort of have better precision and recall than than most human radiologists already, and we have for a while.
01:33:13.359 --> 01:33:18.079
And yet there are more human radiologists uh out there in the market than there ever were.
01:33:18.319 --> 01:33:32.560
And part of it is that, you know, the profession, one of the reasons doctors are a profession and not a job, is that like we don't meter doctors by their piecework because you need to have an ethos to sort of know that somebody's behaving correctly as a doctor, right?
01:33:32.640 --> 01:33:43.199
It's not enough to just sort of be like, yeah, capitalism, you know, the number of films I look at as a radiologist determines how much money I work I make, incentives, blah, blah, blah.
01:33:43.359 --> 01:33:49.680
We need more from people if we're gonna let them cut people open and prescribe drugs and do all the things that society trusts doctors to do.
01:33:49.840 --> 01:33:51.840
And so medicine has an ethos.
01:33:52.079 --> 01:33:54.159
It has this sort of set of professional norms.
01:33:54.319 --> 01:33:55.920
It has people with skin in the game, right?
01:33:56.000 --> 01:34:07.119
It has professionals who basically like take an oath to society formally or informally, saying, like, we're gonna live to this sort of higher standard of conduct than a guy who runs a used car dealership or whatever, right?
01:34:07.199 --> 01:34:10.800
We we're not just sort of supposed to be out here and market giving people what they want.
01:34:10.880 --> 01:34:13.760
We're supposed to be living up to what it is to be a doctor.
01:34:13.920 --> 01:34:21.039
And I think engineers also have an ethos, um, whether you're sort of a licensed engineer or not, civil engineers and things like that in Canada, right?
01:34:21.119 --> 01:34:24.479
They make you do something with the state to be certified as an engineer before you have that.
01:34:24.560 --> 01:34:25.920
It's not just a job title.
01:34:26.079 --> 01:34:29.279
It's also a kind of like more explicit social contract.
01:34:29.520 --> 01:34:36.880
And I think being an engineer is taking that same kind of accountability about whether the artifacts you're responsible for work as designed, right?
01:34:36.960 --> 01:34:40.960
Whether they're whether they're safe, fit to purpose, reasonably efficient.
01:34:41.119 --> 01:34:47.600
Um, probably a few other figures of merit for getting in there too, uh, hopefully that they have a future and things like that.
01:34:47.840 --> 01:34:52.720
And to some extent, people have internalized the emotional reality of that already, right?
01:34:52.880 --> 01:34:57.920
The kind of the crusty gray beard who resigns rather than sort of ship garbage, right?
01:34:58.000 --> 01:35:01.520
Or ship something that they know they're not going to be able to tell whether it works or not.
01:35:01.680 --> 01:35:06.159
Um, doesn't just sit there saying, like, well, okay, this is my job and I cash my paycheck and I do what the boss tells me.
01:35:06.239 --> 01:35:20.880
Like, no, they've got some sense that like it would be wrong actually to earn a living by selling software to people that doesn't do what it's supposed to do or appears to, but then breaks, or, you know, doesn't have a, you know, is making promises that it can't really deliver on in one way or another.
01:35:21.039 --> 01:35:26.720
And I think the that thing that you said about trust, like I was talking yesterday with a friend Josh Wills.
01:35:26.880 --> 01:35:28.319
Uh, you know, Josh, I think, right?
01:35:28.399 --> 01:35:29.600
Like, uh, maybe not.
01:35:29.920 --> 01:35:32.880
So like uh was uh was head of data at Slack.
01:35:32.960 --> 01:35:38.960
We shipped a bunch of software together when I was at Slack, is now at Datology, and we're talking about his job at Datology.
01:35:39.359 --> 01:35:49.119
Um I think that there's something here that is that requires basically skin in the game that basically requires that like I am an actual human carcass that you can imprison, right?
01:35:49.199 --> 01:35:54.239
Like that you can you can take my wealth and you can take my family and take my ability to spend my time freely.
01:35:54.399 --> 01:35:55.920
And I don't want that to happen.
01:35:56.159 --> 01:36:07.760
And so since that's grounded in this, like I'm, you know, I'm going to make sure, even if it's an LLM writing the code, I'm gonna try and kind of pour over it later to make sure, like, hey, are we running an honest business here?
01:36:07.920 --> 01:36:10.479
Like, are we actually, you know, does this thing do what it's supposed to do?
01:36:10.720 --> 01:36:22.560
And and if I fail to do that, or if you replace me with an LLM that sort of doesn't have skin in the game in that way, it's not clear that the actual that the sort of latent trust contract of somebody buying this piece of software is gonna actually work.
01:36:22.720 --> 01:36:25.840
And one of the examples he used, like his day job is data pipelines.
01:36:25.920 --> 01:36:31.520
He's like helping customers build, you know, pre-training data sets and things like that for their LLMs that they're building.
01:36:31.840 --> 01:36:39.119
And um for data pipelines, you've kind of like got this intrinsic what's eager and what's lazy question, right?
01:36:39.279 --> 01:36:42.319
Like if conceptually, it's some big join, right?
01:36:42.479 --> 01:36:44.880
Conceptually, it's some big hierarchy of joins.
01:36:45.039 --> 01:36:48.960
You've got the static data set, and then I need to go train an LLM with it.
01:36:49.119 --> 01:36:56.399
And which parts should I do, you know, right before I get the data in memory or right after I get the data in memory, which points should I do ahead of time?
01:36:56.720 --> 01:37:04.079
And that basically requires kind of like grounding in the real world, like requires kind of knowing how big disks are, knowing what CPUs are good at and bad at.
01:37:04.319 --> 01:37:09.840
All that stuff changes over time, all that stuff is sort of contingent on what the customer considers acceptable.
01:37:10.000 --> 01:37:15.279
Um, and so there's like a kind of a grounding in the actual economy there that's hard to just whistle past.
01:37:15.439 --> 01:37:22.079
And I think that part of it is is not it's not obvious to me that machine learning is like the right paradigm to build a machine that does all of that.
01:37:23.439 --> 01:37:24.960
Yeah, I agree with you.
01:37:25.520 --> 01:37:28.239
But I yeah, I we'll see.
01:37:28.479 --> 01:37:42.239
We'll see because what happens when we get to the point where the machine and if we get to the point where the machine just understands the code so much better than us that you know it's it's a bit like chess, right?
01:37:42.319 --> 01:37:50.159
Where what happens to the if if we get to the point where the the level of understanding is just an order of magnitude better, like chess, right?
01:37:50.239 --> 01:37:52.560
Like nobody can even dream of beating.
01:37:53.039 --> 01:37:55.039
Yeah, if it says this weird move is good, it's right.
01:37:55.439 --> 01:37:59.359
Yeah, if it's if it says this weird this move looks funky to me is the right move, it's just correct.
01:37:59.600 --> 01:37:59.840
Yeah.
01:38:00.640 --> 01:38:01.119
Exactly.
01:38:01.359 --> 01:38:38.880
And so we could get to this level with programming too, where you know it says you need to change this function to do that, uh, otherwise it's a security risk, and you don't see it at all, but maybe you've seen it claim make these kind of claims and always be right for the last couple of years, and you're like, yeah, we probably should be making that change because the people attacking us are also LLMs with superhuman intelligence, and if we don't protect our source code, the our code the the way we should, then it's going to come back to bite us.
01:38:38.960 --> 01:38:42.079
So And we're all about to be I mean we we'll see.
01:38:42.319 --> 01:38:42.720
Yeah, yeah.
01:38:43.119 --> 01:38:43.920
I I mean that that part is.
01:38:44.319 --> 01:38:48.479
So now I think for now And then the the back and forth goes like, well, but it must be explainable, right?
01:38:48.560 --> 01:38:59.439
Like it must there must be a reason, it must be possible to follow some argument about like what the security hole is, but then maybe that that argument takes a human being more than a lifetime to understand or something, or is sort of so long and so complex that yeah.
01:38:59.760 --> 01:39:01.439
Or it could be just so complex.
01:39:01.520 --> 01:39:10.399
Like I think they're still trying to decipher the latest proof that they have on an LLM proved you know some kind of difficult math problem recently.
01:39:10.479 --> 01:39:15.199
I'm not good at math, so I wouldn't be able to tell you exactly what what what this is about.
01:39:15.520 --> 01:39:20.399
But um I think they're still trying to decipher what what the LLM came up with.
01:39:20.479 --> 01:39:33.199
So now imagine 10, 20 years from now, after you know, multiple exponentials of um, you know, optimizations at every level, because we are going to optimize these things at pretty much every level.
01:39:33.279 --> 01:39:38.479
We we already have to all the best human talent and all the best machine talent in the world is laser focused on this now.
01:39:38.640 --> 01:39:41.680
And yeah, it's gonna go faster, if anything, for a long time.
01:39:42.640 --> 01:39:43.199
Exactly.
01:39:43.359 --> 01:39:48.000
So wait ten, twenty years and imagine what this stuff can do.
01:39:48.159 --> 01:39:54.560
But I think the one thing that's not gonna change anytime soon is they cannot be trusted.
01:39:54.880 --> 01:39:55.520
For now.
01:39:55.840 --> 01:39:57.359
For now they cannot be trusted.
01:39:57.840 --> 01:39:58.640
Yeah, interesting.
01:39:59.039 --> 01:40:03.119
And if you're not in a closed world like chess, I think for a long time.
01:40:03.920 --> 01:40:07.039
So thank you so much for spending this time with us here on Computers Coffee and Beer again.
01:40:07.119 --> 01:40:12.960
If you liked what you heard, remember to smash, like, and subscribe so hard that there's a little hole in the monitor where you smash it.
01:40:13.279 --> 01:40:15.760
Um and uh please join us again next time.