We Rebuilt PHP From the Inside | The HHVM Story

WEBVTT
00:00:00.239 --> 00:00:05.360
Today we're gonna be talking about HHVM and how HHVM came to be at Facebook.
00:00:05.519 --> 00:00:13.839
So HHVM is a JIT compiler developed by Keith, um Keith Adams, right here, Jason Evans, and Drew Poroski.
00:00:14.080 --> 00:00:16.239
And this is the HHVM story.
00:00:16.480 --> 00:00:20.160
I'm Julian Verdeguet, I'm the CEO of uh Skip Labs.
00:00:20.480 --> 00:00:23.679
So I'm Keith Adams, sometimes known as KMA for my login at various companies.
00:00:24.559 --> 00:00:29.839
I'm founder and managing partner at PebbleBed, an early stage VC firm that works with technically ambitious founders.
00:00:30.079 --> 00:00:37.280
You managed to do pre-IPO in pretty cool companies like three times, right?
00:00:37.520 --> 00:00:47.200
Like, like, you know, among engineers, you're like, that guy, you know, wherever wherever Keith is going next, you know, that's where we need to go, you know?
00:00:47.359 --> 00:00:51.920
Because VMware, you were what employee number, number what, 80, something like that?
00:00:52.240 --> 00:00:54.320
Yeah, yeah, like just under 100, I think.
00:00:54.479 --> 00:00:55.920
Um, about 20 engineers.
00:00:56.079 --> 00:01:00.159
So, you know, this wasn't like two guys and their dog dreaming about virtual machines or something, right?
00:01:00.240 --> 00:01:04.079
There was like a real direction set and a product and and customers and things.
00:01:04.239 --> 00:01:06.319
But honestly, like VMware was just kind of a lucky hit, right?
00:01:06.400 --> 00:01:08.640
Like that was I was graduating from college in 2000.
00:01:08.879 --> 00:01:20.000
I went to Brown University, a lot of really bright people from the computer science class as at Brown ended up at Sun Microsystems, especially people who were excited about OS's and CPUs and the things that I was excited about.
00:01:20.239 --> 00:01:23.680
I had an offer from Sun Microsystems actually, which I was incredibly excited about.
00:01:23.920 --> 00:01:26.400
But uh there was also this crazy VMware place.
00:01:26.560 --> 00:01:36.319
And when uh when I went to see a talk by a guy named Ole Agason, who I had the privilege of working with for nine years at VMware, he came to Brown University and gave like a pizza talk about how this thing worked.
00:01:36.400 --> 00:01:38.799
I thought it was amazing and thought he was amazing.
00:01:39.040 --> 00:01:46.640
I thought, wow, I gotta, I've gotta, I've gotta expose myself to this person any way I can uh while I still while it's still an easy decision to make.
00:01:46.799 --> 00:01:50.239
And if this VMware thing blows up, hey, at least Sun Microsystems will always be there.
00:01:50.400 --> 00:01:55.040
Which of course, like, you know, ended up being the opposite of how this all worked out.
00:01:55.120 --> 00:01:59.760
You know, Pachay and my friends at Sun who uh went on to do amazing work in the 2000s.
00:02:00.079 --> 00:02:01.200
It feels like a ton of luck.
00:02:01.280 --> 00:02:22.240
It was an incredible place, really brilliant people, really uh in hindsight, like really high standards of sort of technical excellence, really high standards of just like what it meant to do a good job and what it meant to ship something quality to customers and how good things really needed to be before they were sort of ready for other people to consume, which I'm glad I've had those standards kind of uh imbued in me relatively early.
00:02:22.479 --> 00:02:28.000
After the 2008 financial crisis, um VMware had already been acquired by EMC at this point.
00:02:28.240 --> 00:02:39.120
EMC had uh just like decapitated VMware, just gotten rid of Diane Green, who was the founding CEO of VMware, um, and had sort of absorbed started absorbing VMware more into EMC proper.
00:02:39.360 --> 00:02:45.680
There was a lot of kind of mis, you know, general bad vibes, I'd say, uh, among the rank and file at VMware.
00:02:45.840 --> 00:02:49.439
Certainly for the first time, I'd always assumed VMware was going to be my entire professional identity, right?
00:02:49.520 --> 00:02:53.599
I'd I was 30 years old at this point and had been there since I left college.
00:02:53.840 --> 00:02:59.759
Um and for the first time I thought, like, well, if Diane's disposable, I guess I'm disposable too, and what does that mean?
00:02:59.919 --> 00:03:06.240
And there's a recruiter at Facebook named Sarah Currien who like got the VMware uh like got the phone book, basically.
00:03:06.479 --> 00:03:09.599
But this is back when people had landlines on their desks at work, right?
00:03:09.680 --> 00:03:12.080
So this is the little Cisco phone that you pick up.
00:03:12.240 --> 00:03:19.360
And she war dialed the VMware directory for engineers, and like anybody who picked up, uh, she was like, Hey, my name's Sarah, I'm a recruiter from Facebook.
00:03:19.439 --> 00:03:21.039
Like, can I get coffee with you?
00:03:21.280 --> 00:03:27.120
And 99 times out of a hundred, I'd have probably said no, but like, you know, of course this was the week to do it, and that was why she was doing it.
00:03:27.360 --> 00:03:29.120
Um and kind of one thing led to another.
00:03:29.360 --> 00:03:36.719
So that's actually funny because my uh the way I was recruited at Facebook originally I didn't want to join Facebook at all.
00:03:36.879 --> 00:03:43.599
I wanted to join Google and for me, um Facebook was not serious people.
00:03:44.319 --> 00:03:50.719
So you know, for me Facebook was just a bunch of clown pushing PHP, don't know what they're doing whatsoever.
00:03:51.039 --> 00:04:02.560
And uh I was working in a startup back then in in Mountain View, and uh a recruiter from Google um contacted me and uh my roommate was working at Facebook back then.
00:04:02.719 --> 00:04:04.000
I don't know if you remember pad.
00:04:04.560 --> 00:04:11.039
Yeah, yeah, and and pad was like, Well, you should definitely try Facebook or refer you and whatnot.
00:04:11.199 --> 00:04:18.240
So I I I tried a loop at Facebook, but really my eye was set on I mean my my mind was set on on Google.
00:04:18.639 --> 00:04:23.120
And uh it was the interview process, and you were actually part of it.
00:04:23.279 --> 00:04:29.759
Like the the crowd that interviewed me, my my screen interview was with Jason Evans.
00:04:30.000 --> 00:04:30.959
Let's start with that.
00:04:31.279 --> 00:04:32.879
You know, the the screen, right?
00:04:33.040 --> 00:04:36.480
And so I I have him on the phone.
00:04:36.639 --> 00:04:38.079
I mean he's he's he's great.
00:04:38.160 --> 00:04:38.879
I love him.
00:04:39.120 --> 00:04:47.519
Um I have him on the phone, and uh he says his name, but uh you know I don't I don't recall, I'm I'm like I I don't pay attention.
00:04:47.680 --> 00:04:49.360
I'm like, okay, well let's do the interview.
00:04:49.680 --> 00:04:51.680
By the way, Jason Evans of still of J.E.
00:04:51.759 --> 00:04:53.120
Mallock fame, I would say, right?
00:04:53.360 --> 00:04:53.600
Exactly.
00:04:53.920 --> 00:04:57.519
Probably wrote wrote the memory allocator you're using to listen to this, for instance.
00:04:57.759 --> 00:04:58.800
Sorry, go ahead.
00:04:59.120 --> 00:05:01.759
And um so that's actually what what I said.
00:05:01.920 --> 00:05:06.000
I was like, well, wait a minute, you're Jess Jason Evans from J.E.
00:05:06.160 --> 00:05:06.879
Mallock?
00:05:07.040 --> 00:05:10.000
And he's like, yeah, like I know your stuff.
00:05:10.160 --> 00:05:11.759
And I started talking about his code.
00:05:11.839 --> 00:05:12.800
I mean, it's pretty cool.
00:05:12.879 --> 00:05:16.079
You have cool data structures in there, and that that was amazing.
00:05:16.240 --> 00:05:26.639
So then I wrote some code, which I'm sure was wrong, and I think at the end he said that I'm pretty sure that's wrong, but I wouldn't be able to tell you why.
00:05:27.040 --> 00:05:30.160
And so and so that's how I I passed that interview.
00:05:30.319 --> 00:05:32.720
And then the the the crowd was great.
00:05:32.879 --> 00:05:36.800
I had um uh Ben Matthews who interviewed me.
00:05:37.040 --> 00:05:38.800
I had uh I had you.
00:05:38.959 --> 00:05:40.720
You you was a pretty good one.
00:05:40.879 --> 00:05:44.879
It was like uh, well, how do you write a garbage collector?
00:05:45.040 --> 00:05:46.160
What's a garbage collector?
00:05:46.319 --> 00:05:48.480
What are the trade-offs of a garbage collector?
00:05:48.560 --> 00:05:51.519
And and we started you know, ranting about garbage collectors.
00:05:51.600 --> 00:05:52.319
And yeah, I wonder.
00:05:53.519 --> 00:05:58.639
There's something loopy in it because you were writing OCaml and you sort of like wrote it a little bit like it was C.
00:05:59.040 --> 00:06:01.199
Like you wrote it with like a bunch of loops constructs in it.
00:06:01.279 --> 00:06:03.120
And I was like, let's be OCaml programmers.
00:06:03.279 --> 00:06:03.920
Like go ahead.
00:06:04.000 --> 00:06:04.720
Like, I get it.
00:06:04.800 --> 00:06:09.279
Like you know I'm a C programmer, but you know, take me do this idiomatically for my benefit, please.
00:06:09.600 --> 00:06:11.920
So actually, I was dumbing it down for you, Keith.
00:06:12.000 --> 00:06:19.199
I I was hoping that you know the C programmer you are would not understand the OCaml beauty.
00:06:19.279 --> 00:06:22.879
So no, but uh in all seriousness, it was it was a really cool interview.
00:06:22.959 --> 00:06:27.279
And I walked out of it thinking, wow, I really want to work for for Facebook.
00:06:27.360 --> 00:06:28.879
They have really cool people there.
00:06:29.040 --> 00:06:31.040
I think that's in hindsight, by the way, Julian.
00:06:31.199 --> 00:06:33.600
I probably had a similar set of preconceptions, right?
00:06:33.759 --> 00:06:37.279
So uh I guess a little bit of of backstory here too, by the way.
00:06:37.360 --> 00:06:44.240
Like when I was interviewing in 2008, they actually had been in Silicon Valley for very long, a recent transplant from Boston, right?
00:06:44.319 --> 00:06:46.160
When it was a you know Harvard startup.
00:06:46.399 --> 00:06:57.600
And the kind of perception of them among locals, which like mostly didn't know a lot of people who worked there, was like, okay, there's this weird thing, it's a big hit on college campuses or something.
00:06:57.759 --> 00:07:03.519
And the assumption was that these are people that like maybe had some product sense, but that there was no sort of serious technology going on.
00:07:03.600 --> 00:07:07.120
And so I wasn't really sure what I was getting myself into when I showed up for an interview.
00:07:07.279 --> 00:07:20.800
Uh Jeff Rothschild was the first person that that gave me the interview there, and I'd like had some contact with people from Veritas uh in the process of of working at VMware because Veritas kind of was deep in the DNA of of System Z Silicon Valley at the time.
00:07:20.959 --> 00:07:23.920
Um and he actually kind of nagged me a little bit about VMware in the interview.
00:07:24.000 --> 00:07:28.079
He's kind of like, yeah, this stuff's kind of good for test and dev, I guess, but like nobody's ever gonna run that garbage in production.
00:07:28.160 --> 00:07:33.360
And I was like, well, actually, sir, I mean, I kind of do use it and you know, uh so you know it kind of put me on my back foot a little bit.
00:07:33.439 --> 00:07:37.439
I wasn't expecting to have to defend the validity of what I'd been doing for the last nine years.
00:07:37.680 --> 00:07:43.279
Like I thought I'd be teaching these PHP bozos a thing or two about how computers worked, and it was kind of it was a little bit the other way around.
00:07:43.360 --> 00:07:46.720
And Bobby Johnson also um other really bright people.
00:07:46.879 --> 00:07:47.040
Yeah.
00:07:47.199 --> 00:07:57.759
You know, Facebook at some level was like uh in terms of its impact on the economy, it was a pump that turned venture capital dollars into top line for network appliance, for NetApp, right?
00:07:57.920 --> 00:08:06.560
It was it was storing photos in these big NetApp filers, many, many, you know, uncountably many petabytes of this stuff, um, because it had a huge hit on its hands for photos.
00:08:06.639 --> 00:08:11.120
It was like the first time people in a mass consumer way were really sharing photographs with each other online.
00:08:11.279 --> 00:08:15.279
There were things like Flickr, but they were kind of for prosumers who were you know hobbyist photographers.
00:08:15.360 --> 00:08:19.199
It wasn't like normal people snapping photos with their cell phones and things until right then.
00:08:19.360 --> 00:08:26.560
And so uh those were extremely expensive ways to store those photos on the one hand, but they needed kind of fast, reliable storage of all these things.
00:08:26.879 --> 00:08:38.720
Uh when I joined one of my goals, and in fact it was, you know, during the the onboarding, there was one day where they asked you, where do you think you could have impact for Facebook, da-da-da.
00:08:38.960 --> 00:08:44.000
And there's a piece of paper where you wrote it down, and I wrote down that I was gonna make PHP better.
00:08:44.159 --> 00:08:45.759
That this was shit.
00:08:46.559 --> 00:08:51.279
This this was not, you know, this is not gonna I I I wasn't gonna leave it at that.
00:08:51.360 --> 00:08:57.360
I mean, I spent a bunch of years working on, you know, um critical embedded software.
00:08:57.600 --> 00:09:12.159
Uh so you know, compilers for things that end up in Airbus, in trains, in planes, in nuclear plants, and I mean kind of things where if the software fails, people are gonna die, kind of thing, right?
00:09:12.240 --> 00:09:15.200
Hence the name, uh safety critical software.
00:09:15.440 --> 00:09:20.320
And when we were writing code at this first company, the goal was zero bug.
00:09:20.480 --> 00:09:20.639
Right?
00:09:20.799 --> 00:09:28.000
If you had a bug, you had to email all your customers, it was very long and complicated process.
00:09:28.080 --> 00:09:33.440
So the goal was really to get it down to zero bug, no matter how long it takes.
00:09:33.840 --> 00:09:47.679
And from that to dub dub dub, like the PHP code base at Facebook, it's like, whoa, what this is so and and all in all fairness, it was actually a pretty good code base.
00:09:47.840 --> 00:09:54.080
Like if I compare to other companies that I worked for afterwards, especially for being like a paleo PHP code base, right?
00:09:54.159 --> 00:10:00.799
Especially for being a code base that never got rotor tiled from Zuck hacking away in his dorm room in 2004 or whatever.
00:10:00.960 --> 00:10:01.200
Yeah.
00:10:01.759 --> 00:10:10.080
I mean, the one thing that I would say was detrimental to the quality of the code was that they would never finish transitions.
00:10:10.240 --> 00:10:11.200
So migrations.
00:10:11.360 --> 00:10:20.399
So they would start migrations to new things, and then somebody would come along with a new idea, and they they hadn't finished off the migration before.
00:10:20.720 --> 00:10:26.720
And so what ended up one of my sort of party tricks I would do in bootcamp as late as 2015, right?
00:10:26.799 --> 00:10:34.559
When I when people when new people would join the company, uh, was actually doing a little tour of places in the code where things still get blamed to Zuck.
00:10:34.799 --> 00:10:40.720
And you could still find like SQL string constants that are like select star from users where ID equals blah.
00:10:40.879 --> 00:10:42.480
That's like git blamed zuck or whatever.
00:10:42.559 --> 00:10:43.919
And is that code even live?
00:10:44.080 --> 00:10:44.559
Probably not.
00:10:44.720 --> 00:10:49.919
Actually, there's probably like no way to actually get to that code in the code base anymore, at least not with any URLs that are live.
00:10:50.080 --> 00:10:51.600
Um sorry, I didn't mean to interrupt your story there.
00:10:52.240 --> 00:10:59.200
No, I I but it's the transitions went all the way down to the the unfinished transitions go all the way down to select stars and users where ID equals.
00:10:59.600 --> 00:11:00.480
Yeah, exactly.
00:11:00.639 --> 00:11:04.559
I mean, that was also the first the first uh stuff I was working on, right?
00:11:04.639 --> 00:11:11.120
The first stuff I was working on at Facebook was static analysis to uh find security holes.
00:11:11.679 --> 00:11:21.200
And in fact, I was very surprised how well the the the the technique was working because normally you know in a dynamic language you really don't get very far.
00:11:21.360 --> 00:11:27.279
And the the reason is relatively simple, which is that if you don't have the types, you lose the control very quickly.
00:11:27.360 --> 00:11:42.960
Like the moment you call a method, if the method has a somewhat generic name, like render, if you're rendering HTML, uh you really don't know what you're calling, and therefore you really don't know what to analyze, and basically your analysis and is not going very far.
00:11:43.200 --> 00:12:04.799
But I was still able to build an analysis that found a lot of security holes, uh, because there was a lot of old style PHP, you know, where everything is a top-level, the kind of stuff that you're describing, like SQL queries, where clearly it wasn't escaped the way it was supposed to be escaped at all.
00:12:05.039 --> 00:12:13.200
And so so we were we were able to found a lot of uh a lot of interesting uh bugs this way uh back in the code base.
00:12:13.600 --> 00:12:16.159
I think it's cool for us to say that now.
00:12:16.240 --> 00:12:17.679
It has been what 15 years?
00:12:17.919 --> 00:12:18.320
Because you know.
00:12:20.879 --> 00:12:24.000
I assume it's all like outer space, laser beam, sci-fi now.
00:12:24.159 --> 00:12:27.600
And if it isn't, some reachability analysis has been performed or whatever.
00:12:28.000 --> 00:12:33.360
Because 15 years ago, I would have been wary, you know, saying that there was security holes within Facebook.
00:12:33.600 --> 00:12:41.039
Uh, because you know that was my job, and I didn't want too many people to know outside of Facebook that this was the case.
00:12:41.279 --> 00:12:45.600
But uh there was this funny story of a guy who built an endpoint.
00:12:45.759 --> 00:12:50.480
So I was in the security team, that was my first team before joining HHVM Hack.
00:12:51.279 --> 00:13:00.799
And there was a guy who built an endpoint, and the endpoint was literally take the parameter and run it in hyper shell.
00:13:01.120 --> 00:13:07.279
So hyper shell being the shell that runs this command as root on all machines in production.
00:13:07.519 --> 00:13:08.240
Oh, yeah, yeah.
00:13:08.320 --> 00:13:10.240
The root production command.
00:13:10.320 --> 00:13:12.159
Yeah, yeah, the the destroy all of the.
00:13:12.399 --> 00:13:24.159
So he had a little web page, a little internal web page where you typed any text you liked, then you press run, and the thing was root in production everywhere.
00:13:24.960 --> 00:13:35.200
So that I have I have this static analysis that immediately goes like alert, alert, what is this?
00:13:35.519 --> 00:13:43.919
I can see the veins of Erling, who was working on the team, like his veins were pumping, you know, like what the fuck is this?
00:13:44.159 --> 00:13:53.440
And he went to see this guy, and and the guy was like, Yeah, but it's really useful when I'm on holiday, you know, I can push stuff in production.
00:13:53.840 --> 00:13:57.039
And I saw the moment where Erling was gonna struggle him.
00:13:58.480 --> 00:13:59.679
Like, shut up.
00:14:01.360 --> 00:14:07.200
And I mean, it's just to give you an idea of how how far things were back then.
00:14:07.440 --> 00:14:14.480
We went from, you know, cowboy cow cowboy land to something actually pretty serious.
00:14:14.639 --> 00:14:22.240
Like if you wanna if you wanna hack into Facebook these days, hmm, you bet that's a pretty major project, I think, right?
00:14:22.399 --> 00:14:24.559
That's uh there's some research professionals of hard.
00:14:24.720 --> 00:14:28.960
Yeah, we're talking about like the mid-2000, you know, the early 2010s, roughly, right?
00:14:29.039 --> 00:14:31.440
Like late, late aughts to early 2010s.
00:14:31.600 --> 00:14:48.639
Like the people that started to sort of come into Facebook at that time might have had backgrounds that were more like yours and mine, where lives are on the line or billions of dollars were on the line, or you know, there's sort of an expectation of fitness to purpose of any of the software you're making that wasn't necessarily there for internet products, you know, that were just sort of consumery before.
00:14:48.720 --> 00:14:54.399
I think like there was a time with Facebook where you could kind of be like, well, we're like, we're not negotiating with aliens for the future of Earth, right?
00:14:54.480 --> 00:14:55.919
These are just people posting pictures of their cats.
00:14:56.000 --> 00:15:02.399
Like it's more important that this be fun and easy to iterate on than it is that like, you know, we're not lives aren't on the line broadly.
00:15:02.559 --> 00:15:11.200
But past a certain point, like there's sort of so much value and sensitivity and private data and everything else involved that like lives essentially are on the line.
00:15:11.519 --> 00:15:14.960
I was always pretty impressed how seriously people took that responsibility.
00:15:15.120 --> 00:15:18.799
It's just it's impossible to do it perfectly, if that makes any sense, right?
00:15:18.960 --> 00:15:24.320
So um, for example, like there was a conflict happening in a region, right?
00:15:24.960 --> 00:15:27.279
And uh like a civil war, basically.
00:15:27.840 --> 00:15:32.480
And it's a civil war where like nobody in the Menlo Park office speaks the local languages.
00:15:32.639 --> 00:15:34.799
Okay, this is before you know AI, right?
00:15:34.879 --> 00:15:37.440
We can't just translate it and we can't just read the messages.
00:15:37.759 --> 00:15:44.720
And it comes to our attention that people are using Facebook Messenger basically to organize roving gangs, right?
00:15:44.799 --> 00:15:48.159
To organize like groups of people that are going out and killing other people.
00:15:48.399 --> 00:15:50.960
Now, we have no context about the civil war.
00:15:51.679 --> 00:15:52.799
Are these the good guys?
00:15:52.879 --> 00:15:53.759
Are these the bad guys?
00:15:53.919 --> 00:15:56.559
Does the concept of good guys and bad guys have any relevance?
00:15:56.720 --> 00:15:58.799
We have we have no bloody idea, right?
00:15:59.200 --> 00:16:01.759
Uh do you want to turn messenger off in that region?
00:16:01.840 --> 00:16:03.039
Is that the right thing to do?
00:16:03.279 --> 00:16:04.720
Uh again, AI doesn't work.
00:16:04.799 --> 00:16:11.279
There's not like some neural net you can snap on the front of it to try to classify messages as being like potentially about violence or something like that.
00:16:11.519 --> 00:16:18.159
And I mean, there this was a pretty big emergency that ran for a while, and there was no kind of obvious commercial impact.
00:16:18.320 --> 00:16:19.840
I just was literally lives on the line.
00:16:19.919 --> 00:16:24.240
People were using our communication product to do stuff of extreme consequence.
00:16:24.720 --> 00:16:27.440
And it reminded me, like it was a strange problem.
00:16:27.519 --> 00:16:31.759
It sort of reminded me a little bit of a capitalist version of the social the socialist calculation problem.
00:16:31.919 --> 00:16:36.879
One of the reasons communism ends up being really hard to do in practice is that economies are chaotic, right?
00:16:36.960 --> 00:16:48.159
If I'm trying to make a plan for the economy of the Soviet Union and I'm trying to figure out in three years or five years, uh, one of the things I need to figure out is like how much yellow paint are we going to need in the city of Kiev three years from now.
00:16:48.399 --> 00:16:49.679
And that's like almost impossible.
00:16:49.919 --> 00:16:50.960
Information doesn't really exist.
00:16:51.039 --> 00:16:57.360
Like you need things like price signals and you know companies getting created and destroyed to kind of actually meet that demand.
00:16:57.919 --> 00:17:01.360
And this was like a weird consumer internet version of that, if that makes any sense.
00:17:01.519 --> 00:17:08.000
It was like there's sort of so much power concentrated in this one platform that there was almost no way to wisely wield that power.
00:17:08.400 --> 00:17:10.640
And I don't know what to do about this this observation, by the way.
00:17:10.720 --> 00:17:16.640
I don't think there's any sort of super clear um sort of civilization-wide question like what you should do about this.
00:17:17.200 --> 00:17:21.519
I remember I remember some of those uh some of those problems too.
00:17:21.839 --> 00:17:24.480
And there was no clear solution.
00:17:24.640 --> 00:17:33.039
And I also there was a lot of suspicion um in the I mean towards the company back then.
00:17:33.119 --> 00:17:43.920
I remember especially when you know there was the bug, uh where apparently Facebook was publishing all your private messages on your timelines.
00:17:44.240 --> 00:17:44.720
Do you remember that?
00:17:45.279 --> 00:17:47.440
Back then I was working on Facebook, you were two.
00:17:47.839 --> 00:17:51.279
And I had a bunch of friends who just wouldn't believe me.
00:17:51.599 --> 00:17:59.200
You know, they they thought it was a c cover up because the real story was before, you know, at the beginning of Facebook, there was no messenger.
00:17:59.519 --> 00:18:02.160
And there was barely anyone on Facebook.
00:18:02.400 --> 00:18:05.920
You know, it was your wall, was you and your five friends.
00:18:06.160 --> 00:18:08.079
And so people were talking to each other.
00:18:08.799 --> 00:18:09.119
By the way, right?
00:18:09.200 --> 00:18:09.519
There was no news.
00:18:09.680 --> 00:18:10.720
There was no news feed either.
00:18:10.799 --> 00:18:10.960
Yeah.
00:18:11.440 --> 00:18:13.519
Airsatz distribution of content, right?
00:18:13.599 --> 00:18:15.680
It was like you'd go to pro from profile to profile.
00:18:15.759 --> 00:18:17.920
I'd go to Facebook to like look at your profile.
00:18:18.480 --> 00:18:18.960
Exactly.
00:18:19.039 --> 00:18:25.200
And so what people were doing is they were messaging each other on the wall, on their wall directly.
00:18:25.519 --> 00:18:37.759
And so once Timeline was, you know, was launched, basically you would take all these old messages that were on the wall that have always been public, right?
00:18:37.920 --> 00:18:48.880
But now they're in a form that looks like private messages are actually on your on your wall or were brought brought into your wall, except that they're they've always been there, right?
00:18:49.359 --> 00:19:04.240
So all all the media in France went nuts, including media that I really trusted, you know, like like um newspapers I'd been reading forever, and they published this thing that there was this bug at Facebook, da-da-da.
00:19:05.039 --> 00:19:07.440
Same thing up in the States, by the way, the New York Times is all over this.
00:19:07.680 --> 00:19:08.640
Like name your favorite.
00:19:08.799 --> 00:19:10.079
I assume Le Monde is one of the ones you're thinking about.
00:19:10.400 --> 00:19:12.160
Le Monde is the is the one I was thinking about.
00:19:12.319 --> 00:19:12.720
Yeah, yeah.
00:19:12.960 --> 00:19:21.680
And what really surprised me is that no journalists, no team, no newspaper, nothing actually corrected themselves.
00:19:21.920 --> 00:19:26.319
Because the fact that they made a mistake, that was one thing, you know, that happens.
00:19:26.640 --> 00:19:36.720
But there was, you know, in Le Monde, which was a newspaper I read for many, many, many years, all we had in there was page 34.
00:19:37.119 --> 00:19:49.759
There was a little, a little a little article that said, Oh yeah, Facebook maybe didn't really have a bug, but that doesn't mean anything because there was all the these other bugs that were there before, and those were weird.
00:19:50.960 --> 00:19:51.119
Exactly.
00:19:52.559 --> 00:19:52.960
Don't worry.
00:19:53.039 --> 00:19:54.079
You should still think they suck.
00:19:54.720 --> 00:19:54.880
Okay.
00:19:55.440 --> 00:19:58.400
You're like, okay, so truth doesn't really matter, right?
00:19:58.480 --> 00:20:04.079
Like it doesn't matter that there was no bug, that this is all blown out of proportion for nothing at all.
00:20:04.240 --> 00:20:09.839
But it it yeah, I mean, it it changed my perspective on many things.
00:20:10.079 --> 00:20:20.720
Because I mean, if you know, today I was asked to actually run a social media platform, I I think I would struggle.
00:20:21.039 --> 00:20:23.440
You know, like like how getting that.
00:20:26.880 --> 00:20:27.119
Yeah, yeah.
00:20:27.359 --> 00:20:28.400
You'd be sort of hemmed in.
00:20:28.480 --> 00:20:29.039
Yeah, I know what you mean.
00:20:29.440 --> 00:20:34.720
You're like, you know, either you make it too restrictive and then your network is never gonna grow.
00:20:35.039 --> 00:20:40.559
Nobody is gonna because they're gonna feel like imprisoned, like they cannot say the stuff that they care about.
00:20:41.359 --> 00:20:47.680
Or you open the gates, but then you know, you end up in this cesspool hate.
00:20:47.920 --> 00:20:58.079
I mean not to point at some specific, you know, uh social network that has done that recently, like open the floodgates and see what happens.
00:20:58.960 --> 00:21:09.440
Uh but you know, it's I mean, uh don't get me wrong, I'm I'm in favor of um uh freedom of uh of speech, and I really believe in freedom of speech.
00:21:09.680 --> 00:21:14.400
Uh but yeah, running a social social media platform is is rough.
00:21:14.640 --> 00:21:17.839
But tell me more about uh I want to hear the story about HHVM.
00:21:17.920 --> 00:21:18.720
And okay, yeah.
00:21:18.799 --> 00:21:24.240
So we know how we say this back and forth to each other, and like normal people don't know or care what the hell this thing is, by the way.
00:21:24.319 --> 00:21:27.839
So we should of course everybody knows what HHVM is, you know.
00:21:27.920 --> 00:21:33.039
They're Taylor Swift, the kids today, Michael Jordan, yeah, yeah.
00:21:33.440 --> 00:21:36.319
HHVM, you know, of course.
00:21:36.880 --> 00:21:59.680
Yeah, so uh HHVM, which stood for the hip hop virtual machine, uh a little bit of a long story on the name, uh was uh basically a JIT compiler, a just in time compiler for PHP that me, Jason Evans, who we talked about a second ago, and Drew Pirofsky started working on in uh 2010 and eventually delivered to production in in early 2013, late 2012.
00:22:00.640 --> 00:22:07.759
And uh it was a reaction to kind of economic pressures, actually, that Facebook faced in certain ways.
00:22:07.920 --> 00:22:12.000
So, you know, you hear a lot about compute costs in the context of AI right now.
00:22:12.480 --> 00:22:18.799
In the microcosm of Facebook, we were spending a lot of money on compute per user at the time as well.
00:22:19.039 --> 00:22:26.640
It was one of the things that actually kind of threatened the economic model of Facebook, in part because the ads model hadn't really taken off yet at the time we started this.
00:22:26.880 --> 00:22:32.319
And uh the amount of CPU we actually had to spend per user to run the site was really, really enormous.
00:22:32.480 --> 00:22:40.000
Um and a big chunk of that we felt, and we had some reason to believe was because of the PHP language, which is a it's a developer productivity language, right?
00:22:40.079 --> 00:22:52.000
It's interpreted language, it's you know really easy to change, it's really easy to get started with, but it's really, really uh uh, you know, really free with its use of CPU resources and memory resources.
00:22:52.160 --> 00:22:58.160
And so you have to buy a lot of computer if you're if you're gonna try to run a mass market site with it, and we were trying to do something to tamp those resources down.
00:22:58.400 --> 00:23:03.279
This is the second major effort that Facebook undertook to get this problem under control.
00:23:03.440 --> 00:23:14.480
And the first one was called hip hop, uh, or it originally was called HPHP, because Hai Ping Zhao, uh, an engineer at Facebook before I started there, uh got it started.
00:23:14.640 --> 00:23:20.079
And then it turns out that PHP is a trademark that's owned by like a company in Israel, and they're a little bit litigious.
00:23:20.160 --> 00:23:21.599
Some words were exchanged.
00:23:21.680 --> 00:23:27.359
And so if you sort of grep for HP, HP in user-dicked words, you'll find like hip-hop and not that many other words.
00:23:27.440 --> 00:23:29.440
So hip-hop was the name of that thing.
00:23:29.519 --> 00:23:30.640
And it was an ahead-of-time compiler.
00:23:30.720 --> 00:23:42.799
So it worked, you know, the way your C compiler works, or your Go compiler works, or your Rust compiler works these days, um, where it like takes this gigantic code base that was, you know, dub dub dub, which we've referred to before, and it spits out a big an elf binary.
00:23:42.960 --> 00:23:48.240
It spits out a thing that listens on port 80, um, you know, the same way that a static language compiler would.
00:23:48.400 --> 00:23:54.559
We should also mention, you know, not only is Facebook kind of a PHP application at this time, it's a PHP monolith.
00:23:54.720 --> 00:23:57.279
It's a it's one big PHP application.
00:23:57.440 --> 00:24:02.400
Um so that's then it's like 80% of the Facebook code is PHP, right?
00:24:02.480 --> 00:24:02.880
Something like that.
00:24:03.200 --> 00:24:03.359
Exactly.
00:24:03.599 --> 00:24:04.079
Something like that.
00:24:04.319 --> 00:24:12.000
At the time, a gigantic amount of the intellectual property and the thing that makes Facebook go is this one repo in in PHP.
00:24:12.400 --> 00:24:21.119
And it was a sort of understanding across the engineering organization that there was a you know big, hairy, audacious goal would be make www run fast.
00:24:21.440 --> 00:24:26.960
Um and one approach to do that is like take it all, turn it into C, run it all through GCC.
00:24:27.119 --> 00:24:30.880
And that was what uh what hip hop did, what HPHP did.
00:24:31.119 --> 00:24:32.880
And do you want to let me know?
00:24:33.359 --> 00:24:34.880
The original goal was not that, right?
00:24:34.960 --> 00:24:45.119
The original goal was actually to transpile it to C and then get all the PHP developers to switch to C to develop the the web backend, right?
00:24:45.200 --> 00:24:45.599
That that was.
00:24:45.920 --> 00:25:01.279
This is uh this is a slightly, you're right, this is a little bit of a side quest here, but but hyping you know was a C programmer, like a lot of people who were coming into Facebook at the time, and um, you know, kind of squinted at a PHP from across the room, said, Hey, there's curly braces, there's semicolons, like what's the problem here?
00:25:01.519 --> 00:25:03.440
This is basically C anyway.
00:25:03.519 --> 00:25:11.119
Um if we just kind of turn this lame, slow PHP code base into a beautiful, fast C code base, then we can really start cooking here.
00:25:11.279 --> 00:25:22.960
And so the idea was really initially that like you'd run it one day, check in all the crazy C that you generated, and then move forward into a glorious C based future.
00:25:23.200 --> 00:25:26.319
Um this proved to not be the way that things could work.
00:25:26.400 --> 00:25:28.720
It turned out there were reasons people liked PHP.
00:25:28.880 --> 00:25:31.440
Some of those reasons included the kind of interpreted workflow, right?
00:25:31.519 --> 00:25:34.480
That I like save one file and can you know see what happened.
00:25:34.640 --> 00:25:34.880
Yeah.
00:25:35.119 --> 00:25:40.160
Um whereas if you know you've got at this time it's already like tens of millions of lines of code, it's already a huge application.
00:25:40.319 --> 00:25:44.640
Where if you've got some huge compile cycle in the middle of that, that really kind of messes that flow up.
00:25:44.880 --> 00:25:49.599
So that would have faced enormous like people who just quit their jobs who worked on the product side of things.
00:25:49.920 --> 00:25:51.759
I mean, imagine the compilation time.
00:25:52.000 --> 00:25:52.240
Right?
00:25:52.400 --> 00:25:54.720
Imagine you're working on www in C.
00:25:55.759 --> 00:25:56.000
Yeah.
00:25:56.160 --> 00:25:59.759
And in a in you know, in a code base that hasn't been thought out for it.
00:26:00.240 --> 00:26:05.440
Imagine the compilation time of you know, making getting anything done in W.
00:26:06.559 --> 00:26:08.720
You would wait hours to compile it.
00:26:09.279 --> 00:26:11.039
PHP has no concept of a header file, right?
00:26:11.119 --> 00:26:17.920
So there's no there there were there would be a lot of work getting it into the shape where it's sort of a project that you can actually work on in an incremental way uh in a way.
00:26:18.240 --> 00:26:19.119
I don't think it was possible.
00:26:19.200 --> 00:26:38.559
So I looked into that back in the days because I wanted to, in fact, Pad was the first, so Yoan Padiolo was the first to go on this crusade to try to um split up the code base into you know clean submodules and get the dependency layout figured out properly.
00:26:38.960 --> 00:26:51.440
Um and and he he he went into this whole visualization tool that was called overlay, something like that, that would show you the code base and the different regions of the code base and what depends on what.
00:26:51.839 --> 00:27:04.480
The problem is that there was no incentive to go after that, but to give you an idea, there was chunks of code with cyclic dependencies that would pull in more than eight million lines of code.
00:27:04.799 --> 00:27:05.519
Yep, exactly.
00:27:05.839 --> 00:27:10.720
This is the problem is you end up with like some big cycle somewhere that like where the bottom calls the top and now you're done.
00:27:11.039 --> 00:27:18.480
Because of autoload, so the fact that you know whenever you use a class, it's just automatically loaded by legit, same with functions.
00:27:18.720 --> 00:27:21.599
There's no reason to pay attention to dependencies, right?
00:27:21.680 --> 00:27:24.880
If something works, you're just going to use it and it works, right?
00:27:25.119 --> 00:27:40.079
And if you don't pay at all attention to dependencies, you end up in really interesting situations where that we had the logger that depended on ant and stuff like that, because you know there was some some useful method in there.
00:27:40.559 --> 00:27:50.480
And if I could observe something too, by the way, this is like a you know very broad, very abstract claim that like you should not please don't take this at please don't bet your company on this advice, okay?
00:27:50.799 --> 00:27:55.039
But broadly, like there are clean engineers and dirty engineers.
00:27:55.119 --> 00:27:56.079
Like this is something I've observed.
00:27:56.160 --> 00:27:58.079
And that's nothing to do with that's not a quality judgment.
00:27:58.160 --> 00:28:01.119
I actually like dirty engineers, there are great dirty engineers.
00:28:01.200 --> 00:28:03.680
They're dirty engineers that like can do incredible stuff.
00:28:03.920 --> 00:28:06.240
Julian, I think of you as a very much a clean person.
00:28:06.319 --> 00:28:10.400
Like anybody that can do something new with a type system, you're probably a clean temperamentally.
00:28:10.480 --> 00:28:11.920
Um clean people find me dirty.
00:28:12.079 --> 00:28:16.319
Clean people like find me hard to work with because I'm so because I'm making these layering violations all over the place.
00:28:16.480 --> 00:28:20.960
I'm the only time really dirty engineers find me kind of persnickety and like I'm somewhere in the middle, basically, right?
00:28:21.359 --> 00:28:24.400
I mean it's funny because I see myself as in the middle too.
00:28:24.559 --> 00:28:31.599
Like what you're just saying is I mean, in my world, building a type system for PHP is the dirtiest thing you can do.
00:28:31.920 --> 00:28:37.599
Like I mean I mean, imagine you go to a type system conference and you're the PHP guy.
00:28:38.160 --> 00:28:38.640
I'm sorry.
00:28:40.720 --> 00:28:41.279
It was hard.
00:28:45.920 --> 00:28:49.759
So a type is a set of values.
00:28:50.960 --> 00:28:54.720
Some values are in the type, but other values are not.
00:28:55.680 --> 00:28:55.920
Right.
00:28:56.160 --> 00:29:00.240
Yeah, no, I mean it's it's uh it was harder than you thought.
00:29:00.400 --> 00:29:04.079
And and and so what's interesting is I see myself in the middle too.
00:29:04.319 --> 00:29:08.640
So I agree that temperament has a lot to do with it.
00:29:09.119 --> 00:29:11.359
Um, but it's not the only thing.
00:29:11.519 --> 00:29:16.480
The the other thing that can matter is where are we at in the in the life of a project.
00:29:16.880 --> 00:29:17.119
Right.
00:29:17.359 --> 00:29:25.839
So if you're starting a project and you don't know if it has value yet, I think you should be as dirty as you can be to get to to that information.
00:29:26.400 --> 00:29:33.599
Uh while if you are, you know, later down the lifetime of the project, then you should pay more attention.
00:29:34.079 --> 00:29:43.920
Another distinction too between clean and dirty engineers is another dimension, sorry, is people who like to start things from scratch.
00:29:44.640 --> 00:29:48.160
And people who like to maintain or optimize what's there.
00:29:48.400 --> 00:29:48.559
Yep.
00:29:48.880 --> 00:29:50.240
And you need both.
00:29:50.400 --> 00:29:55.599
And what's interesting is that both classes of engineers tend to despise each other.
00:29:55.839 --> 00:30:00.079
So you've got, you know, the people who decide as like not solving any of the real hard problems.
00:30:00.480 --> 00:30:00.720
Exactly.
00:30:01.200 --> 00:30:14.880
So you've got the people who are maintaining the existing system and optimizing for it, like, oh yeah, all these clowns, you know, building this pie in the sky project that's not never gonna go in production, like this is all bullshit while we're doing the real work.
00:30:15.200 --> 00:30:26.400
And then then the other people, you know, who are building the new system, they're like, oh yeah, all these clowns maintaining this old ass code base going nowhere when the new shit's gonna solve all of that.
00:30:26.720 --> 00:30:35.279
And so what's interesting is that you need both, and sometimes they are both right and sometimes they're both wrong.
00:30:35.440 --> 00:30:40.799
I've seen people cling on to old shit that was gonna retire.
00:30:40.880 --> 00:30:53.279
Like I can remember some people still optimizing HPHP a couple of days before HHVM was entering production, and at some point you're like, dude, let it go.
00:30:54.799 --> 00:31:04.160
Um, and and at the same time, I've also seen the other the the the other way around, like people going off building new shit, like rewriting everything in Rust.
00:31:04.240 --> 00:31:06.480
That is, let's rewrite this in Rust.
00:31:06.640 --> 00:31:07.519
It's like, why?
00:31:07.680 --> 00:31:09.119
It will be faster.
00:31:09.440 --> 00:31:11.519
Do we have performance problems?
00:31:11.839 --> 00:31:15.279
No, but it will be faster and greater and better.
00:31:15.359 --> 00:31:16.799
But and memory safe, yeah.
00:31:17.119 --> 00:31:18.559
And and what problem are you solving?
00:31:20.799 --> 00:31:23.920
Rust, by the way, is a fascinating of talking about this like dirty clean thing.
00:31:24.000 --> 00:31:26.880
Like, Rust is like the clean's attempt to remake the world, right?
00:31:26.960 --> 00:31:30.240
It's their attempt to sort of be like we remake the C well, not the entire one.
00:31:30.559 --> 00:31:30.640
Sure.
00:31:30.799 --> 00:31:31.359
Sure, sure, sure.
00:31:31.440 --> 00:31:35.119
But like what if we kind of wipe the earth of like these nasty things in C.
00:31:35.759 --> 00:31:36.799
And I find it fascinating.
00:31:36.960 --> 00:31:47.440
This is not a novel observation, but I do find it fascinating that like we've got a whole bunch of Rust platforms at this point, and not so many Rust products that anybody finds compelling, right?
00:31:47.599 --> 00:31:53.359
Like there are far more Rust game engines than there are Rust games you want to play.
00:31:53.519 --> 00:31:53.759
Yeah.
00:31:54.000 --> 00:32:03.039
And and there's at least one person who's who suggested that um that that might be because like layering violations are what where fun and surprise kind of come from, right?
00:32:03.119 --> 00:32:05.920
That like that you need to be able to try weird shit in a game.
00:32:06.000 --> 00:32:10.240
Like both as a player and as a developer, you need to be able to kind of be like, well, what if this happens or whatever?
00:32:10.640 --> 00:32:12.720
I mean, I want to talk shit about Rust for a second.
00:32:12.880 --> 00:32:22.240
So, first of all, before I talk shit about Rust, um I was a huge believer in linear types, unique types, call them whatever you want.
00:32:22.640 --> 00:32:33.680
Um that you know, unique types were going to solve the problem of garbage collection, and that we were we would be able to, you know, do system programming in uh memory-safe languages.
00:32:33.839 --> 00:32:52.720
In fact, I believed in it so much that I spent three years of my life developing a compiler which was called Linear ML, which was a version of ML with linear types, so very much like Rust's before its time with an ML syntax and without um mutation.
00:32:53.279 --> 00:32:58.319
I used to believe, I used to believe that you know this stuff was gonna solve everything.
00:32:59.119 --> 00:33:01.519
And now I don't know.
00:33:01.680 --> 00:33:05.359
So part of me really wants Rust to succeed.
00:33:05.519 --> 00:33:12.079
I mean, succeed, it has succeeded in some some way, but like kind of overtake the world of system programming.
00:33:12.400 --> 00:33:20.240
And the main reason is I dislike C so much that I really would like to see it go away.
00:33:20.559 --> 00:33:30.160
Um, but I mean I I have a hard time believing that you know Rust is is entirely going to conquer the the realm of system programming.
00:33:30.319 --> 00:33:31.359
And and here's why.
00:33:32.000 --> 00:33:37.599
So I'd say if you really care about performance, right?
00:33:37.759 --> 00:33:58.960
If you really want to squeeze out everything you can out of the machine, I feel like a language that is an ocean of undefined behavior is always gonna have an edge because the compiler has just so many degrees of freedom that it will be able to do more things than you know a safe language.
00:33:59.279 --> 00:34:02.960
So some people tell me, yeah, but they're unsafe in Rust.
00:34:03.359 --> 00:34:22.960
That's true, but but given that it's not the default, um there isn't there's I mean, unless all your code is unsafe everywhere, like to get the same level of freedom for the compiler, I feel like it's it's going to be a tough one to to to to uh to succeed in.
00:34:23.280 --> 00:34:33.119
And then the one thing that I'm happy with though, with Rust, is that they are um taking over what I call the C Bros.
00:34:33.440 --> 00:34:37.440
So I don't know if I ever told you this concept, but the C Bros.
00:34:37.599 --> 00:34:39.039
So there are people who do Cross.
00:34:39.199 --> 00:34:40.480
Am I one of them before we get into this?
00:34:40.880 --> 00:34:41.679
You're not one of them.
00:34:42.880 --> 00:34:45.199
Let me explain to you what I call the C pro.
00:34:45.599 --> 00:34:57.599
So you've got people who do C for good reasons, like they want to actually, they they need performance, they need to do low-level stuff, like building a JIT, fine, completely legit.
00:34:57.920 --> 00:35:00.239
Um uh, so I'm not talking about that.
00:35:00.400 --> 00:35:06.480
I'm talking about people where you go, you you go, you talk to them, you're like, hey Bob, why why are you writing this in C?
00:35:07.119 --> 00:35:14.480
This is a very, you know, it's it's a very simple service that you could run in Java and go.
00:35:14.559 --> 00:35:17.760
And and the answer is like, because it's fast, bro.
00:35:18.239 --> 00:35:24.480
And so and so you go in, you look at the code, and you're like, that's not fast.
00:35:24.880 --> 00:35:27.679
You're copying everything everywhere.
00:35:27.920 --> 00:35:47.679
Oh, wait, there's undefined behavior here, there, there, there, and you're like, well, and and so for these people who are using C because you know GC is slow and because they're fast, and you know, because they're they're the shit, if they could all switch to Rust, great, right?
00:35:47.840 --> 00:35:48.239
Right, right.
00:35:48.559 --> 00:36:03.199
But I think that the serious hardcore um uh system programmers, I think they will always need to do enough dirty tricks that I don't know that you know Rust can take over.
00:36:03.360 --> 00:36:05.519
Now, don't get me wrong, I hope they will.
00:36:05.679 --> 00:36:10.960
I hope Rust destroys C because I hate C so much.
00:36:11.199 --> 00:36:16.800
And we host like uh we host a weekly Rust learners meetup uh Wednesday evenings at Pebble Bud, by the way.
00:36:16.880 --> 00:36:21.119
So we have like, you know, people who are and the the target audience is basically like, I don't know any Rust.
00:36:21.679 --> 00:36:26.239
Um help me to get like the Rust tool chain on my laptop and just teach me what's going on.
00:36:26.400 --> 00:36:36.880
And watching that population cycle through, the thing I'm sort of grateful to the Rust community for is just like getting kids excited about systems programming and getting people excited about the potential of programming languages in a way.
00:36:37.119 --> 00:36:46.800
Because uh, for a lot of people who don't think of themselves as systems programmers or don't think of themselves as PL people, they are having this experience where they're like, oh, this notation is making a difference, right?
00:36:46.960 --> 00:36:51.920
This this this different way of writing a program is making it easier to be safe in ways I care about.
00:36:52.559 --> 00:37:00.880
And it is like making my program faster and sort of putting things in reach in terms of machine orientation that like weren't in reach before.
00:37:01.119 --> 00:37:03.360
And so I think those are kind of positive things broadly.
00:37:03.440 --> 00:37:06.079
I think that demystifies some of the low-level machine stuff.
00:37:06.320 --> 00:37:10.880
If I had to rebuild civilization from scratch, like I probably would still start with a C compiler, honestly.
00:37:11.039 --> 00:37:14.639
Um, because like that's where that's where the productives are gonna come from.
00:37:16.880 --> 00:37:21.920
Like tide defs and you know, syntax for function function signatures.
00:37:22.400 --> 00:37:23.760
A couple path dependent things aside.
00:37:24.000 --> 00:37:24.800
I get what you're saying, yeah.
00:37:25.039 --> 00:37:26.719
Yeah, a couple path-dependent things aside, right?
00:37:26.800 --> 00:37:30.159
There's there's unforced errors in C and C inherited them all, unfortunately.
00:37:30.239 --> 00:37:46.480
And with when you look at sort of some of these projects like Zig and Rust that sort of feel like these huge breaths of fresh air, there just used to be this um assumption that you you know, one of the things that constrained Street Strip's views of what C had to be was that it had to literally compile C source code.
00:37:46.960 --> 00:37:47.599
But it doesn't.
00:37:47.760 --> 00:37:48.320
Most of the time.
00:37:48.480 --> 00:37:50.000
I mean it does in the basis.
00:37:50.159 --> 00:37:51.280
But these days it doesn't.
00:37:51.440 --> 00:37:52.320
Yeah, these days it doesn't.
00:37:52.480 --> 00:37:54.079
At the time it had to for a while.
00:37:54.239 --> 00:37:54.320
Yeah.
00:37:54.960 --> 00:37:58.800
At least it had to be able to include the headers with the you know, with the right if-def garbage.
00:37:58.960 --> 00:37:59.679
Anyway.
00:38:00.079 --> 00:38:05.679
Um we're we're off in a in a weird programming language rut what when we meant to be talking about H H VM for a second.
00:38:06.320 --> 00:38:07.599
So let's go back to HHVM.
00:38:07.760 --> 00:38:08.880
Which is C.
00:38:09.519 --> 00:38:13.679
So tell us for a second why you chose C because I know Jason is a C guy.
00:38:13.920 --> 00:38:15.199
You are a C guy as well.
00:38:15.440 --> 00:38:15.920
I was a C guy.
00:38:16.239 --> 00:38:16.880
So was it true?
00:38:17.039 --> 00:38:18.480
Uh why do you choose C?
00:38:19.920 --> 00:38:21.199
A little bit multifactorial.
00:38:21.280 --> 00:38:24.079
First of all, Facebook's a C shop at this time, right?
00:38:24.159 --> 00:38:32.800
So the FB code code base, which is a sort of back end static language, you know, has a build step code base to it, uh, was primarily C code base.
00:38:32.960 --> 00:38:35.760
Uh at this point, I'd been working in C for about a year and a half.
00:38:35.840 --> 00:38:40.719
So I my first job at Facebook was working on the search backend, and because it was a C shop, I've been doing this.
00:38:40.880 --> 00:38:42.159
That's where you met Jordan, right?
00:38:42.400 --> 00:38:44.159
Yeah, that was where Jordan and I first crossed paths.
00:38:44.320 --> 00:38:48.079
Jordan DeLong, by the way, is one of our ex-colleagues on HHPM.
00:38:48.400 --> 00:39:00.320
The the other kind of complication here was a big chunk of the HPHP effort was moving all of the PHP extensions into the code base of hip hop, right?
00:39:00.400 --> 00:39:15.920
So PHP is not just a core language, it's also sort of a thing that lets you output PostScript and a thing that lets you parse PDFs and a thing that lets you parse XML and a thing that lets you do read JSON and blah, blah, blah, blah, all these you know, regular expressions, all the stuff that you're sort of used to pip installing in Python land or whatever.
00:39:16.239 --> 00:39:18.480
Isn't it why there has never been a JIT for Python?
00:39:18.639 --> 00:39:19.760
It's the same reason, right?
00:39:19.840 --> 00:39:26.079
Like there's there's so many extensions that make assumptions about you know what's the internal.
00:39:26.480 --> 00:39:28.400
I mean, there have been several JITs for Python, as a matter of fact.
00:39:28.639 --> 00:39:31.360
No, but JITs for Python that actually took off.
00:39:31.679 --> 00:39:32.480
Yeah, they take over the world.
00:39:32.559 --> 00:39:32.639
Yeah.
00:39:32.800 --> 00:39:33.360
Yeah, take over the world.
00:39:33.679 --> 00:39:41.199
That's a big chunk of it, is that like the whatever the runtime you have for that Python JIT has to look basically exactly like the like CPython's runtime or else.
00:39:41.440 --> 00:39:42.480
Um we should try.
00:39:42.960 --> 00:39:47.039
I don't know if you ever talked to Sam Gross about the stuff he's doing about trying to get rid of the lock.
00:39:47.519 --> 00:39:51.519
Yeah, he's trying to get rid well, he did get rid of it, the global locks, right?
00:39:51.920 --> 00:39:52.320
Yeah, yeah.
00:39:52.400 --> 00:39:53.599
It's a it's a going concern.
00:39:53.679 --> 00:39:54.880
You can if def it out now.
00:39:55.199 --> 00:40:03.920
The core of what became HHVM, right, which is the JIT compiler that that we were ultimately building, in some ways is kind of like a C program, right?
00:40:04.000 --> 00:40:12.000
It's organized like around structs and is pretty procedural and is you know passing around references to fairly flat data types and things like that.
00:40:12.239 --> 00:40:18.079
It's not you know deeply, deeply C in the in the ways that were fashionable at the time.
00:40:18.239 --> 00:40:22.079
Um it got sort of more metaprogrammed, especially post-C11.
00:40:22.719 --> 00:40:29.440
Um, but honestly, we're doing a lot of the metaprogramming stuff with C preprocessor in the early days of this, just because that was what I was more comfortable with.
00:40:29.519 --> 00:40:31.920
But anyway, we're getting into implementation details here.
00:40:32.159 --> 00:40:33.039
Why even do this, right?
00:40:33.119 --> 00:40:34.079
Like why why agit?
00:40:34.159 --> 00:40:37.679
Like why would we do better than a ahead of time compiler while that's I know, I know.
00:40:37.840 --> 00:40:41.519
It's because the interpreter was getting too slow.
00:40:41.760 --> 00:40:44.239
So I was taking well, first two things.
00:40:44.480 --> 00:40:46.400
First, the interpreter was getting too slow.
00:40:46.559 --> 00:41:07.280
So the developers, whenever they're trying to make a change on their dev because so so that people know, uh on the HPHP setup, there was the uh the compiler that was taking the PHP source code and producing the C ⁇ , transpiling to C and then compiling that to native.
00:41:07.599 --> 00:41:15.679
This is not what people were running on their sandboxes, because that process was super long and um super super expensive.
00:41:16.000 --> 00:41:18.639
So there was also an interpreter.
00:41:18.960 --> 00:41:22.639
And the problem with the interpreter is that it was really getting too slow.
00:41:22.880 --> 00:41:32.719
Uh and so both so there was both the interpreter that was getting too slow, and the compiler also that was running into some kind of hard limits.
00:41:32.800 --> 00:41:33.840
Was it the linker?
00:41:34.000 --> 00:41:35.280
Was it the binary size?
00:41:35.360 --> 00:41:36.159
What was it?
00:41:36.559 --> 00:41:39.280
There well, so the hard limits were were gonna be fixable.
00:41:39.440 --> 00:41:45.280
So among the hard limits it was running into was that the uh it was outputting a binary that was greater than two gigabytes.
00:41:45.440 --> 00:41:57.519
And we know we were among the first people to try to do that with the GNU linker chain toolchain because we were getting into sort of signed 32-bit integer errors dealing with uh you know text segments basically in the toolchain.
00:41:57.679 --> 00:42:08.480
Um these were just basically mysterious like crashes or even worse, mysterious corruptions of the output binary when you get this big static binary out of this link process.
00:42:08.719 --> 00:42:10.239
We worked through them one at a time.
00:42:10.400 --> 00:42:18.800
That wasn't the thing that sort of was the death knell for HPHP, but it was an example where it's like, okay, we're really starting to abuse this toolchain from the perspective of what was engineered for.
00:42:18.960 --> 00:42:27.679
Um and then on HPHP i, this thing was just sort of supposed to be there for these goofball PHP programmers who, for some reason, can't be bothered to compile things.
00:42:27.760 --> 00:42:28.719
We don't understand why.
00:42:28.800 --> 00:42:30.880
They're they must not be good at computers.
00:42:31.119 --> 00:42:36.239
But they get this crummy interpreter, and the crummy interpreter was an AST traversing interpreter, right?
00:42:36.400 --> 00:42:40.079
Basically had a big syntax tree in memory of the entire code base.
00:42:40.400 --> 00:42:44.159
Um it would incrementally update that syntax tree when you change stuff on disk.
00:42:44.880 --> 00:43:03.519
And you know, the if you can imagine uh the difference between sort of a bytecode interpreted, you know, the it's usually kind of an order of magnitude implementation improvement going from I traverse an AST to I you know run around switching on bytecodes, and then typically maybe another order of magnitude to get from that bytecode interpreter to native code.
00:43:03.840 --> 00:43:08.719
And the first PHP interpreter was reading the file character by character, right?
00:43:09.280 --> 00:43:10.159
Oh, yeah, fun fact.
00:43:10.400 --> 00:43:14.800
That's another order of magnitude, and it was implementing loop with L seq.
00:43:15.679 --> 00:43:16.800
Oh, I didn't know that.
00:43:17.039 --> 00:43:17.599
Yes.
00:43:18.239 --> 00:43:18.400
Wow.
00:43:18.639 --> 00:43:20.000
It was actually So is this a file pointer?
00:43:20.079 --> 00:43:20.960
Like it just was a file pointer?
00:43:21.199 --> 00:43:24.639
Yeah, it was a file pointer that was moving back.
00:43:25.280 --> 00:43:32.079
And I mean, as far as I know, I I I I hope I'm not making stuff up here, but that's somebody who told me this story.
00:43:32.559 --> 00:43:37.039
We should uh we should try and get Rasmus Leredorf on here sometime and get all the way up.
00:43:37.440 --> 00:43:42.719
Rasmus Leredorf, the father of PHP, by the way, which originally was like a very modest little project.
00:43:42.880 --> 00:43:45.599
Uh PHP stood for personal homepage.
00:43:45.840 --> 00:43:52.079
The idea was like it was mostly HTML, but you wanted to like have a little integer in it that you read out of a database or something to be like you're not.
00:43:53.760 --> 00:43:57.679
It wasn't when it was like you're the 1003rd visitor to this page.
00:43:57.840 --> 00:44:05.760
People just wanted to be able to do that, or they just wanted to be able To like do one little database populated field in the middle of a mostly static page or whatever.
00:44:05.920 --> 00:44:11.039
And that was the problem PHP was meant to solve, not some big complicated dynamic web 2.0 thing like that.
00:44:11.360 --> 00:44:16.639
Although for arrays and references, I really would like to know what happened there.
00:44:16.960 --> 00:44:26.079
Because arrays and especially the way they interact with references and how you know it's looking at the reference count to determine if it's going to copy or not.
00:44:26.320 --> 00:44:39.519
That's that's for me is just there's some odd stuff there that like especially because the syntax is pearl-like and because like the culture of doing web stuff was very Perl-derived at the time.
00:44:39.679 --> 00:44:43.280
I suspect, and I don't know if anybody's coped to this exactly.
00:44:43.440 --> 00:44:46.239
I suspect there's a misunderstanding of Perl semantics.
00:44:47.440 --> 00:44:48.960
I have a theory regarding references.
00:44:49.199 --> 00:44:49.519
Please.
00:44:49.840 --> 00:44:56.800
I think the theory goes like this like they introduced value semantics, which by the way is great.
00:44:56.880 --> 00:44:58.000
I like value semantics.
00:44:58.079 --> 00:44:58.559
That's not the problem.
00:44:58.880 --> 00:45:00.239
Well copying things, yeah, one another.
00:45:00.480 --> 00:45:14.480
So so the idea was you pass around an array, if somebody modifies this array, you're actually modifying a copy so that you don't affect the original array, which in it's is very functional in nature, so I I have no problem with that.
00:45:14.639 --> 00:45:17.679
Except so good.
00:45:17.920 --> 00:45:22.719
Except that, well, when you make your copy, what happens when you hit a cycle?
00:45:22.880 --> 00:45:23.440
Right?
00:45:23.679 --> 00:45:33.679
And the moment you hit a cycle, the easiest way to handle this cycle is just to look at the reference count and say, you know what, this could be a cycle, I'm just not gonna copy this.
00:45:33.920 --> 00:45:40.239
And and that's my theory, that you know, managing cycles would have been too hard and too expensive.
00:45:40.320 --> 00:45:49.760
And so the easiest way to get a copy and write that you know worked at reasonably well quickly was to just look at the reference count.
00:45:50.159 --> 00:45:54.400
Okay, yeah, we're definitely like in in PHP nerd corner here, just super briefly.
00:45:54.639 --> 00:45:59.840
It is a the way that PHP implements automatic memory management is with reference counting.
00:45:59.920 --> 00:46:02.480
Like when the count on an object goes to zero, it reclaims it.
00:46:02.639 --> 00:46:04.000
That's not such a terrible strategy.
00:46:04.159 --> 00:46:05.679
Lots of perfectly respectable languages do this.
00:46:06.159 --> 00:46:08.960
It's part of the semantics in PHP, so that's a bit different.
00:46:09.360 --> 00:46:16.800
The thing, the the thing we're referring to here is that it's programmer visible, that that's how it implements it uh in the way that Julian's referring to.
00:46:16.880 --> 00:46:19.599
And that's uh that is unusual.
00:46:19.679 --> 00:46:27.519
And it does, and when when you get down to trying to do a high performance implementation like we were doing in HHVM, it is a pretty heavy constraint to work around.
00:46:27.599 --> 00:46:32.639
Um and is one of the reasons that it'll probably never be competitive with you know Java or something like that, no matter how clever.
00:46:33.360 --> 00:46:34.079
Like arrays.
00:46:34.159 --> 00:46:42.719
I mean, for me, arrays was the the the harder stuff for the type checker, for all the problems were with arrays and references.
00:46:43.360 --> 00:46:46.559
A lot of it is actually is actually okay.
00:46:46.880 --> 00:47:01.920
Like, you know, the it's object model there are a few weirdness in it, like protected does not apply to your children, but it applies to the whole tree because you know there's no object in PHP.
00:47:02.639 --> 00:47:09.519
And so normally protected means it's something that anybody uh underneath you can, you know, redefine.
00:47:10.079 --> 00:47:11.599
That's not the case in PHP, right?
00:47:11.679 --> 00:47:21.760
In PHP protected means underneath and on the side, because in fact you don't have a tree, you have a forest because there is no object type.
00:47:22.480 --> 00:47:31.119
And so you are so this is weird, but and and also checking for visibility at runtime is weird.
00:47:31.360 --> 00:47:36.960
You know, that's that's so they had some weird things, but frankly nothing too unreasonable.
00:47:37.280 --> 00:47:50.000
But arrays and references, which is the bread and butter of you know the language, that was like just very, very difficult to to type check and get that that get that in a good place, unfortunately.
00:47:50.480 --> 00:47:54.800
One reason for why we thought a JIT was necessary was development environment.
00:47:54.960 --> 00:47:59.760
I think another uh you know, another implicit desire here was unified the two implementations, right?
00:47:59.840 --> 00:48:05.760
There was sort of a uh class of bugs which were the interpreter behaves subtly differently from the compiler.
00:48:05.920 --> 00:48:11.280
Um some of this was, for example, degrees of freedom that C has that you didn't have in the interpreter.
00:48:11.519 --> 00:48:23.519
So uh if we're relying on the C toolchain to do things like uh evaluating arguments, the order that you that you evaluate arguments to a C function call, that's actually up to the compiler how it does it, which people don't expect, by the way.
00:48:23.679 --> 00:48:25.440
Like most C programmers implicitly.
00:48:25.679 --> 00:48:27.920
You guys were running in sandboxes for a while, right?
00:48:28.079 --> 00:48:29.760
Um before going to production.
00:48:29.840 --> 00:48:30.000
Yeah.
00:48:30.320 --> 00:48:30.639
That's true.
00:48:30.719 --> 00:48:37.519
Yeah, we got to be in the development environments for a while before we hit prod, probably almost a year, actually, now that I think about it, before we hit prod.
00:48:38.079 --> 00:48:44.320
Um and so we wanted to make it so that you'd run into the same bugs running in production that you're running in your sandbox.
00:48:44.480 --> 00:48:54.880
That also influenced a little bit of sort of the at least the early days philosophy of HHVM, which was like instead of having, you know, one of the ways to sort of get a promotion at Google is to like make a new gear for V8.
00:48:55.199 --> 00:48:55.519
Right?
00:48:55.599 --> 00:48:58.159
Like so V8's got you know 27 modes now, right?
00:48:58.239 --> 00:49:02.159
Because everybody who makes a new mode for V8 gets promoted to senior staff engineer.
00:49:02.880 --> 00:49:07.679
But it means that the landscape of sort of V8's behavior is actually kind of jagged, right?
00:49:07.760 --> 00:49:13.920
And if you're an expert JavaScript programmer and you care a lot about how your code performs, it can be difficult to reason about how it's going to handle it.
00:49:14.159 --> 00:49:23.679
One of the things that C programmers, C pre C programmers, and and good enough C programmers like about their tool chain is they feel like they're able to reason about the compiler's behavior.
00:49:24.079 --> 00:49:26.239
I mean, C developers come on.
00:49:26.480 --> 00:49:28.960
There's only a handful of them who are capable of that.
00:49:29.199 --> 00:49:39.840
Like even at Facebook, even you know, in in those kind of shops with top notch notch engineers, the people who actually know what C does, a few.
00:49:40.400 --> 00:49:48.800
You don't think they wouldn't be able to kind of like hand expand all the templates and everything and kind of be like here's what this loop that reversed a string really would look like.
00:49:48.880 --> 00:49:49.920
You don't think they could do that?
00:49:50.800 --> 00:49:51.519
I don't know.
00:49:52.960 --> 00:49:54.320
Maybe that's an interesting question.
00:49:54.639 --> 00:49:55.280
I think it's possible.
00:49:55.519 --> 00:49:57.280
And maybe maybe this is like a sign of the times.
00:49:57.360 --> 00:50:04.880
Like I came up in an era in the 90s, like they still made you take courses where you built little toy compilers and where you wrote a machine code and stuff like that.
00:50:05.360 --> 00:50:09.440
But no, they do that now, but they just vibe code the compiler is completely possible.
00:50:09.679 --> 00:50:10.639
Well, yeah, duche.
00:50:11.199 --> 00:50:12.079
It just works.
00:50:12.400 --> 00:50:13.119
Yeah, yeah.
00:50:13.440 --> 00:50:17.920
The other thing that this is the reason to sort of HHVM is is what you know JIT people would call a baseline JIT.
00:50:18.079 --> 00:50:23.440
It didn't have it basically was like just trying to do an okay job turning your PHP into machine code.
00:50:23.679 --> 00:50:32.960
It wasn't 15 different layers of optimizers on top of that that were going to you know optimize, then de-optimize, then re-optimize, then inline again, then re-optimize a third time, and blah, blah, blah.
00:50:33.280 --> 00:50:37.199
And you had to fight a lot of people who wanted to build things like that.
00:50:37.360 --> 00:50:37.599
Right.
00:50:37.760 --> 00:50:40.960
I remember back then there was competing projects.
00:50:41.119 --> 00:50:46.719
There was a team that was attempting to compile to the JVM, if I remember well.
00:50:47.039 --> 00:50:55.360
And you also had people within the company who wanted to for you to target the JavaScript V8, because V8 was all the rage back then.
00:50:55.519 --> 00:50:57.760
Now there's you know a little bit more dynamic.
00:50:58.239 --> 00:51:05.199
The actual hackathon project I worked on that was sort of the the dawn of HHVM with Jason Evans and Drew Poroski was actually V8 targeted.
00:51:05.280 --> 00:51:07.360
Like we were we were jitting to JavaScript.
00:51:07.519 --> 00:51:16.800
We were sort of translating PHP to JavaScript and then running it through v8 and kind of got the the programming language shootout benchmarks or some subset of them working a little bit and said, ah, look how blazing fast this is.
00:51:17.039 --> 00:51:19.039
But yeah, v8 was an inspiration at the time.
00:51:19.199 --> 00:51:23.280
I think I also was coming from from VMware, which was a machine-level JIT, right?
00:51:23.440 --> 00:51:33.519
The the software-only version of VMware took unsafe x86, spit out safe x86, and that was how we were able to run Windows on Linux and Linux on Linux and FreeBSD on Linux and so on.
00:51:33.840 --> 00:51:39.920
We didn't try to make the code faster, it just tried to sort of preserve reasonably linear slowdowns.
00:51:40.320 --> 00:51:45.360
And where possible, zero slowdowns, when there had to be a slowdown, make it a constant factor.
00:51:45.760 --> 00:51:53.760
Um and as long as we did that, you know, we knew we'd be at least able to sort of boot and run Windows in an acceptable fashion, and that was what our customers needed.
00:51:54.079 --> 00:52:18.239
Dude, I remember once, sorry to interrupt, but we were at Epic Cafe, we were having lunch, and um you were ranting about um, you know, people who wanted to run some kind of optimization on HHVM, and your whole approach was we don't need to, we just have to translate the bytecode into you know native code, and it's gonna be good enough.
00:52:18.559 --> 00:52:30.079
And and I'm with my bagel in my mouth, and you're like, Julian, there's no amount of money you can give to a compiler person not to implement those optimizations.
00:52:30.239 --> 00:52:36.719
There's just no no amount of money I can give them to say no GCE, no people, it's not happening.
00:52:37.119 --> 00:52:38.800
You're not doing this.
00:52:41.199 --> 00:52:42.960
And so it's an unnatural act for them, right?
00:52:43.039 --> 00:52:48.800
Like it's like you know, they're they've got this enormous bag of like awesome tricks to make to make code fast.
00:52:48.880 --> 00:52:52.159
Like, why would you just not make the code fast when you can make the code fast?
00:52:52.719 --> 00:53:01.760
The problem with running PHP wasn't, you know, that we failed to strength reduce multiplying by five into shift two and add a copy, right?
00:53:01.920 --> 00:53:06.239
The problem with like your PHP not being fast was that like it didn't even know it was a freaking integer.
00:53:06.400 --> 00:53:06.880
You know what I mean?
00:53:07.199 --> 00:53:11.440
The problem with not being able to run PHP fast was like you're sitting there being like, is it a string?
00:53:11.599 --> 00:53:12.400
Is it an array?
00:53:12.880 --> 00:53:16.400
Around this time we were lucky to have a little bit of literature support here too.
00:53:16.480 --> 00:53:32.880
There was uh the the term the repurpose JIT phenomenon kind of entered the programming language lexicon with a publication around then, which is basically when you are building a just-in-time compiler for a language, you're implicitly making lots of decisions about what to make fast and what to make slow.
00:53:33.039 --> 00:53:39.280
And those decisions are going to be basically tuned to the characteristics of the language that you're supporting.
00:53:39.440 --> 00:53:45.119
So, for example, we talked before about the fact that PHP uses reference counting and that it exposes that to the programmer.
00:53:45.280 --> 00:53:56.239
If you want to piggyback on something like the Java virtual machine, well, the Java virtual machine's got a tracing collector, and it's you know, incredible thousands and thousands of man hours have been poured into making that tracing collector amazing.
00:53:56.400 --> 00:54:02.559
But no matter how amazing it is, you're gonna be basically paying to garbage collect twice by running reference counting on top of this tracing collector.
00:54:02.800 --> 00:54:09.119
So it's really, really hard to kind of make up a gigantic headwind like that out of the gate, no matter how smart your compiler is.
00:54:09.360 --> 00:54:12.800
So trying to recycle another JIT, we thought was going to be a loser.
00:54:12.960 --> 00:54:22.079
We thought PHP deserved its own first class thing, the same way JavaScript had V8, uh, the same way VMware had to make its own thing for the x86, because the x86 doesn't look like anything else.
00:54:22.320 --> 00:54:24.639
I thought PHP didn't look like anything else either.
00:54:24.880 --> 00:54:30.320
And if you've ever worked on one of these systems, by the way, like this was true in the early days of VMware as well.
00:54:30.559 --> 00:54:36.000
There's this terrifying thing where for sort of, you know, 18 months to two years or something, just nothing works.
00:54:36.320 --> 00:54:41.440
And your professional life is full of people saying, KMA is an idiot.
00:54:41.599 --> 00:54:47.360
This thing he's saying is foolhardy, it's a big, uh, you know, it's a big hole in the ground that we're pouring money into, and it won't work.
00:54:47.519 --> 00:54:53.679
And they have a lot of evidence on their side because they basically have six quarters in a row of me saying lots of stuff that sounds good, and what works?
00:54:53.840 --> 00:54:54.239
Nothing.
00:54:54.400 --> 00:54:57.599
All right, the site doesn't work, people aren't able to use it, and so on.
00:54:57.840 --> 00:55:08.159
So we were really happy when we got to the point where the site runs, uh, because at least we can get a little bit of win here, where at least it's faster than the old school interpreter, and at least we're getting a little bit of benefit from it.
00:55:08.400 --> 00:55:17.840
Um, however, one thing that was concerning is that the the day that we first were able to run the site, remember, the whole point of this project is that we want it to run fast.
00:55:18.239 --> 00:55:29.920
And when you compared the JITS version of the site to the ahead of time compiled version of the site, you know, the approach that we thought was inferior and that we thought we'd be able to blow the doors off of, we were something like 10x behind.
00:55:30.079 --> 00:55:37.039
It was more than that, I think, but like 12, 14x, but like a discouraging distance, like an ocean to swim across in terms of performance.
00:55:37.360 --> 00:55:39.519
And you know, you find a couple of low-hanging fruits, right?
00:55:39.599 --> 00:55:44.239
You profile a little bit, you find a few things right out of the gate, and you get that down pretty quick.
00:55:44.559 --> 00:55:46.400
But it's like we got it down 2x.
00:55:46.480 --> 00:55:49.760
So now it's 7x behind, which is still really, really far.
00:55:50.079 --> 00:55:52.239
And the profile looks really flat, right?
00:55:52.320 --> 00:55:54.159
There's not one place where it's all going.
00:55:54.639 --> 00:55:58.159
There was like a week or two where I was basically just like, well, shit.
00:55:58.400 --> 00:56:03.519
You know, I may have made a really major mistake with my career and the career of the people who bet on me here.
00:56:03.760 --> 00:56:14.719
Um, but what we eventually ended up doing was uh was just kind of taking a, in hindsight, what I'd now think of as sort of a machine learning approach, which is like, no, the fundamentals are still on our side, right?
00:56:14.800 --> 00:56:16.079
The truth is out there somewhere.
00:56:16.239 --> 00:56:19.280
There is some way better JIT to build than this one.
00:56:19.519 --> 00:56:23.360
We just don't know quite how to get there, and it's not going to be one step to get there from here.
00:56:23.679 --> 00:56:33.519
So uh we got the team together, got some pizza, got some beer, and spent a whole bunch of hours just kind of covering a wall with post-it notes of optimization ideas.
00:56:34.159 --> 00:56:38.880
And at this point, the thing was kind of mature enough that most optimization ideas actually didn't pan out.
00:56:39.039 --> 00:56:45.440
And it wasn't because they were bad ideas, it was just because like computers are frustrating and hard because performance is elusive.
00:56:45.599 --> 00:56:48.239
There are many ways to be slow and few ways to be fast.
00:56:48.960 --> 00:57:02.239
Um and what the the mental transformation I've learned you have to make in these environments where you're past the easy part of doing performance work is you have to view the work as just pulling the post-it notes off the wall.
00:57:02.320 --> 00:57:02.559
Right?
00:57:02.639 --> 00:57:05.840
The work is just I took this thing and I tried it.
00:57:06.079 --> 00:57:07.039
And maybe it worked.
00:57:07.199 --> 00:57:07.679
Awesome.
00:57:07.840 --> 00:57:10.000
We're closer to the goal, but maybe it didn't work.
00:57:10.159 --> 00:57:10.960
Who cares?
00:57:11.119 --> 00:57:17.119
There's few there's fewer things on the wall and we know more now, even if all we know is is the things that won't work.
00:57:17.360 --> 00:57:30.960
Um so the mental shift of like I now get the dopamine hit when uh I take a note off the wall versus I get the dopamine hit when the thing goes faster is a really important transformation to go through in one of these things, right?
00:57:31.039 --> 00:57:34.639
If you start to view it as like, this is just a search process, it's out there somewhere.
00:57:34.800 --> 00:57:38.559
We will eventually find the right veins to mine, even though I don't know what they are right now.
00:57:38.800 --> 00:57:44.639
And, you know, it can be a great day at the office if we didn't make anything faster, as long as we pulled 10 notes off the wall today.
00:57:44.880 --> 00:57:51.280
That's still a lot that we know today that we didn't know yesterday, even if all we know is about things that didn't work instead of things that do work.
00:57:51.519 --> 00:57:54.159
That's still gonna mean we put better posted notes up tomorrow.
00:57:54.320 --> 00:58:00.480
That's still gonna mean that the ones that we haven't taken down yet are more likely to be where the juice is.
00:58:00.719 --> 00:58:06.079
And then once, you know, once you started getting rolling on this process, it kind of took on a life of its own, right?
00:58:06.159 --> 00:58:12.719
Like it kind of you get into this rhythm where you just show up every day and that's what you do, and it starts to be kind of habituating, right?
00:58:12.880 --> 00:58:21.280
Once we got through from 7X, uh, you know, Jordan DeLong actually found something really early on that was a gigantic win, like a weird 14% win.
00:58:21.519 --> 00:58:27.280
Um, it changed the way that we thought about this because there was nothing in the profile that was 14%, right?
00:58:27.360 --> 00:58:30.719
The the part of the system that Jordan optimized wasn't taking 14%.
00:58:31.280 --> 00:58:33.119
So how did it make it 14% faster?
00:58:33.280 --> 00:58:42.000
Well, it turns out caches explain a lot of weird mysteries that Andal's law will hide from you if you're thinking in terms of, oh, we have to spend all the time on the thing that's the top of the profile.
00:58:42.320 --> 00:58:48.079
Caches are this, you know, mysterious medium that can cause this part of your program to slow down that part of your program.
00:58:48.239 --> 00:58:49.360
And it was a really important clue.
00:58:49.440 --> 00:58:54.480
And we started sort of digging, you know, pulling on those threads more actively once we once we had that clue.
00:58:54.719 --> 00:59:03.280
Um and you know, once we got through parity, once we got to, you know, 100% of the performance of the of the production system, we still had a ton of momentum.
00:59:03.360 --> 00:59:05.599
And then the week after that, it was sort of we were 10% faster.
00:59:05.760 --> 00:59:07.519
The week after that, we were 20% faster.
00:59:07.760 --> 00:59:12.000
And pretty soon, uh, you know, Zuck was in our pod ringing the gong to turn it on in production.
00:59:12.400 --> 00:59:12.880
I remember that.
00:59:13.199 --> 00:59:20.159
So I don't know if you remember this story, but this was maybe uh a few weeks before you guys were going into production.
00:59:20.400 --> 00:59:25.920
And back then, I was in your team meetings because hack had been merged with HHVM.
00:59:26.079 --> 00:59:32.480
So Hack was before a team of its own, and then uh Pobar was actually the one who pulled it in.
00:59:32.639 --> 00:59:38.239
So you know, Pobar became the manager of Hack, and that's what Joel Pobar, by the way.
00:59:38.719 --> 00:59:42.960
Joel Pobar, who's now I mean, uh he's back at Facebook now, right?
00:59:43.039 --> 00:59:48.239
Like he's been brought back to run AI stuff, and I mean, great guy.
00:59:48.400 --> 00:59:51.599
He wasn't like, exactly.
00:59:52.159 --> 00:59:56.559
I don't I don't know if it was that briefly, but I mean I think he's back at Meta now.
00:59:56.880 --> 01:00:01.280
I remember you guys are churning to get to get to poverty, right?
01:00:01.519 --> 01:00:08.079
And from the last meeting, I know that there's just 8% missing to get to poverty.
01:00:08.239 --> 01:00:12.559
And I show up at your pods, and there's you, there's Jason, there's there's a whole crew.
01:00:12.719 --> 01:00:30.960
I'm like, hey Keith, so you know, I was working on a task on dub dub dub recently, I think, you know, some minor stuff, and you know, I I I I noticed something strange that you know I feel like my code could be a little bit faster, like eight percent faster.
01:00:31.119 --> 01:00:32.880
Do you think you could do that for me?
01:00:34.239 --> 01:00:40.400
The whole team was like looking at me like like shut the fuck up, frame of smoke, yeah, help on us, right?
01:00:40.880 --> 01:00:49.840
With smoke coming out of the yes, really, you're doing this, and and again, I mean, uh HP HP was still optimized back then, right?
01:00:49.920 --> 01:00:54.239
There was still growing in there drag racing uh moving target at that time.
01:00:54.800 --> 01:00:55.199
Exactly.
01:00:55.360 --> 01:01:02.880
So it was 8%, but it could have been could have been 12 the next day if there was because I I mean you kept it fair the whole time.
01:01:03.039 --> 01:01:10.400
You never there was never any push to prevent people from optimizing HPHP.
01:01:10.480 --> 01:01:14.639
And I think that's great because this was fair game the whole time.
01:01:15.199 --> 01:01:21.039
You never said, all right, you people who are still optimizing the old system, stop doing what you're doing.
01:01:21.199 --> 01:01:22.079
It's not your system.
01:01:25.599 --> 01:01:25.679
Right.
01:01:25.920 --> 01:01:28.159
So it got a lot, lot harder to compete with.
01:01:28.400 --> 01:01:29.599
Yeah, yeah, exactly.
01:01:29.760 --> 01:01:40.159
And so so then came the gong, uh, you know, when you know you were you uh Facebook um H H VM got got into production.
01:01:40.480 --> 01:01:42.000
And then what happens after that?
01:01:42.159 --> 01:01:53.840
Then you then you you're kind of knighted by by Zuck, and then and then people just bow as they see you, you know, going through going through the the corridors.
01:01:54.000 --> 01:01:55.280
What's what's going on after that?
01:01:55.679 --> 01:01:58.239
I mean you you heave a big sigh of relief, right?
01:01:58.320 --> 01:02:01.119
Uh me and Joel you know went shopping for a cake.
01:02:01.519 --> 01:02:08.079
We we uh had a little party for the team, you know, because everybody uh did pull pretty long hours to get this over the line.
01:02:08.400 --> 01:02:11.039
And uh then after that was kind of like back to work, right?
01:02:11.119 --> 01:02:14.480
So there was still, you know, there's still a bunch of post-it notes on the wall after that, actually, right?
01:02:14.559 --> 01:02:16.400
So you said, well, okay, we still got all this roadmap.
01:02:16.559 --> 01:02:28.159
And he kept kind of just burning that roadmap because you know, every 10% of CPU you save is billions of dollars of of you know essentially profit margin for the entity Facebook at that time.
01:02:28.400 --> 01:02:32.559
To be clear, there's like a bunch of tooling that we relied on that other people built that was amazing, right?
01:02:32.639 --> 01:02:41.840
I'm thinking right now Perf Lab, um, for instance, which like had a bunch of really interesting tricks and a lot of pretty sophisticated statistics to make the there's a lot of cool stuff that came out of Facebook.
01:02:41.920 --> 01:02:47.440
I mean, React came out of Facebook at this time, GraphQL came out of Facebook at this time.
01:02:47.679 --> 01:02:56.960
It feels like those early 2010s for me were like the best you know engineering time of my life.
01:02:57.119 --> 01:03:07.360
Like it seemed like a bunch of really smart people who just wanted to get shit done and who were moving super fast and building building a ton of interesting stuff.
01:03:07.440 --> 01:03:20.320
And wherever I go now, I'm trying to rebuild that vibe, you know, re-re-get this vibe of Facebook 2010, Facebook 2011, 2012, around there.
01:03:20.559 --> 01:03:26.800
That's really the time that for me was the peak of you know, interesting people all over the place.
01:03:27.039 --> 01:03:32.320
You wanted to know anything about the kernel, you had a bunch of kernel experts, the best in the world.
01:03:32.559 --> 01:03:41.039
You wanted to know something about programming languages, you had great we we hired Simon Marlowe, we hired, you know, and and there was interesting.
01:03:41.599 --> 01:03:45.440
The Haskell community left and right, yeah, the Haskell compiler guy.
01:03:45.760 --> 01:03:48.960
Um and yeah, it was it was a great time.
01:03:49.280 --> 01:04:03.519
There was something sort of in the air in hindsight where there was a lot of fearlessness, a lot of willingness to try new things, a lot of conviction that um you could kind of blow up the way that we've been doing things and and do it in a massively better way, and a lot of positive examples of people actually achieving that.
01:04:03.920 --> 01:04:04.880
Yeah, exactly.
01:04:05.119 --> 01:04:15.119
I think it's a rare thing when you work in a company where people are convinced that they can build uh the next big thing.
01:04:15.440 --> 01:04:15.760
You know?
01:04:15.920 --> 01:04:16.079
Yeah.
01:04:16.239 --> 01:04:25.039
Uh in most companies I work for uh before Facebook, um, they always believe that, you know, whatever the other guys have been building is better.
01:04:25.199 --> 01:04:25.519
Right.
01:04:25.679 --> 01:04:27.840
And and Facebook was not like that.
01:04:28.000 --> 01:04:35.199
They thought, no, we we can we can build something better, and if we just put enough time and enough energy into it, we can do it.
01:04:36.079 --> 01:04:38.320
All right, everyone, thank you for listening.
01:04:38.800 --> 01:04:41.199
It was it was really great to spend this time with you guys.
01:04:41.360 --> 01:04:42.239
Thanks, Keith.
01:04:42.639 --> 01:04:43.519
Likewise, William.
01:04:43.599 --> 01:04:44.320
Thank you.