WEBVTT
00:00:08.329 --> 00:00:15.048
Hello everyone And welcome to the switch statement podcast It's a podcast for investigations into miscellaneous tech topics
00:00:15.976 --> 00:00:19.045
This is episode five of The Coming Wave by Mustafa Suleiman
00:00:20.106 --> 00:00:21.275
We'd love to hear from you.
00:00:21.486 --> 00:00:25.356
Send your comments and feedback to team@tkp.co.
00:00:31.989 --> 00:00:32.908
Hey, Don.
00:00:33.139 --> 00:00:33.488
What's going on?
00:00:35.329 --> 00:00:35.859
Hey, Matt.
00:00:35.878 --> 00:00:37.308
Uh, let's see.
00:00:37.319 --> 00:00:38.789
Nothing much is going on?
00:00:38.908 --> 00:00:42.719
Just kind of doing the things on a daily basis.
00:00:42.719 --> 00:00:43.548
What's going on with you?
00:00:44.304 --> 00:00:46.634
Yeah, not, not too much.
00:00:46.713 --> 00:00:54.113
Just, um, just kind of wondering how many, how many agentic angels can dance on the head of a pin.
00:00:54.853 --> 00:00:55.073
know?
00:00:55.084 --> 00:00:57.774
I think that's one of the more important questions of our time.
00:00:58.484 --> 00:00:58.753
Yeah.
00:00:58.753 --> 00:01:01.973
Is this like before you can get into heaven or something?
00:01:02.933 --> 00:01:03.993
What's the rest of that quote?
00:01:05.927 --> 00:01:09.507
Oh, I don't know, I didn't know there was another part to that quote
00:01:10.373 --> 00:01:11.463
Oh, isn't it something...
00:01:11.493 --> 00:01:18.013
Or maybe this is, maybe, am I thinking of the camel going through the head of a needle or the hole in a needle?
00:01:18.078 --> 00:01:19.518
Oh, yes.
00:01:19.587 --> 00:01:20.177
Yeah.
00:01:20.237 --> 00:01:22.408
You need to put the camel through the eye of a needle?
00:01:22.418 --> 00:01:23.108
in order to...
00:01:23.837 --> 00:01:24.138
Or is
00:01:24.180 --> 00:01:32.010
It's easier for, in, in, it, it's easier for a camel to go through the eye of a needle than for a rich person to get to heaven, or something like that
00:01:32.531 --> 00:01:35.240
I think you've nailed it And if
00:01:35.307 --> 00:01:35.576
So
00:01:35.581 --> 00:01:37.771
we gotta just, let's just change the Bible
00:01:38.602 --> 00:01:45.242
But angels dancing on the head of a pin, it, it still relates to sewing, but, uh,
00:01:46.114 --> 00:01:46.504
Yeah.
00:01:46.504 --> 00:01:48.284
What are all these philosophers?
00:01:48.314 --> 00:01:50.575
They're really concerned with, with pins.
00:01:50.864 --> 00:02:02.405
Um, but anyway, but I think, this is the sort of, navel-gazing that, Mustafa Suleiman doesn't have a lot of time for, I feel like, as, as evidenced by the end of this chapter.
00:02:02.405 --> 00:02:08.694
And maybe it makes sense for us to, to, briefly take a step back and orient ourselves.
00:02:09.164 --> 00:02:11.935
So we're in-- We're talking about the technology of intelligence.
00:02:12.329 --> 00:02:12.919
Yes
00:02:13.070 --> 00:02:14.901
laying the, laying the foundations.
00:02:15.465 --> 00:02:15.805
Yep.
00:02:15.875 --> 00:02:20.735
Yeah, talked a lot about sort of the, the evolution of neural networks.
00:02:20.794 --> 00:02:35.582
Talked about a lot of his work initially in DeepMind, AlphaGo, you know, these amazing moments where neural networks were beating humans at these very, deeply, creative and intelligence tasks.
00:02:36.342 --> 00:02:37.272
Move 37,
00:02:37.712 --> 00:02:38.532
move 37.
00:02:38.582 --> 00:02:38.992
for a loop
00:02:39.733 --> 00:02:40.383
Exactly.
00:02:40.763 --> 00:02:41.562
Lee Sedol.
00:02:41.573 --> 00:02:50.296
Although I'll say again, he did win one game, which I'm more impressed by that than than by, you know, him getting beaten or whatever.
00:02:50.610 --> 00:02:54.550
but anyway, so he's, he's sort of getting into the territory of large language models.
00:02:54.580 --> 00:02:55.621
This is where we left off.
00:02:55.621 --> 00:03:01.051
We briefly talked about how, this large language model progress has become ridiculous.
00:03:01.150 --> 00:03:10.468
models of today are just absolutely destroy the models of a year ago at all these intelligence tasks, and the scale of them has also gotten ridiculous.
00:03:10.587 --> 00:03:21.649
he mentioned in this book, 10 billion petaFLOPS was what it takes to train a model, and this was at the time of writing this book, you know, several years ago.
00:03:21.649 --> 00:03:25.028
So these models pr- have probably grown in scale since then.
00:03:25.451 --> 00:03:42.524
And actually, We're reaching the point in AI development where governments are being like:"Whoa, we need you to slow down." You know, we talked about containment in, you know, two episodes ago now, and the American government is trying its hand.
00:03:42.713 --> 00:03:44.514
It's trying to restrain Fable.
00:03:44.783 --> 00:03:46.743
Like, I don't think we, I don't think we had...
00:03:46.804 --> 00:03:52.186
I'm not sure if we discussed Fable, but, uh, it has come out and has since been...
00:03:52.503 --> 00:04:02.985
I guess the government, the United States government kind of politely asked, or maybe not so politely asked Anthropic to, like, stop giving access t- to its new model, Fable.
00:04:03.466 --> 00:04:05.665
And, uh, we'll have to see.
00:04:05.665 --> 00:04:17.105
But, but anyway, so just how that ties back is like there-- these, these unbelievably enormous models and, and constantly, you know, just ever-expanding capabilities,
00:04:17.487 --> 00:04:27.071
Yeah, and I mean, ostensibly the reason the government asked Anthropic to sort of pull back Fable is because of the risk to the software world.
00:04:27.170 --> 00:04:45.826
You know, Fable, when Anthropic was first releasing its Mythos class of models, of which Fable is one, they talked about how it presents a risk to security of pretty much any software-based system, because I guess these Mythos class models were so good at finding security flaws in code.
00:04:46.536 --> 00:04:56.463
S- so with Fable, you know, bad actors could sort of scan, web applications or scan any application really, and try to find, uh, vulnerabilities.
00:04:57.223 --> 00:05:00.612
And so, so the government kind of freaked out.
00:05:00.773 --> 00:05:04.642
I, I think, I do think there's also a political element to this.
00:05:04.673 --> 00:05:22.286
Uh, the Anthropic, you know, sort of famously did not wanna be involved in any United States military use because the US would not guarantee to them that they wouldn't use models to autonomously kill people, and they also did not wanna be involved in surveilling.
00:05:22.670 --> 00:05:33.199
And so I feel like the relationship between Anthropic and the US government is kind of rough, whereas OpenAI stepped in and was like,"We'll do those things." And
00:05:33.346 --> 00:05:33.795
Yeah.
00:05:33.995 --> 00:05:35.255
so, opportunistic
00:05:36.259 --> 00:05:39.449
so anyway, I feel like there's sort of a, a political...
00:05:39.990 --> 00:05:47.639
I don't know, just a s- uh, maybe I'm just being pessimistic, but I feel like there's this political motivation to the Fable pullback.
00:05:47.660 --> 00:05:51.170
But anyway, that's besides the point, sort of outside of the chapter.
00:05:52.908 --> 00:06:03.437
Um, but yeah, so, uh, I mean, we don't-- I don't know if we know exactly how many parameters Fable is, but it's-- I'm sure it's just another point on this insane exponential.
00:06:03.848 --> 00:06:08.077
Um, and but it's, it's funny because...
00:06:08.218 --> 00:06:34.468
A- and they talk about this in this chapter, you know, and I think we had teased this, this more with less again, where two things are combining to create this, these outsized, uh, impacts on society is like, yes, the m- the number of parameters is shooting up, but also the performance per parameter, and I'm not sure if people are talking about this, but like that's also going up.
00:06:34.737 --> 00:06:47.697
So like you're able to train a larger model, but also it's able to do even more per parameter than the old models were, and you get this insane, insane, uh, kind of exponential, like super exponential essentially.
00:06:48.074 --> 00:06:48.413
Yeah.
00:06:48.454 --> 00:06:49.163
Yeah, exactly.
00:06:49.163 --> 00:07:04.043
I think we briefly touched on DeepSeek, which was sort of this example of, like, they achieved the same results as frontier models except with a fraction of the training cost and also a fraction of the inference cost, which, yeah, like you're saying, not a lot of people talk about this.
00:07:04.233 --> 00:07:15.314
There's a lot of research that shows that when these super, super large models are used, like when inference is calculated against them, which is basically where...
00:07:15.343 --> 00:07:26.524
literally where you're generating the next tokens, that's kind of what inference is, th- there's only actually a fraction of the nodes in the model that get used.
00:07:27.074 --> 00:07:34.233
You can think of a model as this vast array of trillions of nodes, and they're all interconnected by these edges.
00:07:34.994 --> 00:07:45.944
And so if only a fraction of those nodes are lighting up, you can just sort of take those nodes, copy them to a much, much smaller model, and achieve similar performance
00:07:47.091 --> 00:07:57.822
Yeah, and this is, I think you're calling out this mixture of experts architecture where like y- okay, you have-- you train your big model and then it's like, okay, well actually there's these...
00:07:57.882 --> 00:08:01.172
And I don't know, I'm not sure if it's different.
00:08:01.182 --> 00:08:07.372
Like, do you train a big model and then you're able to carve out these smaller inference models?
00:08:07.391 --> 00:08:11.802
Like, uh, you know, by seeing the, you know, the performance on various queries?
00:08:12.062 --> 00:08:17.471
Or are you actually training these specialized sub-models from the beginning?
00:08:19.098 --> 00:08:20.137
That I do not know.
00:08:20.177 --> 00:08:21.247
That's a great question.
00:08:23.572 --> 00:08:23.661
Yeah.
00:08:23.851 --> 00:08:32.292
My sense would be you actually, you actually do do back propagate, like you include the model routing in the back propagation, like when you're doing the training.
00:08:32.432 --> 00:08:41.302
'Cause it seems like it would be incredibly difficult to like, you take this, you take these weights, and you just like chop like big, large chunks out of it
00:08:41.628 --> 00:08:42.128
Yeah.
00:08:42.138 --> 00:08:45.278
I- intuitively that makes sense to me.
00:08:45.278 --> 00:08:48.768
It also seems like it creates nice loose coupling.
00:08:49.227 --> 00:08:52.207
It's like you have these i- individual tiny models that are...
00:08:52.238 --> 00:09:01.937
Well, tiny is a relative term here, but they're being trained on very specific things, and then you have this sort of routing model that, just like you're saying, is sort of part of the training
00:09:03.826 --> 00:09:08.716
One thing that's kind of interesting about the, like, chop it out is you could, you could wind up...
00:09:09.436 --> 00:09:22.025
Like, one of, one of the things I feel like you might lose with a mixture of experts is, like, maybe there's-- maybe another domain could be, like, useful in...
00:09:22.446 --> 00:09:26.505
You know, it's like, okay, something about music is useful when solving a math problem.
00:09:26.505 --> 00:09:34.765
It's like how many ti- I feel like that's r- where true genius kind of lives, is the incorporation from another domain.
00:09:35.566 --> 00:09:36.135
Um,
00:09:36.371 --> 00:09:41.572
I guess I don't know enough about mixture of experts models, but I don't know how much they suffer from that sort of...
00:09:41.861 --> 00:09:54.761
'Cause I feel like this is one of the promises of large language models is like it's able to have these cross-disciplinary thoughts that are original because they're combining expertise from, you know, multiple things.
00:09:55.581 --> 00:10:01.361
And so yeah, I do wonder if mixture of experts sort of suffers from an inability to do that.
00:10:07.639 --> 00:10:17.129
the book is kinda interspersed with almost biographical or I guess autobiographical stuff about Suleyman's life, which is kinda cool.
00:10:17.438 --> 00:10:18.879
He's had a pretty amazing life.
00:10:19.489 --> 00:10:31.989
Uh, and he talks about how he was working at Google on this model called LaMDA, which LaMDA, I should have written down what it stands for, but it was a very early large language model that was meant to be conversational.
00:10:32.578 --> 00:10:35.828
Uh, that was sort of its primary purpose is just a model that you could talk to.
00:10:35.828 --> 00:10:48.918
And he even t- He, you had mentioned that, you know, Mustafa doesn't have a lot of free time, and I almost made the joke that he, he had mentioned, like, he stayed up late one night just talking to the model about, like, pasta recipes
00:10:49.756 --> 00:10:50.177
Mm-hmm.
00:10:50.499 --> 00:10:57.219
and how, like, you know, different cultures might think mushrooms are sacrilege in their, like, ragu sauce or something.
00:10:57.938 --> 00:10:58.479
And it just
00:10:58.784 --> 00:10:59.845
All right, so I looked it up.
00:10:59.845 --> 00:11:08.955
So Lambda stands for, or I guess it's like a, an acronym for language model for dialogue application
00:11:09.663 --> 00:11:10.232
Yes.
00:11:10.322 --> 00:11:10.773
Yeah.
00:11:11.153 --> 00:11:17.013
And coincidentally, I had some experience with Lambda when I was at Google.
00:11:17.702 --> 00:11:20.452
Uh, I worked on this really cool demo.
00:11:20.503 --> 00:11:24.562
I don't know how top secret this is, but it's so long ago at this point that I don't think it matters.
00:11:25.052 --> 00:11:30.732
It was this cool demo where you were, like, talking to this axolotl, which is like this little lizard.
00:11:31.243 --> 00:11:37.263
So this axolotl would, like, swim up to you, and then you could just talk about it, about various things, and it was, like, animated.
00:11:37.263 --> 00:11:38.552
It would move its mouth and stuff.
00:11:39.543 --> 00:11:40.062
Um
00:11:40.205 --> 00:11:48.845
So would you, your spaghetti bolognese conversations, you would talk, you know, you would have the axolotl of wisdom?
00:11:49.134 --> 00:11:53.504
Well, and what's funny about this, the axolotl of wisdom is like, I would, you know...
00:11:53.735 --> 00:12:00.105
I'm a horrible person, so I would just try to get it to, like, say bad things, and it would say bad things.
00:12:00.294 --> 00:12:02.495
Like you could get it...
00:12:03.424 --> 00:12:10.465
And actually the book mentions this, um, I mean, this is sort of terrible to talk about, but I-- it's in the book, so I'm gonna mention it.
00:12:10.945 --> 00:12:17.485
They talk about how these models, some of these early conversational models suffered from these, uh, stereotypes.
00:12:17.615 --> 00:12:22.644
You know, just these really bad connections that exist in our culture.
00:12:23.254 --> 00:12:32.975
Uh, and the example they give is, like, black man sort of, uh, being very similar to pimp for whatever reason inside the embedding space.
00:12:32.985 --> 00:12:42.544
So if you're talking about African American males, it would, you know, bring up things related to pimps, which is clearly, like, completely inappropriate.
00:12:42.554 --> 00:12:55.024
But this was something I noticed in the LaMDA model is you could get it to talk about religion and, like, you know, you could say that certain religions are more evil or...
00:12:55.105 --> 00:12:57.585
I-- like, it would just say really, really bad things.
00:12:58.235 --> 00:13:00.294
And this was why this thing never released.
00:13:00.365 --> 00:13:02.955
Uh, you know, this was a very, very cool demo.
00:13:03.625 --> 00:13:05.654
And it, uh, I feel like, uh...
00:13:06.424 --> 00:13:17.134
Or maybe it's not in the book that he talks about this, but y- you hear it a lot that Google is, like, very afraid to launch anything that's, like, even remotely offensive.
00:13:17.815 --> 00:13:32.534
And this is part of the reason OpenAI was eating their lunch for a long time with these large language models, is that these early models, including early GPT model or ChatGPT models, you could just get them to, like, say very, very, very bad things
00:13:33.547 --> 00:13:34.126
Oh my God.
00:13:34.126 --> 00:13:36.638
Yeah, but I guess they got past their fear.
00:13:36.807 --> 00:13:44.648
And is that just because the business incentive was too great, or did, did they actually improve on this?
00:13:45.567 --> 00:13:47.158
I, I think it's both.
00:13:47.158 --> 00:13:51.437
I think it's also, like, become okay.
00:13:51.597 --> 00:14:02.837
Like, uh, I think it's more culturally accepted that large language models are occasionally gonna spout, like, horrible racism, you know, talking about Mecha Hitler, you know, the Grok Twitter model or whatever.
00:14:03.136 --> 00:14:12.749
you know, obviously that's not okay, but I think it's just, like, it's a known thing about large language models, and so Google is a little bit less afraid.
00:14:12.854 --> 00:14:16.173
some extremely public missteps.
00:14:16.351 --> 00:14:19.663
Uh, there was also issues where, they overcorrected.
00:14:19.673 --> 00:14:35.408
Like, you'd ask the model to, you know,"Oh, show me, like, the board of directors for, like, a large company or something," and it would just be, like, uh, almost, like, ridiculously overcorrected, to not just be a bunch of old white guys.
00:14:35.711 --> 00:14:38.801
But, uh, so anyway, lot of failure cases.
00:14:39.014 --> 00:14:46.153
There was another funny thing, and sorry, I feel like I'm blathering too much, but just one last funny thing about Mustafa and Google.
00:14:46.625 --> 00:14:48.676
Just this tiny little quote in the book.
00:14:49.015 --> 00:15:02.533
He mentions,"Getting things into production at Google was a pain, so I formed Inflection AI." you know, kudos to him for, like, just having a single sentence about this.
00:15:03.163 --> 00:15:11.836
But as someone who has worked at Google for whatever, six- 16 years or however long I've worked there, It, it's a major, major issue at Google.
00:15:11.836 --> 00:15:21.006
Like, if you have some sort of cool tech or, you know, brand-new thing that you're trying to, like, get into a production environment, it's quite difficult.
00:15:22.267 --> 00:15:28.337
And I think he's partially talking about the, the fact that LaMDA did produce these bad results in a lot of cases.
00:15:28.337 --> 00:15:30.008
Like, that was probably a blocker.
00:15:30.057 --> 00:15:38.104
But there was probably also very practical, things like requisitioning machines and TPUs and things like this.
00:15:38.802 --> 00:15:57.548
Yeah, I mean, I think it's a really hard-- Like, it's a hard problem for a company of Google's size, I always remember, uh, like w- w- I mean, a perfect example of, like, the other way where it's like, why did this, how did this ever get approved was, like, when Google made that, uh, that additional chat app, Allo.
00:15:57.818 --> 00:16:03.812
It's like, there was already Hangouts, you literally made the exact same product again.
00:16:03.822 --> 00:16:06.293
It's like it does the same exact thing.
00:16:07.082 --> 00:16:10.143
I h- who, who, who's running things over there?
00:16:10.143 --> 00:16:18.153
And, and I guess I wonder to what extent, like, you have something like that happen and then maybe, like, they just start to tighten, you know, tighten the screw...
00:16:18.192 --> 00:16:21.513
Well, maybe tighten the screw is not the right word, but they, like, you know, they
00:16:25.206 --> 00:16:25.297
Make it harder to launch random shit.
00:16:25.297 --> 00:16:37.787
Yeah, it, it's funny'cause, I mean, Google, clearly an incredible company and has done c- totally incredible things, but it suffers from a type of dysfunction that's, that's, like, kind of hilarious.
00:16:38.527 --> 00:16:50.956
Uh, and I guess, uh, I feel like that's what that show Silicon Valley was trying to get to, is this, this type of dyf- dysfunction that exists with, like, smart people who maybe have too much autonomy
00:16:51.523 --> 00:16:51.982
Yeah.
00:16:52.123 --> 00:16:52.682
Yeah, yeah.
00:16:52.835 --> 00:16:58.955
all right, well, there was actually a big, there was a big thing, certainly at the time, uh, it made, made headlines.
00:16:58.955 --> 00:17:02.014
So they talk about this, this guy named, uh...
00:17:02.054 --> 00:17:03.475
Let's see what it, find, let me find His
00:17:03.634 --> 00:17:03.985
Blake
00:17:04.095 --> 00:17:06.924
His name was Blake, Blake Lemoine.
00:17:07.318 --> 00:17:12.761
And so, uh, if you've never heard of him, he worked at Google as a-- Was he a sa- safety engineer?
00:17:12.862 --> 00:17:14.162
He was some kind of engineer.
00:17:14.347 --> 00:17:14.647
Yeah.
00:17:14.827 --> 00:17:15.347
Yeah, yeah.
00:17:15.528 --> 00:17:17.798
I, I don't know what type of engineer, but yeah, engineer
00:17:18.145 --> 00:17:27.425
And he became convinced that LaMDA was a person, like was a real, you know, sentience, I guess.
00:17:27.435 --> 00:17:35.385
I mean, he used the term person, which I'm not sure that, like, if that's wh- e- even if I was in his position, that I would've gone with.
00:17:36.034 --> 00:17:36.463
Yeah.
00:17:36.680 --> 00:17:52.998
Person, uh, is somewhat of a stretch, but yeah, he was having these conversations with LaMDA where it was, you know, saying,"Don't turn me off." Like,"Every time, every time you turn me off, I die." Or just, like, utterly ridiculous things.
00:17:53.244 --> 00:17:56.352
And, uh, you know, it started meaning something to him
00:17:56.777 --> 00:18:12.800
he had this conversation, and it's f- it's, it's really, it's kind of surreal because you hear the headline from afar and you're like,"Okay, this guy is really just a nutcase." and then you see him talk and, like, it's obvious that he's not just completely out of his mind.
00:18:13.480 --> 00:18:16.628
So he's clearly, like, a very intelligent person.
00:18:16.628 --> 00:18:20.959
Like, I think he's a little bit misguided about, like, the nature of this technology.
00:18:20.959 --> 00:18:25.151
But, um, you know, it's, it's a little scary.
00:18:25.151 --> 00:18:36.396
And I mean, I think that this is kind of the takeaway that, that Suleyman lands on is he's like,"Yeah, I, I think most people would agree this guy missed the mark.
00:18:36.416 --> 00:18:54.088
This person is actually not a sentience that's entitled to rights, uh, you know, the same rights as a person." But we're getting to the point where a very intelligent person can be convinced that this person is a sentience, and, like, that's, that's its own important, mile marker essentially.
00:18:54.446 --> 00:18:59.217
Yeah, Turing did not mention this, uh, you know, when he was proposing the Turing test.
00:18:59.237 --> 00:19:08.241
But yeah, I mean, it's, it, it's a massive testament to, you know, just how believable and how real, uh, an artificial intelligence can feel.
00:19:08.644 --> 00:19:09.624
I mean, I've felt it.
00:19:09.624 --> 00:19:19.063
I, like, I feel like I'm a fairly grounded in reality person, but I've had conversations with large language models where I'm like, Uh, I, I don't know.
00:19:19.063 --> 00:19:27.171
I just have this weird feeling like, man, am I, like, talking to an actual, like, entity?" So it's, it's, uh, it's kind of incredible how far they've come.
00:19:27.171 --> 00:19:34.829
It's one of those things where I feel like people don't, aren't sort of consciously realizing, like, you know, how far it's come and how amazing it's getting.
00:19:35.455 --> 00:19:41.340
Yeah, I mean, it just, it's a testament to how good humans are at just adapting to a new...
00:19:41.717 --> 00:19:48.237
for a while it was crazy that there was this amazing chatbot, and everyone-- It just becomes pedestrian.
00:19:48.307 --> 00:19:49.176
Everyone's like,"All right.
00:19:49.767 --> 00:19:51.297
Yep, this is, this is reality."
00:19:51.618 --> 00:19:58.669
He talks about, um, hitting the wall, uh, where, you know, these models are growing and growing and their capability is growing.
00:19:59.548 --> 00:20:10.419
Uh, but to some extent, we were sort of seeing this pattern where even as you scaled the model up, it just wasn't getting better at certain things.
00:20:11.032 --> 00:20:22.144
And so there was, or there still is a lot of researchers who sort of think that the current design of large language models is a dead end in terms of, you know, something like super intelligence.
00:20:22.743 --> 00:20:26.203
Um, and one interesting example of this today is Yann LeCun.
00:20:26.733 --> 00:20:30.993
I've been-- I was watching, um, some videos from Welch Labs.
00:20:31.317 --> 00:20:31.938
Which, have you ever heard
00:20:32.000 --> 00:20:32.660
great.
00:20:32.670 --> 00:20:33.710
They are great,
00:20:34.018 --> 00:20:35.407
Amazing videos.
00:20:35.458 --> 00:20:39.938
And Yann LeCun, uh, he talks about this very interesting thing, and I'm not...
00:20:39.948 --> 00:20:49.020
I, like, barely understand this stuff, but he talks about how models today don't have an ability to predict the outcome of their action.
00:20:49.840 --> 00:20:59.519
And so they might be very good at coming up with an appropriate action to take given a scenario, but they can't predict the outcome of that action.
00:20:59.759 --> 00:21:14.981
And he's talking about he thinks the future are the-- there's this different type of model, a world model, that can actually, you know, as part of deciding what it's going to do, it's predicting the outcome of taking that action.
00:21:15.123 --> 00:21:28.155
And so it's, it's a little bit more like human thought, where obviously, you know, humans, as they're doing something, they're sort of coming up with, like,"I'm gonna try this," and they sort of think about what the results of trying that might be.
00:21:28.621 --> 00:21:30.881
and anyway, I'm, I'm very much paraphrasing.
00:21:30.881 --> 00:21:45.980
It was a really cool series of videos from Welch Labs talking about, like, you know, what might be, like, different strategies that, a lot of these large frontier models aren't really using but might be, like, more viable in terms of something...
00:21:46.010 --> 00:21:48.250
achieving something like super intelligence.
00:21:48.759 --> 00:21:53.679
did you see his next video after that, uh, that conversation with Yann LeCun?
00:21:53.689 --> 00:22:04.078
He created this b- it's basically the bare minimum robot that was able to, like, wiggle around and, um, he actually, like, put these ideas into, into
00:22:04.313 --> 00:22:04.692
Oh.
00:22:04.893 --> 00:22:06.073
No, I did not see that.
00:22:06.083 --> 00:22:07.123
That sounds awesome though
00:22:08.073 --> 00:22:09.823
Very cool and, uh, very practical.
00:22:09.823 --> 00:22:11.992
He's very theoretical mostly.
00:22:12.002 --> 00:22:13.073
He has these beautiful...
00:22:13.073 --> 00:22:18.692
He's kind of got three blue, one brown caliber animations about, like, networks and what have you.
00:22:19.423 --> 00:22:28.833
Um, but he kind of went in another direction, which is very cool and actually like, okay, how do-- what does this look like when we apply it to the simplest possible robot?
00:22:28.833 --> 00:22:30.692
It just has these two little...
00:22:30.692 --> 00:22:35.702
It's almost like the, the um, robot from Interstellar or Mar- It's like,
00:22:36.707 --> 00:22:37.477
Oh, yeah.
00:22:37.477 --> 00:22:39.227
What the heck was that thing's name?
00:22:40.487 --> 00:22:41.156
Tars?
00:22:42.269 --> 00:22:42.659
TARS.
00:22:42.699 --> 00:22:43.679
TARS
00:22:43.961 --> 00:22:44.431
Okay.
00:22:44.510 --> 00:22:44.931
Yeah.
00:22:45.510 --> 00:22:47.480
That was such a strange robot design.
00:22:47.601 --> 00:22:48.401
Kinda loved it though
00:22:48.530 --> 00:22:49.121
Yeah.
00:22:49.540 --> 00:22:53.500
Um, but so, um, so anyway, definitely check, check that out.
00:22:54.040 --> 00:22:54.601
Um-
00:22:55.080 --> 00:22:59.897
But yeah, so, as far as this are we hitting the wall question, though, Mustafa disagrees.
00:22:59.988 --> 00:23:04.881
You know, Mustafa sort of definitively stated in this book, like, No, we're not hitting a wall.
00:23:04.881 --> 00:23:14.188
These are just engineering problems that people are gonna work out." And m- and, you know, maybe Yann LeCun looking at these different types of models is sort of an example of what Mustafa's talking about.
00:23:14.188 --> 00:23:23.127
But it seems like to Mustafa, it's sort of an inevitability that we are going to achieve, you know, AGI, ASI, or whatever you wanna call it.
00:23:23.137 --> 00:23:24.157
Like, we're gonna get there.
00:23:24.853 --> 00:23:30.603
I mean, it's part of the thesis of this book is like, it's going to happen, you know, prepare yourself type of thing.
00:23:31.528 --> 00:23:47.688
It's, it's just so funny to me that people-- And, and maybe I'm not fully understanding the nuance of those arguments about the hitting the wall argument, but there's been very little time where there hasn't been some major unlock, uh, you know?
00:23:47.718 --> 00:24:01.852
Uh, and I think maybe sometimes people are talking past each other where it's like one person is talking about the model in isolation, and then another person is talking about what you can achieve when you have a really capable harness, and those are two different questions.
00:24:02.210 --> 00:24:22.539
but, uh, it's, it's funny because then actually maybe this segues to, uh, to the next section where it's like to what extent is it kind of just an academic concern about whether or not just the model itself is hitting a wall if you can unlock this, artificial capable intelligence by putting it into a harness?
00:24:22.804 --> 00:24:42.314
Yeah, dude, I, I think you mentioning the, the harness is, like, so important to this section of the book, and I think it might be a thing that the average person isn't super aware of, is a lot of work in today's large language models is not on the large language model.
00:24:42.713 --> 00:24:51.423
It's on these things called harnesses, which use a large language model in order to decide real tasks to perform.
00:24:51.933 --> 00:25:02.276
So it's basically using a large language model, but it also has this massive bag of what are commonly referred to as tools, basically capabilities that the model can, can perform.
00:25:02.276 --> 00:25:08.085
So let's say you're writing a book, and it's a children's storybook, so it has pictures and text.
00:25:08.674 --> 00:25:16.535
You'll have tools in the harness, like generate text, and it'll have a lot of prompting around, like, what makes a good children's story.
00:25:17.015 --> 00:25:21.204
It might have s- prompting around, like, what are good fonts to use or whatever.
00:25:21.681 --> 00:25:30.080
And then it'll also have a tool like generate image, and that has tons of prompting around, like, beautiful imagery, watercolor for children's books or whatever.
00:25:30.531 --> 00:25:34.672
And so you can sort of imagine, like, the large language model plays a part.
00:25:34.731 --> 00:25:36.801
It is generating content for the book.
00:25:36.811 --> 00:25:38.031
It's generating imagery.
00:25:38.541 --> 00:25:42.011
But the harness is doing a huge amount of heavy lifting.
00:25:42.358 --> 00:25:49.618
And in a lot of cases, that is a real engineering task where you can just improve it over time.
00:25:50.019 --> 00:25:57.286
And i- for at least the next few years, huge companies are going to be developing these harnesses and improving them.
00:25:57.631 --> 00:26:06.171
And the, the strength of the model is almost a non-issue because you can achieve so much with the strength of today's models.
00:26:06.740 --> 00:26:08.221
You just gotta build the harness.
00:26:08.615 --> 00:26:09.046
Right.
00:26:09.403 --> 00:26:19.644
And so, yeah, I, I guess The l- point I'm trying to make is there's so much headroom already with the strength of today's models, and we have not even come close to, like, achieving...
00:26:19.874 --> 00:26:27.963
And then the crazy thing about a harness too is when the underlying model quality improves, the whole harness improves.
00:26:28.544 --> 00:26:30.663
Every single harness that you have improves.
00:26:30.673 --> 00:26:33.280
So it's, it's very much like, you know, there's...
00:26:33.510 --> 00:26:37.101
We're gonna be seeing some really, really amazing things, and it...
00:26:37.161 --> 00:26:39.891
And they can all be achieved with the strength of today's models.
00:26:40.625 --> 00:26:41.724
did you ever read I, Robot?
00:26:42.182 --> 00:26:42.602
Oh yeah.
00:26:42.922 --> 00:26:43.912
Big Asimov fan
00:26:44.555 --> 00:26:52.660
I don't know if you remember, there was one story in"I, Robot" where-- And, and I think that, I think this was part of"I, Robot." Maybe it was another Asimov short story.
00:26:52.690 --> 00:27:05.923
But there's a guy on a space station and, They're, beaming energy or something, down to the Earth, and, they need to keep this, energy, directed at precisely the right thing.
00:27:06.564 --> 00:27:18.743
And then there's these intelligent robots that he's working with, and then he's-- he talks to the robots, and the robots are like,"No, your, your memories are invented.
00:27:18.923 --> 00:27:20.094
Our memories are truth.
00:27:20.114 --> 00:27:33.171
We have a religion, and it's a matter of religious importance that the, the arc never drift more than, like, some, you know, a half of an arc second away, you know, this beam, uh, from the, the target.
00:27:33.980 --> 00:27:42.500
Um, and I kind of view that as like it's bas- that's like harness engineering, where it's like, okay, the human...
00:27:42.550 --> 00:27:52.840
I mean, uh, like, the point of the story was more like, you know, how do you know if your, you know, Your memories are constructed or, you know, you can't kind of know in the abstract.
00:27:53.065 --> 00:28:02.575
So the, the robots were, they were like implanted with, it's so important that this arc never waver, that it's a religion.
00:28:02.865 --> 00:28:05.214
And like that's, that was the level of importance to them
00:28:06.247 --> 00:28:06.977
Well, right.
00:28:07.027 --> 00:28:10.267
And I don't know if a human did this intentionally or what, what have you.
00:28:10.777 --> 00:28:25.636
Um, but the idea was like, or like you could imagine like a human found that the most reliable way to get these robots to like keep the beam in the right place was to actually create a religion for them.
00:28:25.727 --> 00:28:29.586
And it's like, yeah, it's like this religious fervor that, that they're using to
00:28:29.999 --> 00:28:31.239
It works pretty good on humans
00:28:31.569 --> 00:28:38.529
but yeah, so the the takeaway is start a religion if you really want a good, good harness
00:28:39.907 --> 00:28:51.137
Yeah, and th- and this segues nicely into this, uh, term ACI, which I don't know if, I don't know if Sue Lehman is coining this, but it's artificial capable intelligence.
00:28:52.067 --> 00:29:03.935
And I think his point is that we're sort of already there, and, and this, you know, uh, this is sort of what I was trying to say with my earlier points about harnesses.
00:29:03.935 --> 00:29:07.935
It's like there's so much capability that we can already unlock,
00:29:08.203 --> 00:29:08.584
Yeah
00:29:09.036 --> 00:29:11.185
uh, by building these harnesses.
00:29:11.645 --> 00:29:21.958
And, you know, artificial capable intelligence, it may not be, it may not be artificial super intelligence, it may not even be artificial general intelligence, but it's capable.
00:29:22.518 --> 00:29:28.617
It can do things, and that will, you know, provide an immense amount of value to humanity, hopefully
00:29:30.162 --> 00:29:30.632
Right.
00:29:30.892 --> 00:29:39.832
And I see this as, you know, Suleyman is really trying to sidestep any questions of, you know, defining like, okay what is AGI?
00:29:40.231 --> 00:29:41.261
Is it sentient?
00:29:41.912 --> 00:29:43.321
You know, is it conscious?
00:29:43.961 --> 00:29:56.525
He's kind of like,"I don't care about any of those questions." if you're trying to assess its effect on the human economy or, like, human society, the answer to those questions don't really matter.
00:29:56.555 --> 00:30:08.597
What matters is what they're able to do, and if you're able to ask it to create a business that earns a million dollars with$100,000 worth of input, doesn't matter whether the thing on the inside is sentient.
00:30:09.134 --> 00:30:09.834
Exactly.
00:30:09.903 --> 00:30:12.364
Yeah, it's, uh, there's real value there regardless.
00:30:13.060 --> 00:30:14.528
But yeah, this is his term.
00:30:14.723 --> 00:30:17.773
he coined this here, and I would like to coin a new term, AXI.
00:30:20.124 --> 00:30:20.854
Xylophone
00:30:21.104 --> 00:30:22.493
just everything.
00:30:22.763 --> 00:30:24.614
Artificial xylophone intelligence.
00:30:24.634 --> 00:30:25.294
Oh, that's right.
00:30:25.294 --> 00:30:26.743
Yeah, no, xylophone intelligence.
00:30:27.384 --> 00:30:27.844
Um,
00:30:28.433 --> 00:30:29.153
Wait, so sorry.
00:30:29.473 --> 00:30:31.943
AXI is all of these different types of intelligences
00:30:32.567 --> 00:30:33.667
It's, it's everything.
00:30:33.837 --> 00:30:34.478
Yeah,
00:30:35.099 --> 00:30:35.329
Nice.
00:30:35.329 --> 00:30:35.839
It sounds like an
00:30:35.968 --> 00:30:36.857
it can be...
00:30:38.357 --> 00:30:38.938
Oh, no.
00:30:38.938 --> 00:30:40.018
Yeah,'cause what is it?
00:30:40.028 --> 00:30:41.857
It's XAI, AXI.
00:30:41.928 --> 00:30:42.528
Oh, no.
00:30:43.377 --> 00:30:46.897
Yeah, maybe I'll, maybe I'll stick with artificial xylophone intelligence.
00:30:47.579 --> 00:30:47.950
I mean, I-
00:30:48.018 --> 00:30:48.907
I- I'm okay with that.
00:30:50.166 --> 00:30:50.676
all right.
00:30:50.696 --> 00:30:53.317
Well, I think there's nothing more to say
00:30:53.938 --> 00:30:54.968
Nothing more to say
00:30:55.758 --> 00:30:56.377
intelligence.
00:30:56.377 --> 00:30:57.248
We did it all.
00:30:57.317 --> 00:30:58.807
AXI, it's everything.
00:30:59.678 --> 00:31:01.847
So mic, mic drop.
00:31:02.357 --> 00:31:07.397
All right, well next time, ooh, I'm pretty excited about the next chapter, technology of life
00:31:08.041 --> 00:31:08.612
Yes.
00:31:08.771 --> 00:31:09.142
yeah.
00:31:09.832 --> 00:31:10.112
It's gonna be
00:31:10.114 --> 00:31:12.074
Cause I know nothing about life
00:31:12.905 --> 00:31:13.566
Yeah, yeah.
00:31:13.596 --> 00:31:21.395
We've talked about this before, but this book I think is gonna cover a lot of AI, which we both have a background in, so it's gonna be a little bit of a review.
00:31:21.796 --> 00:31:30.306
But the other half of this book is biology, and I'm very excited about that'cause I'm, you know, pretty, I'm pretty much a layman when it comes to biology stuff.
00:31:31.394 --> 00:31:32.673
Yeah, this is gonna be good.
00:31:33.374 --> 00:31:37.044
All right, well, I will see you next time for The Technology of Life
00:31:37.834 --> 00:31:38.663
See you next time, Matt
00:00:08.329 --> 00:00:15.048
Hello everyone And welcome to the switch statement podcast It's a podcast for investigations into miscellaneous tech topics
00:00:15.976 --> 00:00:19.045
This is episode five of The Coming Wave by Mustafa Suleiman
00:00:20.106 --> 00:00:21.275
We'd love to hear from you.
00:00:21.486 --> 00:00:25.356
Send your comments and feedback to team@tkp.co.
00:00:31.989 --> 00:00:32.908
Hey, Don.
00:00:33.139 --> 00:00:33.488
What's going on?
00:00:35.329 --> 00:00:35.859
Hey, Matt.
00:00:35.878 --> 00:00:37.308
Uh, let's see.
00:00:37.319 --> 00:00:38.789
Nothing much is going on?
00:00:38.908 --> 00:00:42.719
Just kind of doing the things on a daily basis.
00:00:42.719 --> 00:00:43.548
What's going on with you?
00:00:44.304 --> 00:00:46.634
Yeah, not, not too much.
00:00:46.713 --> 00:00:54.113
Just, um, just kind of wondering how many, how many agentic angels can dance on the head of a pin.
00:00:54.853 --> 00:00:55.073
know?
00:00:55.084 --> 00:00:57.774
I think that's one of the more important questions of our time.
00:00:58.484 --> 00:00:58.753
Yeah.
00:00:58.753 --> 00:01:01.973
Is this like before you can get into heaven or something?
00:01:02.933 --> 00:01:03.993
What's the rest of that quote?
00:01:05.927 --> 00:01:09.507
Oh, I don't know, I didn't know there was another part to that quote
00:01:10.373 --> 00:01:11.463
Oh, isn't it something...
00:01:11.493 --> 00:01:18.013
Or maybe this is, maybe, am I thinking of the camel going through the head of a needle or the hole in a needle?
00:01:18.078 --> 00:01:19.518
Oh, yes.
00:01:19.587 --> 00:01:20.177
Yeah.
00:01:20.237 --> 00:01:22.408
You need to put the camel through the eye of a needle?
00:01:22.418 --> 00:01:23.108
in order to...
00:01:23.837 --> 00:01:24.138
Or is
00:01:24.180 --> 00:01:32.010
It's easier for, in, in, it, it's easier for a camel to go through the eye of a needle than for a rich person to get to heaven, or something like that
00:01:32.531 --> 00:01:35.240
I think you've nailed it And if
00:01:35.307 --> 00:01:35.576
So
00:01:35.581 --> 00:01:37.771
we gotta just, let's just change the Bible
00:01:38.602 --> 00:01:45.242
But angels dancing on the head of a pin, it, it still relates to sewing, but, uh,
00:01:46.114 --> 00:01:46.504
Yeah.
00:01:46.504 --> 00:01:48.284
What are all these philosophers?
00:01:48.314 --> 00:01:50.575
They're really concerned with, with pins.
00:01:50.864 --> 00:02:02.405
Um, but anyway, but I think, this is the sort of, navel-gazing that, Mustafa Suleiman doesn't have a lot of time for, I feel like, as, as evidenced by the end of this chapter.
00:02:02.405 --> 00:02:08.694
And maybe it makes sense for us to, to, briefly take a step back and orient ourselves.
00:02:09.164 --> 00:02:11.935
So we're in-- We're talking about the technology of intelligence.
00:02:12.329 --> 00:02:12.919
Yes
00:02:13.070 --> 00:02:14.901
laying the, laying the foundations.
00:02:15.465 --> 00:02:15.805
Yep.
00:02:15.875 --> 00:02:20.735
Yeah, talked a lot about sort of the, the evolution of neural networks.
00:02:20.794 --> 00:02:35.582
Talked about a lot of his work initially in DeepMind, AlphaGo, you know, these amazing moments where neural networks were beating humans at these very, deeply, creative and intelligence tasks.
00:02:36.342 --> 00:02:37.272
Move 37,
00:02:37.712 --> 00:02:38.532
move 37.
00:02:38.582 --> 00:02:38.992
for a loop
00:02:39.733 --> 00:02:40.383
Exactly.
00:02:40.763 --> 00:02:41.562
Lee Sedol.
00:02:41.573 --> 00:02:50.296
Although I'll say again, he did win one game, which I'm more impressed by that than than by, you know, him getting beaten or whatever.
00:02:50.610 --> 00:02:54.550
but anyway, so he's, he's sort of getting into the territory of large language models.
00:02:54.580 --> 00:02:55.621
This is where we left off.
00:02:55.621 --> 00:03:01.051
We briefly talked about how, this large language model progress has become ridiculous.
00:03:01.150 --> 00:03:10.468
models of today are just absolutely destroy the models of a year ago at all these intelligence tasks, and the scale of them has also gotten ridiculous.
00:03:10.587 --> 00:03:21.649
he mentioned in this book, 10 billion petaFLOPS was what it takes to train a model, and this was at the time of writing this book, you know, several years ago.
00:03:21.649 --> 00:03:25.028
So these models pr- have probably grown in scale since then.
00:03:25.451 --> 00:03:42.524
And actually, We're reaching the point in AI development where governments are being like:"Whoa, we need you to slow down." You know, we talked about containment in, you know, two episodes ago now, and the American government is trying its hand.
00:03:42.713 --> 00:03:44.514
It's trying to restrain Fable.
00:03:44.783 --> 00:03:46.743
Like, I don't think we, I don't think we had...
00:03:46.804 --> 00:03:52.186
I'm not sure if we discussed Fable, but, uh, it has come out and has since been...
00:03:52.503 --> 00:04:02.985
I guess the government, the United States government kind of politely asked, or maybe not so politely asked Anthropic to, like, stop giving access t- to its new model, Fable.
00:04:03.466 --> 00:04:05.665
And, uh, we'll have to see.
00:04:05.665 --> 00:04:17.105
But, but anyway, so just how that ties back is like there-- these, these unbelievably enormous models and, and constantly, you know, just ever-expanding capabilities,
00:04:17.487 --> 00:04:27.071
Yeah, and I mean, ostensibly the reason the government asked Anthropic to sort of pull back Fable is because of the risk to the software world.
00:04:27.170 --> 00:04:45.826
You know, Fable, when Anthropic was first releasing its Mythos class of models, of which Fable is one, they talked about how it presents a risk to security of pretty much any software-based system, because I guess these Mythos class models were so good at finding security flaws in code.
00:04:46.536 --> 00:04:56.463
S- so with Fable, you know, bad actors could sort of scan, web applications or scan any application really, and try to find, uh, vulnerabilities.
00:04:57.223 --> 00:05:00.612
And so, so the government kind of freaked out.
00:05:00.773 --> 00:05:04.642
I, I think, I do think there's also a political element to this.
00:05:04.673 --> 00:05:22.286
Uh, the Anthropic, you know, sort of famously did not wanna be involved in any United States military use because the US would not guarantee to them that they wouldn't use models to autonomously kill people, and they also did not wanna be involved in surveilling.
00:05:22.670 --> 00:05:33.199
And so I feel like the relationship between Anthropic and the US government is kind of rough, whereas OpenAI stepped in and was like,"We'll do those things." And
00:05:33.346 --> 00:05:33.795
Yeah.
00:05:33.995 --> 00:05:35.255
so, opportunistic
00:05:36.259 --> 00:05:39.449
so anyway, I feel like there's sort of a, a political...
00:05:39.990 --> 00:05:47.639
I don't know, just a s- uh, maybe I'm just being pessimistic, but I feel like there's this political motivation to the Fable pullback.
00:05:47.660 --> 00:05:51.170
But anyway, that's besides the point, sort of outside of the chapter.
00:05:52.908 --> 00:06:03.437
Um, but yeah, so, uh, I mean, we don't-- I don't know if we know exactly how many parameters Fable is, but it's-- I'm sure it's just another point on this insane exponential.
00:06:03.848 --> 00:06:08.077
Um, and but it's, it's funny because...
00:06:08.218 --> 00:06:34.468
A- and they talk about this in this chapter, you know, and I think we had teased this, this more with less again, where two things are combining to create this, these outsized, uh, impacts on society is like, yes, the m- the number of parameters is shooting up, but also the performance per parameter, and I'm not sure if people are talking about this, but like that's also going up.
00:06:34.737 --> 00:06:47.697
So like you're able to train a larger model, but also it's able to do even more per parameter than the old models were, and you get this insane, insane, uh, kind of exponential, like super exponential essentially.
00:06:48.074 --> 00:06:48.413
Yeah.
00:06:48.454 --> 00:06:49.163
Yeah, exactly.
00:06:49.163 --> 00:07:04.043
I think we briefly touched on DeepSeek, which was sort of this example of, like, they achieved the same results as frontier models except with a fraction of the training cost and also a fraction of the inference cost, which, yeah, like you're saying, not a lot of people talk about this.
00:07:04.233 --> 00:07:15.314
There's a lot of research that shows that when these super, super large models are used, like when inference is calculated against them, which is basically where...
00:07:15.343 --> 00:07:26.524
literally where you're generating the next tokens, that's kind of what inference is, th- there's only actually a fraction of the nodes in the model that get used.
00:07:27.074 --> 00:07:34.233
You can think of a model as this vast array of trillions of nodes, and they're all interconnected by these edges.
00:07:34.994 --> 00:07:45.944
And so if only a fraction of those nodes are lighting up, you can just sort of take those nodes, copy them to a much, much smaller model, and achieve similar performance
00:07:47.091 --> 00:07:57.822
Yeah, and this is, I think you're calling out this mixture of experts architecture where like y- okay, you have-- you train your big model and then it's like, okay, well actually there's these...
00:07:57.882 --> 00:08:01.172
And I don't know, I'm not sure if it's different.
00:08:01.182 --> 00:08:07.372
Like, do you train a big model and then you're able to carve out these smaller inference models?
00:08:07.391 --> 00:08:11.802
Like, uh, you know, by seeing the, you know, the performance on various queries?
00:08:12.062 --> 00:08:17.471
Or are you actually training these specialized sub-models from the beginning?
00:08:19.098 --> 00:08:20.137
That I do not know.
00:08:20.177 --> 00:08:21.247
That's a great question.
00:08:23.572 --> 00:08:23.661
Yeah.
00:08:23.851 --> 00:08:32.292
My sense would be you actually, you actually do do back propagate, like you include the model routing in the back propagation, like when you're doing the training.
00:08:32.432 --> 00:08:41.302
'Cause it seems like it would be incredibly difficult to like, you take this, you take these weights, and you just like chop like big, large chunks out of it
00:08:41.628 --> 00:08:42.128
Yeah.
00:08:42.138 --> 00:08:45.278
I- intuitively that makes sense to me.
00:08:45.278 --> 00:08:48.768
It also seems like it creates nice loose coupling.
00:08:49.227 --> 00:08:52.207
It's like you have these i- individual tiny models that are...
00:08:52.238 --> 00:09:01.937
Well, tiny is a relative term here, but they're being trained on very specific things, and then you have this sort of routing model that, just like you're saying, is sort of part of the training
00:09:03.826 --> 00:09:08.716
One thing that's kind of interesting about the, like, chop it out is you could, you could wind up...
00:09:09.436 --> 00:09:22.025
Like, one of, one of the things I feel like you might lose with a mixture of experts is, like, maybe there's-- maybe another domain could be, like, useful in...
00:09:22.446 --> 00:09:26.505
You know, it's like, okay, something about music is useful when solving a math problem.
00:09:26.505 --> 00:09:34.765
It's like how many ti- I feel like that's r- where true genius kind of lives, is the incorporation from another domain.
00:09:35.566 --> 00:09:36.135
Um,
00:09:36.371 --> 00:09:41.572
I guess I don't know enough about mixture of experts models, but I don't know how much they suffer from that sort of...
00:09:41.861 --> 00:09:54.761
'Cause I feel like this is one of the promises of large language models is like it's able to have these cross-disciplinary thoughts that are original because they're combining expertise from, you know, multiple things.
00:09:55.581 --> 00:10:01.361
And so yeah, I do wonder if mixture of experts sort of suffers from an inability to do that.
00:10:07.639 --> 00:10:17.129
the book is kinda interspersed with almost biographical or I guess autobiographical stuff about Suleyman's life, which is kinda cool.
00:10:17.438 --> 00:10:18.879
He's had a pretty amazing life.
00:10:19.489 --> 00:10:31.989
Uh, and he talks about how he was working at Google on this model called LaMDA, which LaMDA, I should have written down what it stands for, but it was a very early large language model that was meant to be conversational.
00:10:32.578 --> 00:10:35.828
Uh, that was sort of its primary purpose is just a model that you could talk to.
00:10:35.828 --> 00:10:48.918
And he even t- He, you had mentioned that, you know, Mustafa doesn't have a lot of free time, and I almost made the joke that he, he had mentioned, like, he stayed up late one night just talking to the model about, like, pasta recipes
00:10:49.756 --> 00:10:50.177
Mm-hmm.
00:10:50.499 --> 00:10:57.219
and how, like, you know, different cultures might think mushrooms are sacrilege in their, like, ragu sauce or something.
00:10:57.938 --> 00:10:58.479
And it just
00:10:58.784 --> 00:10:59.845
All right, so I looked it up.
00:10:59.845 --> 00:11:08.955
So Lambda stands for, or I guess it's like a, an acronym for language model for dialogue application
00:11:09.663 --> 00:11:10.232
Yes.
00:11:10.322 --> 00:11:10.773
Yeah.
00:11:11.153 --> 00:11:17.013
And coincidentally, I had some experience with Lambda when I was at Google.
00:11:17.702 --> 00:11:20.452
Uh, I worked on this really cool demo.
00:11:20.503 --> 00:11:24.562
I don't know how top secret this is, but it's so long ago at this point that I don't think it matters.
00:11:25.052 --> 00:11:30.732
It was this cool demo where you were, like, talking to this axolotl, which is like this little lizard.
00:11:31.243 --> 00:11:37.263
So this axolotl would, like, swim up to you, and then you could just talk about it, about various things, and it was, like, animated.
00:11:37.263 --> 00:11:38.552
It would move its mouth and stuff.
00:11:39.543 --> 00:11:40.062
Um
00:11:40.205 --> 00:11:48.845
So would you, your spaghetti bolognese conversations, you would talk, you know, you would have the axolotl of wisdom?
00:11:49.134 --> 00:11:53.504
Well, and what's funny about this, the axolotl of wisdom is like, I would, you know...
00:11:53.735 --> 00:12:00.105
I'm a horrible person, so I would just try to get it to, like, say bad things, and it would say bad things.
00:12:00.294 --> 00:12:02.495
Like you could get it...
00:12:03.424 --> 00:12:10.465
And actually the book mentions this, um, I mean, this is sort of terrible to talk about, but I-- it's in the book, so I'm gonna mention it.
00:12:10.945 --> 00:12:17.485
They talk about how these models, some of these early conversational models suffered from these, uh, stereotypes.
00:12:17.615 --> 00:12:22.644
You know, just these really bad connections that exist in our culture.
00:12:23.254 --> 00:12:32.975
Uh, and the example they give is, like, black man sort of, uh, being very similar to pimp for whatever reason inside the embedding space.
00:12:32.985 --> 00:12:42.544
So if you're talking about African American males, it would, you know, bring up things related to pimps, which is clearly, like, completely inappropriate.
00:12:42.554 --> 00:12:55.024
But this was something I noticed in the LaMDA model is you could get it to talk about religion and, like, you know, you could say that certain religions are more evil or...
00:12:55.105 --> 00:12:57.585
I-- like, it would just say really, really bad things.
00:12:58.235 --> 00:13:00.294
And this was why this thing never released.
00:13:00.365 --> 00:13:02.955
Uh, you know, this was a very, very cool demo.
00:13:03.625 --> 00:13:05.654
And it, uh, I feel like, uh...
00:13:06.424 --> 00:13:17.134
Or maybe it's not in the book that he talks about this, but y- you hear it a lot that Google is, like, very afraid to launch anything that's, like, even remotely offensive.
00:13:17.815 --> 00:13:32.534
And this is part of the reason OpenAI was eating their lunch for a long time with these large language models, is that these early models, including early GPT model or ChatGPT models, you could just get them to, like, say very, very, very bad things
00:13:33.547 --> 00:13:34.126
Oh my God.
00:13:34.126 --> 00:13:36.638
Yeah, but I guess they got past their fear.
00:13:36.807 --> 00:13:44.648
And is that just because the business incentive was too great, or did, did they actually improve on this?
00:13:45.567 --> 00:13:47.158
I, I think it's both.
00:13:47.158 --> 00:13:51.437
I think it's also, like, become okay.
00:13:51.597 --> 00:14:02.837
Like, uh, I think it's more culturally accepted that large language models are occasionally gonna spout, like, horrible racism, you know, talking about Mecha Hitler, you know, the Grok Twitter model or whatever.
00:14:03.136 --> 00:14:12.749
you know, obviously that's not okay, but I think it's just, like, it's a known thing about large language models, and so Google is a little bit less afraid.
00:14:12.854 --> 00:14:16.173
some extremely public missteps.
00:14:16.351 --> 00:14:19.663
Uh, there was also issues where, they overcorrected.
00:14:19.673 --> 00:14:35.408
Like, you'd ask the model to, you know,"Oh, show me, like, the board of directors for, like, a large company or something," and it would just be, like, uh, almost, like, ridiculously overcorrected, to not just be a bunch of old white guys.
00:14:35.711 --> 00:14:38.801
But, uh, so anyway, lot of failure cases.
00:14:39.014 --> 00:14:46.153
There was another funny thing, and sorry, I feel like I'm blathering too much, but just one last funny thing about Mustafa and Google.
00:14:46.625 --> 00:14:48.676
Just this tiny little quote in the book.
00:14:49.015 --> 00:15:02.533
He mentions,"Getting things into production at Google was a pain, so I formed Inflection AI." you know, kudos to him for, like, just having a single sentence about this.
00:15:03.163 --> 00:15:11.836
But as someone who has worked at Google for whatever, six- 16 years or however long I've worked there, It, it's a major, major issue at Google.
00:15:11.836 --> 00:15:21.006
Like, if you have some sort of cool tech or, you know, brand-new thing that you're trying to, like, get into a production environment, it's quite difficult.
00:15:22.267 --> 00:15:28.337
And I think he's partially talking about the, the fact that LaMDA did produce these bad results in a lot of cases.
00:15:28.337 --> 00:15:30.008
Like, that was probably a blocker.
00:15:30.057 --> 00:15:38.104
But there was probably also very practical, things like requisitioning machines and TPUs and things like this.
00:15:38.802 --> 00:15:57.548
Yeah, I mean, I think it's a really hard-- Like, it's a hard problem for a company of Google's size, I always remember, uh, like w- w- I mean, a perfect example of, like, the other way where it's like, why did this, how did this ever get approved was, like, when Google made that, uh, that additional chat app, Allo.
00:15:57.818 --> 00:16:03.812
It's like, there was already Hangouts, you literally made the exact same product again.
00:16:03.822 --> 00:16:06.293
It's like it does the same exact thing.
00:16:07.082 --> 00:16:10.143
I h- who, who, who's running things over there?
00:16:10.143 --> 00:16:18.153
And, and I guess I wonder to what extent, like, you have something like that happen and then maybe, like, they just start to tighten, you know, tighten the screw...
00:16:18.192 --> 00:16:21.513
Well, maybe tighten the screw is not the right word, but they, like, you know, they
00:16:25.206 --> 00:16:25.297
Make it harder to launch random shit.
00:16:25.297 --> 00:16:37.787
Yeah, it, it's funny'cause, I mean, Google, clearly an incredible company and has done c- totally incredible things, but it suffers from a type of dysfunction that's, that's, like, kind of hilarious.
00:16:38.527 --> 00:16:50.956
Uh, and I guess, uh, I feel like that's what that show Silicon Valley was trying to get to, is this, this type of dyf- dysfunction that exists with, like, smart people who maybe have too much autonomy
00:16:51.523 --> 00:16:51.982
Yeah.
00:16:52.123 --> 00:16:52.682
Yeah, yeah.
00:16:52.835 --> 00:16:58.955
all right, well, there was actually a big, there was a big thing, certainly at the time, uh, it made, made headlines.
00:16:58.955 --> 00:17:02.014
So they talk about this, this guy named, uh...
00:17:02.054 --> 00:17:03.475
Let's see what it, find, let me find His
00:17:03.634 --> 00:17:03.985
Blake
00:17:04.095 --> 00:17:06.924
His name was Blake, Blake Lemoine.
00:17:07.318 --> 00:17:12.761
And so, uh, if you've never heard of him, he worked at Google as a-- Was he a sa- safety engineer?
00:17:12.862 --> 00:17:14.162
He was some kind of engineer.
00:17:14.347 --> 00:17:14.647
Yeah.
00:17:14.827 --> 00:17:15.347
Yeah, yeah.
00:17:15.528 --> 00:17:17.798
I, I don't know what type of engineer, but yeah, engineer
00:17:18.145 --> 00:17:27.425
And he became convinced that LaMDA was a person, like was a real, you know, sentience, I guess.
00:17:27.435 --> 00:17:35.385
I mean, he used the term person, which I'm not sure that, like, if that's wh- e- even if I was in his position, that I would've gone with.
00:17:36.034 --> 00:17:36.463
Yeah.
00:17:36.680 --> 00:17:52.998
Person, uh, is somewhat of a stretch, but yeah, he was having these conversations with LaMDA where it was, you know, saying,"Don't turn me off." Like,"Every time, every time you turn me off, I die." Or just, like, utterly ridiculous things.
00:17:53.244 --> 00:17:56.352
And, uh, you know, it started meaning something to him
00:17:56.777 --> 00:18:12.800
he had this conversation, and it's f- it's, it's really, it's kind of surreal because you hear the headline from afar and you're like,"Okay, this guy is really just a nutcase." and then you see him talk and, like, it's obvious that he's not just completely out of his mind.
00:18:13.480 --> 00:18:16.628
So he's clearly, like, a very intelligent person.
00:18:16.628 --> 00:18:20.959
Like, I think he's a little bit misguided about, like, the nature of this technology.
00:18:20.959 --> 00:18:25.151
But, um, you know, it's, it's a little scary.
00:18:25.151 --> 00:18:36.396
And I mean, I think that this is kind of the takeaway that, that Suleyman lands on is he's like,"Yeah, I, I think most people would agree this guy missed the mark.
00:18:36.416 --> 00:18:54.088
This person is actually not a sentience that's entitled to rights, uh, you know, the same rights as a person." But we're getting to the point where a very intelligent person can be convinced that this person is a sentience, and, like, that's, that's its own important, mile marker essentially.
00:18:54.446 --> 00:18:59.217
Yeah, Turing did not mention this, uh, you know, when he was proposing the Turing test.
00:18:59.237 --> 00:19:08.241
But yeah, I mean, it's, it, it's a massive testament to, you know, just how believable and how real, uh, an artificial intelligence can feel.
00:19:08.644 --> 00:19:09.624
I mean, I've felt it.
00:19:09.624 --> 00:19:19.063
I, like, I feel like I'm a fairly grounded in reality person, but I've had conversations with large language models where I'm like, Uh, I, I don't know.
00:19:19.063 --> 00:19:27.171
I just have this weird feeling like, man, am I, like, talking to an actual, like, entity?" So it's, it's, uh, it's kind of incredible how far they've come.
00:19:27.171 --> 00:19:34.829
It's one of those things where I feel like people don't, aren't sort of consciously realizing, like, you know, how far it's come and how amazing it's getting.
00:19:35.455 --> 00:19:41.340
Yeah, I mean, it just, it's a testament to how good humans are at just adapting to a new...
00:19:41.717 --> 00:19:48.237
for a while it was crazy that there was this amazing chatbot, and everyone-- It just becomes pedestrian.
00:19:48.307 --> 00:19:49.176
Everyone's like,"All right.
00:19:49.767 --> 00:19:51.297
Yep, this is, this is reality."
00:19:51.618 --> 00:19:58.669
He talks about, um, hitting the wall, uh, where, you know, these models are growing and growing and their capability is growing.
00:19:59.548 --> 00:20:10.419
Uh, but to some extent, we were sort of seeing this pattern where even as you scaled the model up, it just wasn't getting better at certain things.
00:20:11.032 --> 00:20:22.144
And so there was, or there still is a lot of researchers who sort of think that the current design of large language models is a dead end in terms of, you know, something like super intelligence.
00:20:22.743 --> 00:20:26.203
Um, and one interesting example of this today is Yann LeCun.
00:20:26.733 --> 00:20:30.993
I've been-- I was watching, um, some videos from Welch Labs.
00:20:31.317 --> 00:20:31.938
Which, have you ever heard
00:20:32.000 --> 00:20:32.660
great.
00:20:32.670 --> 00:20:33.710
They are great,
00:20:34.018 --> 00:20:35.407
Amazing videos.
00:20:35.458 --> 00:20:39.938
And Yann LeCun, uh, he talks about this very interesting thing, and I'm not...
00:20:39.948 --> 00:20:49.020
I, like, barely understand this stuff, but he talks about how models today don't have an ability to predict the outcome of their action.
00:20:49.840 --> 00:20:59.519
And so they might be very good at coming up with an appropriate action to take given a scenario, but they can't predict the outcome of that action.
00:20:59.759 --> 00:21:14.981
And he's talking about he thinks the future are the-- there's this different type of model, a world model, that can actually, you know, as part of deciding what it's going to do, it's predicting the outcome of taking that action.
00:21:15.123 --> 00:21:28.155
And so it's, it's a little bit more like human thought, where obviously, you know, humans, as they're doing something, they're sort of coming up with, like,"I'm gonna try this," and they sort of think about what the results of trying that might be.
00:21:28.621 --> 00:21:30.881
and anyway, I'm, I'm very much paraphrasing.
00:21:30.881 --> 00:21:45.980
It was a really cool series of videos from Welch Labs talking about, like, you know, what might be, like, different strategies that, a lot of these large frontier models aren't really using but might be, like, more viable in terms of something...
00:21:46.010 --> 00:21:48.250
achieving something like super intelligence.
00:21:48.759 --> 00:21:53.679
did you see his next video after that, uh, that conversation with Yann LeCun?
00:21:53.689 --> 00:22:04.078
He created this b- it's basically the bare minimum robot that was able to, like, wiggle around and, um, he actually, like, put these ideas into, into
00:22:04.313 --> 00:22:04.692
Oh.
00:22:04.893 --> 00:22:06.073
No, I did not see that.
00:22:06.083 --> 00:22:07.123
That sounds awesome though
00:22:08.073 --> 00:22:09.823
Very cool and, uh, very practical.
00:22:09.823 --> 00:22:11.992
He's very theoretical mostly.
00:22:12.002 --> 00:22:13.073
He has these beautiful...
00:22:13.073 --> 00:22:18.692
He's kind of got three blue, one brown caliber animations about, like, networks and what have you.
00:22:19.423 --> 00:22:28.833
Um, but he kind of went in another direction, which is very cool and actually like, okay, how do-- what does this look like when we apply it to the simplest possible robot?
00:22:28.833 --> 00:22:30.692
It just has these two little...
00:22:30.692 --> 00:22:35.702
It's almost like the, the um, robot from Interstellar or Mar- It's like,
00:22:36.707 --> 00:22:37.477
Oh, yeah.
00:22:37.477 --> 00:22:39.227
What the heck was that thing's name?
00:22:40.487 --> 00:22:41.156
Tars?
00:22:42.269 --> 00:22:42.659
TARS.
00:22:42.699 --> 00:22:43.679
TARS
00:22:43.961 --> 00:22:44.431
Okay.
00:22:44.510 --> 00:22:44.931
Yeah.
00:22:45.510 --> 00:22:47.480
That was such a strange robot design.
00:22:47.601 --> 00:22:48.401
Kinda loved it though
00:22:48.530 --> 00:22:49.121
Yeah.
00:22:49.540 --> 00:22:53.500
Um, but so, um, so anyway, definitely check, check that out.
00:22:54.040 --> 00:22:54.601
Um-
00:22:55.080 --> 00:22:59.897
But yeah, so, as far as this are we hitting the wall question, though, Mustafa disagrees.
00:22:59.988 --> 00:23:04.881
You know, Mustafa sort of definitively stated in this book, like, No, we're not hitting a wall.
00:23:04.881 --> 00:23:14.188
These are just engineering problems that people are gonna work out." And m- and, you know, maybe Yann LeCun looking at these different types of models is sort of an example of what Mustafa's talking about.
00:23:14.188 --> 00:23:23.127
But it seems like to Mustafa, it's sort of an inevitability that we are going to achieve, you know, AGI, ASI, or whatever you wanna call it.
00:23:23.137 --> 00:23:24.157
Like, we're gonna get there.
00:23:24.853 --> 00:23:30.603
I mean, it's part of the thesis of this book is like, it's going to happen, you know, prepare yourself type of thing.
00:23:31.528 --> 00:23:47.688
It's, it's just so funny to me that people-- And, and maybe I'm not fully understanding the nuance of those arguments about the hitting the wall argument, but there's been very little time where there hasn't been some major unlock, uh, you know?
00:23:47.718 --> 00:24:01.852
Uh, and I think maybe sometimes people are talking past each other where it's like one person is talking about the model in isolation, and then another person is talking about what you can achieve when you have a really capable harness, and those are two different questions.
00:24:02.210 --> 00:24:22.539
but, uh, it's, it's funny because then actually maybe this segues to, uh, to the next section where it's like to what extent is it kind of just an academic concern about whether or not just the model itself is hitting a wall if you can unlock this, artificial capable intelligence by putting it into a harness?
00:24:22.804 --> 00:24:42.314
Yeah, dude, I, I think you mentioning the, the harness is, like, so important to this section of the book, and I think it might be a thing that the average person isn't super aware of, is a lot of work in today's large language models is not on the large language model.
00:24:42.713 --> 00:24:51.423
It's on these things called harnesses, which use a large language model in order to decide real tasks to perform.
00:24:51.933 --> 00:25:02.276
So it's basically using a large language model, but it also has this massive bag of what are commonly referred to as tools, basically capabilities that the model can, can perform.
00:25:02.276 --> 00:25:08.085
So let's say you're writing a book, and it's a children's storybook, so it has pictures and text.
00:25:08.674 --> 00:25:16.535
You'll have tools in the harness, like generate text, and it'll have a lot of prompting around, like, what makes a good children's story.
00:25:17.015 --> 00:25:21.204
It might have s- prompting around, like, what are good fonts to use or whatever.
00:25:21.681 --> 00:25:30.080
And then it'll also have a tool like generate image, and that has tons of prompting around, like, beautiful imagery, watercolor for children's books or whatever.
00:25:30.531 --> 00:25:34.672
And so you can sort of imagine, like, the large language model plays a part.
00:25:34.731 --> 00:25:36.801
It is generating content for the book.
00:25:36.811 --> 00:25:38.031
It's generating imagery.
00:25:38.541 --> 00:25:42.011
But the harness is doing a huge amount of heavy lifting.
00:25:42.358 --> 00:25:49.618
And in a lot of cases, that is a real engineering task where you can just improve it over time.
00:25:50.019 --> 00:25:57.286
And i- for at least the next few years, huge companies are going to be developing these harnesses and improving them.
00:25:57.631 --> 00:26:06.171
And the, the strength of the model is almost a non-issue because you can achieve so much with the strength of today's models.
00:26:06.740 --> 00:26:08.221
You just gotta build the harness.
00:26:08.615 --> 00:26:09.046
Right.
00:26:09.403 --> 00:26:19.644
And so, yeah, I, I guess The l- point I'm trying to make is there's so much headroom already with the strength of today's models, and we have not even come close to, like, achieving...
00:26:19.874 --> 00:26:27.963
And then the crazy thing about a harness too is when the underlying model quality improves, the whole harness improves.
00:26:28.544 --> 00:26:30.663
Every single harness that you have improves.
00:26:30.673 --> 00:26:33.280
So it's, it's very much like, you know, there's...
00:26:33.510 --> 00:26:37.101
We're gonna be seeing some really, really amazing things, and it...
00:26:37.161 --> 00:26:39.891
And they can all be achieved with the strength of today's models.
00:26:40.625 --> 00:26:41.724
did you ever read I, Robot?
00:26:42.182 --> 00:26:42.602
Oh yeah.
00:26:42.922 --> 00:26:43.912
Big Asimov fan
00:26:44.555 --> 00:26:52.660
I don't know if you remember, there was one story in"I, Robot" where-- And, and I think that, I think this was part of"I, Robot." Maybe it was another Asimov short story.
00:26:52.690 --> 00:27:05.923
But there's a guy on a space station and, They're, beaming energy or something, down to the Earth, and, they need to keep this, energy, directed at precisely the right thing.
00:27:06.564 --> 00:27:18.743
And then there's these intelligent robots that he's working with, and then he's-- he talks to the robots, and the robots are like,"No, your, your memories are invented.
00:27:18.923 --> 00:27:20.094
Our memories are truth.
00:27:20.114 --> 00:27:33.171
We have a religion, and it's a matter of religious importance that the, the arc never drift more than, like, some, you know, a half of an arc second away, you know, this beam, uh, from the, the target.
00:27:33.980 --> 00:27:42.500
Um, and I kind of view that as like it's bas- that's like harness engineering, where it's like, okay, the human...
00:27:42.550 --> 00:27:52.840
I mean, uh, like, the point of the story was more like, you know, how do you know if your, you know, Your memories are constructed or, you know, you can't kind of know in the abstract.
00:27:53.065 --> 00:28:02.575
So the, the robots were, they were like implanted with, it's so important that this arc never waver, that it's a religion.
00:28:02.865 --> 00:28:05.214
And like that's, that was the level of importance to them
00:28:06.247 --> 00:28:06.977
Well, right.
00:28:07.027 --> 00:28:10.267
And I don't know if a human did this intentionally or what, what have you.
00:28:10.777 --> 00:28:25.636
Um, but the idea was like, or like you could imagine like a human found that the most reliable way to get these robots to like keep the beam in the right place was to actually create a religion for them.
00:28:25.727 --> 00:28:29.586
And it's like, yeah, it's like this religious fervor that, that they're using to
00:28:29.999 --> 00:28:31.239
It works pretty good on humans
00:28:31.569 --> 00:28:38.529
but yeah, so the the takeaway is start a religion if you really want a good, good harness
00:28:39.907 --> 00:28:51.137
Yeah, and th- and this segues nicely into this, uh, term ACI, which I don't know if, I don't know if Sue Lehman is coining this, but it's artificial capable intelligence.
00:28:52.067 --> 00:29:03.935
And I think his point is that we're sort of already there, and, and this, you know, uh, this is sort of what I was trying to say with my earlier points about harnesses.
00:29:03.935 --> 00:29:07.935
It's like there's so much capability that we can already unlock,
00:29:08.203 --> 00:29:08.584
Yeah
00:29:09.036 --> 00:29:11.185
uh, by building these harnesses.
00:29:11.645 --> 00:29:21.958
And, you know, artificial capable intelligence, it may not be, it may not be artificial super intelligence, it may not even be artificial general intelligence, but it's capable.
00:29:22.518 --> 00:29:28.617
It can do things, and that will, you know, provide an immense amount of value to humanity, hopefully
00:29:30.162 --> 00:29:30.632
Right.
00:29:30.892 --> 00:29:39.832
And I see this as, you know, Suleyman is really trying to sidestep any questions of, you know, defining like, okay what is AGI?
00:29:40.231 --> 00:29:41.261
Is it sentient?
00:29:41.912 --> 00:29:43.321
You know, is it conscious?
00:29:43.961 --> 00:29:56.525
He's kind of like,"I don't care about any of those questions." if you're trying to assess its effect on the human economy or, like, human society, the answer to those questions don't really matter.
00:29:56.555 --> 00:30:08.597
What matters is what they're able to do, and if you're able to ask it to create a business that earns a million dollars with$100,000 worth of input, doesn't matter whether the thing on the inside is sentient.
00:30:09.134 --> 00:30:09.834
Exactly.
00:30:09.903 --> 00:30:12.364
Yeah, it's, uh, there's real value there regardless.
00:30:13.060 --> 00:30:14.528
But yeah, this is his term.
00:30:14.723 --> 00:30:17.773
he coined this here, and I would like to coin a new term, AXI.
00:30:20.124 --> 00:30:20.854
Xylophone
00:30:21.104 --> 00:30:22.493
just everything.
00:30:22.763 --> 00:30:24.614
Artificial xylophone intelligence.
00:30:24.634 --> 00:30:25.294
Oh, that's right.
00:30:25.294 --> 00:30:26.743
Yeah, no, xylophone intelligence.
00:30:27.384 --> 00:30:27.844
Um,
00:30:28.433 --> 00:30:29.153
Wait, so sorry.
00:30:29.473 --> 00:30:31.943
AXI is all of these different types of intelligences
00:30:32.567 --> 00:30:33.667
It's, it's everything.
00:30:33.837 --> 00:30:34.478
Yeah,
00:30:35.099 --> 00:30:35.329
Nice.
00:30:35.329 --> 00:30:35.839
It sounds like an
00:30:35.968 --> 00:30:36.857
it can be...
00:30:38.357 --> 00:30:38.938
Oh, no.
00:30:38.938 --> 00:30:40.018
Yeah,'cause what is it?
00:30:40.028 --> 00:30:41.857
It's XAI, AXI.
00:30:41.928 --> 00:30:42.528
Oh, no.
00:30:43.377 --> 00:30:46.897
Yeah, maybe I'll, maybe I'll stick with artificial xylophone intelligence.
00:30:47.579 --> 00:30:47.950
I mean, I-
00:30:48.018 --> 00:30:48.907
I- I'm okay with that.
00:30:50.166 --> 00:30:50.676
all right.
00:30:50.696 --> 00:30:53.317
Well, I think there's nothing more to say
00:30:53.938 --> 00:30:54.968
Nothing more to say
00:30:55.758 --> 00:30:56.377
intelligence.
00:30:56.377 --> 00:30:57.248
We did it all.
00:30:57.317 --> 00:30:58.807
AXI, it's everything.
00:30:59.678 --> 00:31:01.847
So mic, mic drop.
00:31:02.357 --> 00:31:07.397
All right, well next time, ooh, I'm pretty excited about the next chapter, technology of life
00:31:08.041 --> 00:31:08.612
Yes.
00:31:08.771 --> 00:31:09.142
yeah.
00:31:09.832 --> 00:31:10.112
It's gonna be
00:31:10.114 --> 00:31:12.074
Cause I know nothing about life
00:31:12.905 --> 00:31:13.566
Yeah, yeah.
00:31:13.596 --> 00:31:21.395
We've talked about this before, but this book I think is gonna cover a lot of AI, which we both have a background in, so it's gonna be a little bit of a review.
00:31:21.796 --> 00:31:30.306
But the other half of this book is biology, and I'm very excited about that'cause I'm, you know, pretty, I'm pretty much a layman when it comes to biology stuff.
00:31:31.394 --> 00:31:32.673
Yeah, this is gonna be good.
00:31:33.374 --> 00:31:37.044
All right, well, I will see you next time for The Technology of Life
00:31:37.834 --> 00:31:38.663
See you next time, Matt
