168 - How to Solve AI Alignment with Paul Christiano

WEBVTT
1
00:00:00.080 --> 00:00:03.880
The most likely way we die involves like not AI comes out of the blue and kills everyone,
2
00:00:04.020 --> 00:00:05.760
but involves we have deployed a lot of AI everywhere.
3
00:00:06.300 --> 00:00:09.760
And you can kind of just look and be like, oh yeah, if for some reason, God forbid, all
4
00:00:09.800 --> 00:00:12.259
these AI systems were trying to kill us, they would definitely kill us.
5
00:00:15.040 --> 00:00:20.020
Welcome to Bankless, where we explore frontier technologies, artificial intelligence on the
6
00:00:20.061 --> 00:00:20.780
episode today.
7
00:00:21.240 --> 00:00:24.680
This is how to get started, how to get better, and how to front run the opportunity.
8
00:00:25.120 --> 00:00:26.060
This is Ryan Sean Adams.
9
00:00:26.160 --> 00:00:30.240
I'm here with David Hoffman, and we're here to help you become more bankless.
10
00:00:30.380 --> 00:00:33.160
Guys, we have a special guest in the episode today, Paul Cristiano.
11
00:00:33.720 --> 00:00:39.680
This is who Eliezer Yudkowsky told us to go talk to, someone he respects on the AI debate.
12
00:00:40.000 --> 00:00:40.840
So we picked his brain.
13
00:00:40.960 --> 00:00:43.740
This is an AI safety alignment researcher.
14
00:00:44.301 --> 00:00:47.540
We asked the question, how we stop the AIs from killing us?
15
00:00:47.700 --> 00:00:51.120
Can we prevent the AI takeover that others are very concerned about?
16
00:00:51.240 --> 00:00:53.660
There are three, actually four takeaways for you today.
17
00:00:54.000 --> 00:00:57.303
Number one, how big is the AI alignment problem?
18
00:00:57.543 --> 00:00:58.804
We ask Paul this question.
19
00:00:58.884 --> 00:01:02.166
Number two, how hard is it to actually solve this problem?
20
00:01:02.526 --> 00:01:04.828
Number three, what are the ways we solve it?
21
00:01:05.028 --> 00:01:05.929
The technical ways?
22
00:01:05.969 --> 00:01:08.330
Can we coordinate around this to solve it?
23
00:01:08.791 --> 00:01:18.498
And finally, we talk about a possible optimistic scenario where we live in harmony with the AIs and they improve our lives and make it quite a bit better.
24
00:01:19.198 --> 00:01:21.980
David, what was the significance of this episode to you in our series?
25
00:01:22.842 --> 00:01:31.800
Yeah, the new intro I think is great when we cover frontier technologies and also instead of for this episode helping you become more bankless.
26
00:01:32.362 --> 00:01:36.000
I think this is a truly a front running the opportunity on this one.
27
00:01:36.300 --> 00:01:39.276
We are trying to help humanity not not die.
28
00:01:40.402 --> 00:01:41.392
We're trying to help you not die.
29
00:01:42.945 --> 00:01:43.839
All of us not die.
30
00:01:44.120 --> 00:01:44.438
All of us.
31
00:01:44.620 --> 00:01:46.500
Yeah, you and the rest of the world.
32
00:01:46.942 --> 00:01:52.360
And I think we're doing this in the best way that we can, which is education and awareness about this AI problem.
33
00:01:52.940 --> 00:02:01.520
Paul Cristiano is, like you said, the man that was recommended by Eliezer about the man approaching this problem head on in a technical way.
34
00:02:01.640 --> 00:02:12.940
So in this podcast, you are going to hear about the technical solutions that people are actively working on who are taking this problem extremely seriously and take the risks very, very seriously.
35
00:02:13.161 --> 00:02:13.582
seriously.
36
00:02:14.023 --> 00:02:17.615
Eliezer gave us more or less a doomsday scenario, like a 99% chance of doom.
37
00:02:20.241 --> 00:02:24.696
Paul Cristiano only gives us a 10 to 20% case of doom scenario.
38
00:02:24.716 --> 00:02:25.920
So much more optimistic.
39
00:02:26.180 --> 00:02:26.560
Like those odds.
40
00:02:27.881 --> 00:02:28.663
Much better odds.
41
00:02:29.205 --> 00:02:33.757
And so we go through why he's still very much concerned.
42
00:02:33.898 --> 00:02:34.700
And he does consider
43
00:02:34.780 --> 00:02:39.376
this the most likely way in which he dies in the future, yet why there's still an 80% chance of
44
00:02:40.941 --> 00:02:41.323
success.
45
00:02:41.804 --> 00:02:45.377
And some of that 80% chance of success actually does have utopia in it.
46
00:02:45.899 --> 00:02:46.280
I think maybe
47
00:02:46.360 --> 00:02:52.820
Ryan in three, five, 10 years, we're going to be able to look back at this episode as hopefully,
48
00:02:53.181 --> 00:02:57.740
if Paul's right, and I think he's right, like ahead of its time in terms of elevating extremely
49
00:02:57.800 --> 00:03:01.315
important conversations to the best of our ability into the mainstream.
50
00:03:01.355 --> 00:03:02.400
So we can get more people to
51
00:03:02.460 --> 00:03:08.698
focus on the actual solution paths to make sure that that 20% risk of doomsday goes down to 0.2%
52
00:03:09.580 --> 00:03:10.423
risk of doomsday.
53
00:03:11.026 --> 00:03:15.080
And I see that's what the significance of this episode has and why we are
54
00:03:15.140 --> 00:03:16.025
doing episodes like this.
55
00:03:16.045 --> 00:03:16.206
Yeah.
56
00:03:16.226 --> 00:03:18.820
I mean, to be clear, Paul really thinks this is AI alignment
57
00:03:18.900 --> 00:03:22.480
is a solvable problem, which is much, much different than, than others in the space.
58
00:03:22.540 --> 00:03:23.926
And he tells us exactly why.
59
00:03:23.967 --> 00:03:27.000
And my takeaway from the Eliezer episode was humanity is screwed.
60
00:03:27.721 --> 00:03:30.734
This was a humanity is screwed, but we're working on it.
61
00:03:31.156 --> 00:03:32.020
We might have some solutions.
62
00:03:32.520 --> 00:03:34.167
Have clear actionable paths.
63
00:03:34.528 --> 00:03:35.010
That's right.
64
00:03:35.191 --> 00:03:36.918
So we get into all of that today.
65
00:03:37.079 --> 00:03:37.400
And David,
66
00:03:37.480 --> 00:03:46.680
Of course, I want to discuss this episode in the debrief with you and hear what you think, because this is our third in a series of episodes and really interesting material.
67
00:03:46.800 --> 00:03:52.620
The debrief episode is our episode that we record directly after the episode with our raw unfiltered thoughts.
68
00:03:52.700 --> 00:03:57.460
If you are a bankless citizen, you have access to that right now on the premium RSS feed.
69
00:03:57.822 --> 00:04:00.220
You can click a link in the show notes and get access to that.
70
00:04:00.600 --> 00:04:02.680
OK, guys, we're going to get right to the episode with Paul.
71
00:04:02.740 --> 00:04:09.459
But before we do, we want to thank the sponsors that made this episode possible, including Kraken, our recommended crypto exchange for 2023.
72
00:04:10.300 --> 00:04:13.840
Kraken has been a leader in the crypto industry for the last 12 years.
73
00:04:14.060 --> 00:04:24.420
Dedicated to accelerating the global adoption of crypto, Kraken puts an emphasis on security, transparency, and client support, which is why over 9 million clients have come to love Kraken's products.
74
00:04:24.681 --> 00:04:28.900
Whether you're a beginner or a pro, the Kraken UX is simple, intuitive, and frictionless,
75
00:04:29.120 --> 00:04:33.060
making the Kraken app a great place for all to get involved and learn about crypto.
76
00:04:33.280 --> 00:04:37.440
For those with experience, the redesigned Kraken Pro app and web experience is completely
77
00:04:37.581 --> 00:04:42.760
customizable to your trading needs, integrating key trading features into one seamless interface.
78
00:04:42.980 --> 00:04:47.580
Kraken has a 24-7, 365 client support team that is globally recognized.
79
00:04:47.780 --> 00:04:52.220
Kraken support is available wherever, whenever you need them, by phone, chat, or email.
80
00:04:52.661 --> 00:04:59.460
And for all of you NFTers out there, the brand new Kraken NFT beta platform gives you the best NFT trading experience possible.
81
00:04:59.640 --> 00:05:03.900
Rarity rankings, no gas fees, and the ability to buy an NFT straight with cash.
82
00:05:04.120 --> 00:05:07.560
Does your crypto exchange prioritize its customers the way that Kraken does?
83
00:05:07.680 --> 00:05:11.200
And if not, sign up with Kraken at kraken.com slash bankless.
84
00:05:11.580 --> 00:05:14.180
Bankless is launching the Bankless Token Hub.
85
00:05:14.580 --> 00:05:20.019
At Bankless, we've been studying the crypto markets ever since 2017, and all of our research has led us to this, the Token Hub.
86
00:05:20.300 --> 00:05:24.000
You're a one-stop shop for Alpha to help you navigate through the crypto markets.
87
00:05:24.180 --> 00:05:27.860
Have you ever wished for a trusted resource that would share their thoughts, ratings,
88
00:05:28.181 --> 00:05:29.980
and their opinions about tokens?
89
00:05:30.240 --> 00:05:31.820
Boy, do we have the product for you.
90
00:05:32.040 --> 00:05:36.060
The Bankless Token Hub is where we provide bankless citizens with the Alpha on the hottest
91
00:05:36.183 --> 00:05:36.920
tokens in crypto.
92
00:05:37.160 --> 00:05:38.860
We do the research so you don't have to.
93
00:05:39.000 --> 00:05:42.920
The Bankless Token Hub includes the token ratings, where our team shares their research
94
00:05:43.122 --> 00:05:45.240
and outlook on the hottest tokens in crypto.
95
00:05:45.500 --> 00:05:49.460
Also, the Token Hub includes Bankless Bags, our own internal investment club.
96
00:05:49.621 --> 00:05:52.660
Bankless Bags is where we put our money where our mouth is.
97
00:05:52.800 --> 00:05:54.500
And for the Bankless Power user out there,
98
00:05:54.640 --> 00:05:58.280
you can access the analyst team 24-7 inside the Bankless Nation Discord.
99
00:05:58.520 --> 00:06:00.980
You can ask them questions and learn from a group of people
100
00:06:01.222 --> 00:06:03.300
deep in the weeds of crypto investing.
101
00:06:03.500 --> 00:06:07.480
The last feature of the Token Hub is the ability to upvote or downvote token ratings.
102
00:06:07.720 --> 00:06:10.580
The Bankless Token Hub lets you learn from your fellow citizens
103
00:06:10.863 --> 00:06:12.320
to rate these tokens yourselves.
104
00:06:12.620 --> 00:06:14.620
The Bankless Token Hub is launching right now
105
00:06:14.741 --> 00:06:17.660
and has already been beta tested by your fellow Bankless citizens.
106
00:06:18.023 --> 00:06:20.380
So stay tuned in the Bankless Discord for updates.
107
00:06:20.520 --> 00:06:21.980
And if you're not a Bankless citizen,
108
00:06:22.220 --> 00:06:23.899
well, you better sign up if you want access
109
00:06:24.102 --> 00:06:25.380
because this corner of Bankless
110
00:06:25.603 --> 00:06:27.060
is available for citizens only.
111
00:06:27.321 --> 00:06:28.100
I'll see you in the Discord.
112
00:06:28.720 --> 00:06:30.520
If you haven't yet experienced the superpowers
113
00:06:30.601 --> 00:06:32.300
that a smart contract wallet gives you,
114
00:06:32.661 --> 00:06:33.360
check out Ambuyer.
115
00:06:33.600 --> 00:06:35.460
Ambuyer works with all the EVM chains,
116
00:06:35.640 --> 00:06:38.000
the layer twos like Arbitrum, Optimism, and Polygon,
117
00:06:38.140 --> 00:06:39.720
but also the non-Ethereum ecosystems
118
00:06:39.903 --> 00:06:40.940
like Avalanche and Phantom.
119
00:06:41.160 --> 00:06:43.040
Ambuyer lets you pay for gas and stable coins,
120
00:06:43.200 --> 00:06:45.360
meaning you'll never have to spend your precious ETH again.
121
00:06:45.581 --> 00:06:46.640
And if you like self-custody,
122
00:06:46.901 --> 00:06:50.780
but you still want training wheels, you can recover a lost Ambuyer wallet with an email
123
00:06:50.780 --> 00:06:54.577
and password, but without giving the Ambuyer team control over your funds.
124
00:06:54.597 --> 00:06:55.300
The Ambuyer wallet is
125
00:06:55.340 --> 00:06:57.489
coming soon for both iOS and Android.
126
00:06:57.550 --> 00:07:00.000
And if you want to be a beta tester, Ambuyer is airdropping
127
00:07:00.100 --> 00:07:02.551
their wallet token for simply just using the wallet.
128
00:07:02.692 --> 00:07:04.399
You can sign up at Ambuyer.com.
129
00:07:04.439 --> 00:07:04.640
And while
130
00:07:04.680 --> 00:07:07.154
you're there, sign up for the web app wallet experience as well.
131
00:07:07.335 --> 00:07:08.120
So thank you, Ambuyer,
132
00:07:08.260 --> 00:07:10.892
for pushing the frontier of smart contract wallets on Ethereum.
133
00:07:11.133 --> 00:07:12.700
Bankless Nation, I'm super excited to
134
00:07:12.740 --> 00:07:14.246
introduce you to our next guest.
135
00:07:14.266 --> 00:07:17.580
We're talking about AI alignment stuff here today because we
136
00:07:17.660 --> 00:07:18.142
can't not.
137
00:07:18.162 --> 00:07:19.307
This is Paul Cristiano.
138
00:07:19.649 --> 00:07:22.320
He runs the Alignment Research Center, which is a nonprofit
139
00:07:22.902 --> 00:07:26.940
research organization whose mission is to align future machine learning systems with human
140
00:07:27.101 --> 00:07:27.462
interests.
141
00:07:27.804 --> 00:07:29.551
Make sure the AIs don't come to kill us.
142
00:07:29.591 --> 00:07:31.519
That's what I take to be the meaning.
143
00:07:31.559 --> 00:07:31.860
And Paul
144
00:07:32.241 --> 00:07:35.895
previously ran the Language Model Alignment team at OpenAI.
145
00:07:36.055 --> 00:07:36.557
You know them.
146
00:07:36.597 --> 00:07:37.360
They're the creators of
147
00:07:37.460 --> 00:07:38.403
chat GPT.
148
00:07:38.884 --> 00:07:44.460
And today we're hoping that Paul can help us explain the solution landscape and
149
00:07:44.580 --> 00:07:48.194
understand the solution landscape of this AI alignment problem.
150
00:07:48.495 --> 00:07:49.760
Paul, welcome to Bankless.
151
00:07:50.521 --> 00:07:51.088
Thanks for having me.
152
00:07:51.149 --> 00:07:51.959
Excited to talk to you.
153
00:07:52.800 --> 00:07:57.500
So Paul, just to get some context here, David and I recorded an episode with Eliezer Yukowski.
154
00:07:57.900 --> 00:08:02.220
We thought this would be, hey, you know, Bankless's first intro, we're primarily a crypto podcast,
155
00:08:02.602 --> 00:08:04.432
but we're exploring other frontier technologies.
156
00:08:04.493 --> 00:08:05.639
Let's go dabble with AI.
157
00:08:06.061 --> 00:08:08.068
Let's just go dabble with AI.
158
00:08:08.509 --> 00:08:11.820
Crypto and AI might have some sort of match in the future.
159
00:08:12.581 --> 00:08:16.960
So we recorded this podcast and we quickly realized the agenda that we were going to use
160
00:08:17.040 --> 00:08:22.016
and talk about didn't matter anymore because Eliezer's message was pretty simple.
161
00:08:22.899 --> 00:08:23.140
We were
162
00:08:23.260 --> 00:08:24.063
all going to die.
163
00:08:24.907 --> 00:08:28.560
Basically, we were on the brink, whether it's years or months away,
164
00:08:28.921 --> 00:08:34.480
from creating some super intelligent AI that would eventually rearrange humanity's atoms and
165
00:08:34.881 --> 00:08:35.462
destroy us.
166
00:08:36.044 --> 00:08:40.816
And he felt pretty convicted on this as the likely outcome.
167
00:08:40.876 --> 00:08:42.240
So when you get a message
168
00:08:42.381 --> 00:08:45.873
like that, Paul, you got to investigate a little further.
169
00:08:46.014 --> 00:08:47.680
You have to get the second doctor's
170
00:08:47.740 --> 00:08:49.687
opinion when the prognosis is terminal.
171
00:08:50.129 --> 00:08:52.699
So that's what this series is all about.
172
00:08:52.719 --> 00:08:53.080
And we're hoping
173
00:08:53.160 --> 00:08:55.931
you can help guide us through these questions today.
174
00:08:55.992 --> 00:08:56.815
Does that sound okay?
175
00:08:57.618 --> 00:08:58.100
Happy to at least
176
00:08:58.140 --> 00:08:58.782
share my thoughts.
177
00:08:59.965 --> 00:09:01.188
I'm a little bit less gloomy.
178
00:09:01.790 --> 00:09:01.971
Yes.
179
00:09:02.071 --> 00:09:02.291
Okay.
180
00:09:02.392 --> 00:09:02.893
Okay.
181
00:09:03.033 --> 00:09:04.959
I suspected so.
182
00:09:04.999 --> 00:09:05.440
By the way,
183
00:09:05.580 --> 00:09:10.097
Eliezer said, we asked him, hey, is there someone else we can talk to about this?
184
00:09:10.258 --> 00:09:10.920
And he mentioned
185
00:09:11.020 --> 00:09:11.181
you.
186
00:09:11.241 --> 00:09:13.027
He said, talk to Paul Cristiano.
187
00:09:13.689 --> 00:09:17.260
He said you were someone he respects and brings some of the
188
00:09:17.320 --> 00:09:18.926
counterpoints to his take.
189
00:09:18.946 --> 00:09:19.929
So let's get into them.
190
00:09:20.552 --> 00:09:22.820
Why don't we just start by wading into the deep
191
00:09:22.880 --> 00:09:23.582
end of the pool here.
192
00:09:24.024 --> 00:09:28.800
What is your percentage likelihood of the full-out Eliezer-Yudkowsky
193
00:09:29.202 --> 00:09:32.220
doom scenario where we're all going to die from the machines?
194
00:09:33.422 --> 00:09:36.920
I think this question is a little bit complicated, unfortunately, because there are a lot of
195
00:09:36.980 --> 00:09:39.771
different ways we could all die from the machines.
196
00:09:40.634 --> 00:09:42.280
So the thing I most think about, and I think
197
00:09:42.320 --> 00:09:46.149
Eliezer most talks about, is the sort of full-blown AI takeover scenario.
198
00:09:48.896 --> 00:09:50.520
I take this pretty seriously.
199
00:09:50.782 --> 00:09:53.419
I think I have a much higher probability than a typical person working in ML.
200
00:09:54.320 --> 00:10:02.180
I think maybe there's something like a 10, 20% chance of AI take over many, most humans dead.
201
00:10:04.806 --> 00:10:05.800
That's really high.
202
00:10:06.764 --> 00:10:07.280
I agree.
203
00:10:07.621 --> 00:10:08.820
And then I think beyond that.
204
00:10:08.902 --> 00:10:09.620
Better than 100, David.
205
00:10:09.860 --> 00:10:10.738
Just better than 100%.
206
00:10:10.922 --> 00:10:12.160
We're trending in the right direction.
207
00:10:13.380 --> 00:10:17.280
I think there are other, yeah, I think in some sense I'm still quite a gloomy person.
208
00:10:17.401 --> 00:10:19.680
So there's other ways that the development of AI can be rough.
209
00:10:19.980 --> 00:10:24.300
Like there's other ways that you can have access to new destructive physical technology, other disruption.
210
00:10:25.722 --> 00:10:29.100
So I think you're maybe looking at some other risks from that transition to AI.
211
00:10:30.203 --> 00:10:31.897
And that adds up to at least another 10%.
212
00:10:34.325 --> 00:10:37.200
And then maybe a bigger background part of both my view and Eliezer's view.
213
00:10:37.340 --> 00:10:41.440
I think Eliezer is into this extremely fast transformation once you develop AI.
214
00:10:42.401 --> 00:10:46.200
I am a little bit, have a little bit less of an extreme view on that.
215
00:10:46.260 --> 00:10:52.600
But I still think it is the case that compared to what is kind of default expectations in the world, things are going to be really fast.
216
00:10:53.141 --> 00:10:58.320
So we could talk about the development of AI, but then you might also want to talk about what happens right over the coming months or years.
217
00:10:59.101 --> 00:11:07.640
I tend to imagine something more like a year's transition from AI systems that are a pretty big deal to kind of accelerating change, followed by further acceleration, et cetera.
218
00:11:08.340 --> 00:11:13.860
I think once you have that view, then sort of a lot of things may feel like AI problems because they happen very shortly after you build AI.
219
00:11:14.967 --> 00:11:17.220
like your AI builds new AI systems, your society keeps changing.
220
00:11:17.320 --> 00:11:21.000
Anyway, so overall, you know, maybe you're getting more up to like 50-50 chance of doom
221
00:11:21.362 --> 00:11:23.820
shortly after you have AI systems that are at human level.
222
00:11:24.601 --> 00:11:24.706
Okay.
223
00:11:25.891 --> 00:11:26.080
Okay.
224
00:11:26.286 --> 00:11:26.560
All right.
225
00:11:26.821 --> 00:11:31.120
Well, let's start with that speed conversation first.
226
00:11:31.200 --> 00:11:35.920
Let's start with the takeoff velocity question, because I think that's something that really
227
00:11:36.301 --> 00:11:41.840
the AI alignment, doomerism perception really, really depends on.
228
00:11:42.826 --> 00:11:45.220
if we do believe that AIs are going to be developed
229
00:11:45.301 --> 00:11:47.000
and they're magically going to become sentient,
230
00:11:47.140 --> 00:11:49.960
not magically, but somehow pragmatically.
231
00:11:50.280 --> 00:11:50.759
It feels like magic.
232
00:11:51.546 --> 00:11:52.820
It feels like magic to humans.
233
00:11:53.864 --> 00:11:55.700
It's going to be because it happens really fast.
234
00:11:56.261 --> 00:11:58.860
And I want to actually try and measure that speed
235
00:11:58.961 --> 00:12:01.620
because fast and slow is like relative, right?
236
00:12:01.740 --> 00:12:04.120
And so the takeoff scenario
237
00:12:04.381 --> 00:12:08.740
that some AGI super intelligence explosions
238
00:12:09.829 --> 00:12:11.180
people articulate is that
239
00:12:11.340 --> 00:12:15.431
as soon as some sort of AI can update itself,
240
00:12:19.241 --> 00:12:20.340
it's like a lightning flash.
241
00:12:20.420 --> 00:12:21.460
It's like a snap of the fingers.
242
00:12:22.042 --> 00:12:23.440
It happens in a blink of an eye.
243
00:12:23.741 --> 00:12:25.658
One day we have chat GPT-7
244
00:12:26.484 --> 00:12:28.760
and then the next day we have an AI takeover.
245
00:12:28.981 --> 00:12:30.300
And that's like the super fast scenario.
246
00:12:30.645 --> 00:12:31.639
I think what you're saying is like,
247
00:12:32.241 --> 00:12:35.000
yeah, pretty quickly, but still not like lightning fast.
248
00:12:35.080 --> 00:12:36.860
I think what you're saying is like, give it a year.
249
00:12:37.342 --> 00:12:39.660
Maybe you can help unpack like the timing.
250
00:12:39.761 --> 00:12:41.780
How do we understand about time around this whole thing?
251
00:12:42.620 --> 00:12:51.060
Yeah, this is one of my most pronounced disagreements with Eliezer, where we've really went back and forth about it a lot over the last, like, Jesus, I don't know, 12 years.
252
00:12:51.822 --> 00:12:54.660
And I think still do not see at all eye to eye.
253
00:12:55.341 --> 00:12:57.500
That said, my view is pretty fast.
254
00:12:57.720 --> 00:13:02.020
So I think, like, how I would think about this, maybe there's like kind of two parts to my answer.
255
00:13:02.701 --> 00:13:06.299
So one is sort of based on how fast things are currently moving in AI.
256
00:13:07.440 --> 00:13:16.200
So the way you could think about it is if you were to try and measure, right, if you're trying to say, suppose you have AI doing some job and has some level of competence at that job this year, how good is it going to be at this job next year?
257
00:13:17.042 --> 00:13:18.200
And how fast is that changing?
258
00:13:18.640 --> 00:13:27.320
So if we're looking at the world and we're like every day, AI is much smarter than the day before, then you kind of are going to expect by default a fast transition over a scale of something like days because you're going to have AI systems that are weak.
259
00:13:27.380 --> 00:13:29.240
And then a couple of days later, you have AI systems that are strong.
260
00:13:30.163 --> 00:13:35.740
I think the way I would describe the situation now is more like timescale of like a year or a couple of years.
261
00:13:36.380 --> 00:13:41.400
And then there's some reasons that the what we give a quantitative number depends a lot on what we're talking about a number for.
262
00:13:42.400 --> 00:13:49.680
So like one way you could measure how fast AI is moving is you could say, suppose you have like an AI from your X and an AI from your X plus one.
263
00:13:50.201 --> 00:13:58.060
And you're wondering, like, how much better is your AI from your X plus one in the sense of like how many your X AIs would you have needed to be comparably useful?
264
00:13:59.300 --> 00:14:03.760
like how many having AI from one year later is kind of like having twice as many computers or
265
00:14:03.780 --> 00:14:05.770
having four times as many computers or something like that.
266
00:14:05.790 --> 00:14:07.980
I think that's typically the regime
267
00:14:07.980 --> 00:14:08.221
we're in.
268
00:14:08.402 --> 00:14:11.680
So like one year of AI progress is kind of similar to increasing the amount of compute you
269
00:14:11.740 --> 00:14:17.200
have by something like 4x from a combination of like hardware progress, software progress,
270
00:14:17.320 --> 00:14:19.108
economies of scale, maybe more.
271
00:14:19.128 --> 00:14:21.660
I think you might get 8x, you might go down to 2x, but it's like
272
00:14:21.680 --> 00:14:22.542
something in this regime.
273
00:14:22.562 --> 00:14:25.649
You have like one to a couple of doublings per year.
274
00:14:28.516 --> 00:14:30.280
So like right now,
275
00:14:30.401 --> 00:14:31.406
it doesn't really matter that much.
276
00:14:31.748 --> 00:14:34.000
Doubling the amount of computers you have has very little
277
00:14:34.060 --> 00:14:34.784
effect on the world.
278
00:14:35.568 --> 00:14:37.940
Like if we doubled the number of computers in the world today, you would not
279
00:14:37.980 --> 00:14:40.830
even notice in like GDP statistics.
280
00:14:41.592 --> 00:14:42.736
Yeah, you wouldn't notice basically.
281
00:14:43.257 --> 00:14:44.020
I think in the future,
282
00:14:44.280 --> 00:14:47.060
there are going to be AI systems that are doing some large fraction of all the work in the world.
283
00:14:47.120 --> 00:14:50.040
And ultimately they can like substitute effectively for humans across many domains.
284
00:14:50.744 --> 00:14:53.500
and then doubling the number of your computers you have is kind of like doubling the effective
285
00:14:53.601 --> 00:14:56.800
population size, like doubling how many people are working as researchers, doubling how many
286
00:14:56.860 --> 00:14:57.685
people are doing jobs.
287
00:14:58.026 --> 00:15:00.560
And so in that world, if you say you're doubling the number of computers
288
00:15:00.640 --> 00:15:06.060
you have effectively every like four to six months, that does imply a very rapid rate of
289
00:15:06.160 --> 00:15:09.800
change in kind of how quickly science is progressing and sort of how much stuff you're
290
00:15:09.840 --> 00:15:10.865
able to accomplish in the world.
291
00:15:11.589 --> 00:15:13.780
And so I kind of think of that as this like first transition
292
00:15:14.162 --> 00:15:17.040
as you move into this world where you can substitute for humans is you're looking at
293
00:15:17.060 --> 00:15:21.180
a rate of growth something like doubling a couple times a year in total output of AI systems.
294
00:15:23.082 --> 00:15:25.600
The main thing that softens this, so we could talk about how fast the transition,
295
00:15:25.700 --> 00:15:28.480
what that actually translates into in terms of how fast the transition is in the world.
296
00:15:29.041 --> 00:15:34.140
I think there's one important consideration that softens that change, which is that you have some
297
00:15:34.180 --> 00:15:36.051
complementarity between AI systems and humans.
298
00:15:36.293 --> 00:15:37.500
That is, AI systems are good at some things,
299
00:15:37.560 --> 00:15:38.685
humans are good at other things.
300
00:15:39.389 --> 00:15:41.640
So as a result, the things that AI's are good at,
301
00:15:42.000 --> 00:15:43.649
you tend to hit diminishing returns on those.
302
00:15:43.991 --> 00:15:45.820
So the transition is a little bit slower than
303
00:15:45.840 --> 00:15:48.680
you would guess if AIs and humans were perfect substitutes, I think you'd be looking at a
304
00:15:48.680 --> 00:15:53.760
transition over like something like 12 months from a world where like humans are doing almost
305
00:15:53.820 --> 00:15:55.146
everything, humans are doing almost nothing.
306
00:15:55.828 --> 00:15:58.780
I think given some complementarity, you're probably
307
00:15:58.780 --> 00:16:00.507
looking at a transition over more like years.
308
00:16:01.229 --> 00:16:03.960
I think it's like kind of hard to run the numbers
309
00:16:04.061 --> 00:16:05.248
to get a transition over decades.
310
00:16:05.470 --> 00:16:07.120
I think a lot of people say that and have a strong intuition
311
00:16:07.200 --> 00:16:07.783
in that direction.
312
00:16:07.803 --> 00:16:11.660
But like when the discussion gets down to brass tacks, I currently don't
313
00:16:11.700 --> 00:16:13.968
really see how to make it work.
314
00:16:14.029 --> 00:16:17.100
I think it is possible to get up to like, yeah, very low decades,
315
00:16:17.280 --> 00:16:20.000
like, you know, having, anyway, we have talked about timeline between what and what, but I think
316
00:16:20.000 --> 00:16:21.248
we're mostly talking about like years.
317
00:16:21.268 --> 00:16:23.060
I think months would be pretty surprising, but possible.
318
00:16:23.960 --> 00:16:26.180
Decades would be also pretty surprising, but possible.
319
00:16:27.080 --> 00:16:32.860
So just, Paul, to get people up to speed on your 12 years of debate with Eliezer and those who
320
00:16:33.081 --> 00:16:34.123
hold this viewpoint.
321
00:16:34.184 --> 00:16:39.920
So does Eliezer think in terms of like minutes or days that this could
322
00:16:39.940 --> 00:16:40.301
happen?
323
00:16:40.402 --> 00:16:44.274
And you're saying, not that fast, it's closer to years.
324
00:16:44.776 --> 00:16:45.539
Is that the difference?
325
00:16:45.639 --> 00:16:45.860
And then
326
00:16:46.602 --> 00:16:51.260
once you answer that, can you tell us, why does this matter so much, whether it happens in minutes
327
00:16:51.340 --> 00:16:54.691
or days versus, you know, years and decades?
328
00:16:54.751 --> 00:16:57.620
Why is that such a fulcrum of this whole debate and
329
00:16:57.660 --> 00:17:02.240
discussion on why the AIs will come kill us or whether they will or whether we'll be okay?
330
00:17:03.700 --> 00:17:08.100
I think probably it is harder to describe Eliezer's views quantitatively in terms of
331
00:17:08.160 --> 00:17:12.060
rates of change because more of his view is about this like kind of phase transition that happens
332
00:17:12.201 --> 00:17:15.740
quite quickly like it's not reasonable to talk another perspective it's not reasonable to talk
333
00:17:15.820 --> 00:17:19.240
about this framing of like how long does it take to like double the population size or something
334
00:17:19.722 --> 00:17:23.240
it's more like how long does it take to move from like chimps who are doing nothing to humans who
335
00:17:23.260 --> 00:17:27.020
are doing a lot of stuff and he's like that i don't know it could just randomly happen one day
336
00:17:27.080 --> 00:17:31.480
that someone like tweaks their code and i went from being a chimp to being a being a human and
337
00:17:31.500 --> 00:17:35.460
that's pretty transformative um i think he sort of just has a broader distribution i think he does
338
00:17:35.500 --> 00:17:36.707
not find ears out of the question.
339
00:17:36.768 --> 00:17:38.840
He's just like, that's kind of the tale of how slow it could be or
340
00:17:38.900 --> 00:17:39.563
something like that.
341
00:17:39.583 --> 00:17:43.020
And like, it's more about this qualitative picture of he's just like,
342
00:17:43.481 --> 00:17:45.471
you're sort of not going to have changes in the world.
343
00:17:45.672 --> 00:17:47.200
Like, in some sense, the more important
344
00:17:47.280 --> 00:17:51.740
thing is I imagine AI systems acting in the world, doing like trillions of dollars of economic value
345
00:17:52.001 --> 00:17:56.560
prior to getting to this point where they're actually causing like this potential catastrophic
346
00:17:56.761 --> 00:18:01.620
risk or where they're significantly or totally transforming the pace of future technological
347
00:18:01.821 --> 00:18:02.142
change.
348
00:18:02.664 --> 00:18:06.880
I think Eliezer imagines more like you move from a world like the world of today,
349
00:18:06.980 --> 00:18:10.120
where AIs are doing maybe billions or tens of billions or hundreds of billions of dollars of
350
00:18:10.200 --> 00:18:10.542
value.
351
00:18:10.562 --> 00:18:14.860
I think that feels to me like the core distinction is kind of like where are you starting
352
00:18:14.961 --> 00:18:15.162
from?
353
00:18:15.222 --> 00:18:17.597
And I'm like more starting from a world with trillions or tens of trillions.
354
00:18:17.738 --> 00:18:18.040
And Eliezer
355
00:18:18.080 --> 00:18:23.160
is more starting from a world of like, I mean, 12 years ago, I think this is probably going to
356
00:18:23.442 --> 00:18:26.157
seem unfair to Eliezer, but this discussion was like maybe more live.
357
00:18:26.257 --> 00:18:26.720
And Eliezer would
358
00:18:26.760 --> 00:18:29.720
frequently talk about like maybe some random people in a small AI group somewhere, like a
359
00:18:29.820 --> 00:18:33.140
company like DeepMind doing like $100 million a year of spending is going to be building
360
00:18:33.220 --> 00:18:34.024
transformative AI.
361
00:18:34.667 --> 00:18:37.240
I think my basic take was like, no way, you're going to look at AI systems that
362
00:18:37.260 --> 00:18:38.747
are doing trillions of dollars of revenue.
363
00:18:38.888 --> 00:18:41.440
And like, this gap is closing pretty rapidly,
364
00:18:41.440 --> 00:18:44.360
because now no one's going to say like, you're doing $100 million of revenue, like, it's pretty
365
00:18:44.420 --> 00:18:46.874
clear going to be in the least like billions or tens of billions of dollars.
366
00:18:47.297 --> 00:18:47.840
I think my take is
367
00:18:47.860 --> 00:18:50.818
just like, pretty soon, it's going to be pretty clear doing 100 billion, that's going to be like,
368
00:18:52.300 --> 00:18:54.620
yeah, I think we're kind of just debating, like, what is the point that you jump from?
369
00:18:54.920 --> 00:18:58.320
Like, when you get to the AI, that's like crazy science stuff, what was happening like six months
370
00:18:58.401 --> 00:18:59.640
before that was what was happening.
371
00:18:59.941 --> 00:19:01.560
Like AI systems really broadly deployed in the world,
372
00:19:01.620 --> 00:19:02.659
doing a ton of crazy things,
373
00:19:03.124 --> 00:19:04.940
or it was happening like actually the impact of AI systems
374
00:19:05.001 --> 00:19:06.540
was pretty limited until right before
375
00:19:07.122 --> 00:19:11.299
you kind of have this process of rapidly accelerating R&D,
376
00:19:11.700 --> 00:19:14.019
recursive self-improvement within like a single firm
377
00:19:14.222 --> 00:19:15.460
or in a like local part of the world.
378
00:19:15.941 --> 00:19:18.740
But to be clear, Paul, do you think it's possible
379
00:19:19.183 --> 00:19:21.459
or more unlikely than Eliezer thinks it is
380
00:19:21.822 --> 00:19:24.720
to go from that big hop software update one day
381
00:19:25.565 --> 00:19:27.920
to move from chimps to human level intelligence?
382
00:19:28.022 --> 00:19:28.960
Do you think that's unlikely?
383
00:19:29.240 --> 00:19:31.360
And do you have kind of technical grounds for this?
384
00:19:31.480 --> 00:19:35.599
Or what are your grounds for believing that that's less likely than others do?
385
00:19:37.020 --> 00:19:38.380
Yeah, I think there's two parts of this.
386
00:19:38.480 --> 00:19:41.560
I mean, again, I want to emphasize that compared to most people in the world, I think I'm into
387
00:19:41.662 --> 00:19:42.860
much, much faster change.
388
00:19:43.060 --> 00:19:45.880
I think the mainstream view in ML is that things will be more gradual, which I think
389
00:19:45.922 --> 00:19:46.340
is mistaken.
390
00:19:46.500 --> 00:19:49.120
And when you get down, again, when we get down to brass tacks, I'm really unpersuaded.
391
00:19:51.102 --> 00:19:53.880
You could then talk about like my view, you're going to views that are quantitatively faster
392
00:19:53.985 --> 00:19:54.300
than mine.
393
00:19:54.360 --> 00:19:55.960
It's just like, actually, this isn't years, this is months.
394
00:19:56.240 --> 00:19:59.863
I think that's like, it's very defensible to run the historical extrapolations and end up with different numbers.
395
00:19:59.883 --> 00:20:01.804
That's just like a really hard empirical question.
396
00:20:01.944 --> 00:20:03.766
And it's like hard to make these predictions about the future.
397
00:20:03.826 --> 00:20:04.846
And I have a lot of sympathy for that.
398
00:20:05.587 --> 00:20:07.028
Then I think there's more like this qualitative claim.
399
00:20:07.048 --> 00:20:08.309
It's like the chimp versus human jump.
400
00:20:08.849 --> 00:20:11.531
I think that's not out of the question, but feels quite unlikely.
401
00:20:11.771 --> 00:20:21.098
I think the basic reason it feels quite unlikely to me is like, that is really not how I would claim that's really not how anything has worked in AI to date.
402
00:20:21.338 --> 00:20:23.740
And it's not how things have worked in almost any other technologies.
403
00:20:24.524 --> 00:20:27.900
like it has it has mostly been the case this is my read of history and i'm very happy to argue
404
00:20:27.940 --> 00:20:31.360
about i think this is like a an important part of the argument with eliezer my read of history is
405
00:20:31.440 --> 00:20:34.640
mostly that before you can do something really crazy you can do something that like works a
406
00:20:34.720 --> 00:20:38.820
little bit less well and it like is a little bit crappier and you do sometimes have these jumps
407
00:20:38.900 --> 00:20:43.420
from like zero to one but you tend to see the zero to one jump not when a technology is worth like
408
00:20:44.064 --> 00:20:46.640
this would allow you to take over the world or this would be worth 10 trillion dollars you tend
409
00:20:46.700 --> 00:20:49.780
to see zero to one jumps when a technology is like you have a bunch of amateurs or hobbyists
410
00:20:49.860 --> 00:20:53.220
or a couple scientists and so i think like if we're going to see such a jump in ai i'd be more
411
00:20:53.260 --> 00:20:56.100
likely to have seen it back when we were talking about like a small academic community and you're
412
00:20:56.200 --> 00:21:00.280
less likely to see it when you have academic or like labs investing billions or tens of billions
413
00:21:00.381 --> 00:21:04.120
of dollars i think the general record is like most of the time before you can do something
414
00:21:04.200 --> 00:21:07.920
really crazy you can do something slightly less crazy and that becomes like a more and more robust
415
00:21:08.061 --> 00:21:11.160
regularity as you increase the number of people thinking about something and increase the amount
416
00:21:11.180 --> 00:21:14.440
of attention you move more and more towards like industries with reasonable roadmaps that actually
417
00:21:14.500 --> 00:21:18.280
are forecast for what's going to happen is there an example that you call to mind for
418
00:21:18.681 --> 00:21:21.108
history that sort of we can compare this to?
419
00:21:22.532 --> 00:21:25.500
I mean, I think I'm happy to compare it to sort
420
00:21:25.520 --> 00:21:28.378
of almost any technology that is like, they are all different, different ways.
421
00:21:28.398 --> 00:21:28.700
I don't know if
422
00:21:28.700 --> 00:21:31.277
this, I think Elias was probably more like wanting to point to a particular thing.
423
00:21:31.297 --> 00:21:31.660
He's like, this
424
00:21:31.680 --> 00:21:32.583
is the really relevant one.
425
00:21:33.205 --> 00:21:36.817
But I would say like, to me, AI seems kind of similar to like future AI
426
00:21:37.660 --> 00:21:41.780
developments seem kind of similar to either past AI developments, other developments in software.
427
00:21:42.400 --> 00:21:48.020
I'd be happy to talk about computing hardware or solar power or nuclear power or nuclear weapons
428
00:21:48.425 --> 00:21:49.257
or flights.
429
00:21:50.602 --> 00:21:53.900
They all take this kind of gradual sort of approach
430
00:21:54.001 --> 00:21:55.660
rather than the big zero to one moments.
431
00:21:56.488 --> 00:21:57.360
And part of the reason, Paul,
432
00:21:57.942 --> 00:21:59.019
people are asking about this
433
00:21:59.121 --> 00:22:00.840
is because I think it seems like,
434
00:22:00.940 --> 00:22:02.180
and you tell us,
435
00:22:02.220 --> 00:22:04.500
so you previously have worked at OpenAI,
436
00:22:04.821 --> 00:22:06.760
very familiar with the methodologies used,
437
00:22:07.301 --> 00:22:09.557
but it feels to some people like chat GPT
438
00:22:10.362 --> 00:22:13.580
has been a big zero to one moment, right?
439
00:22:13.700 --> 00:22:15.400
Like, my God, it's amazing.
440
00:22:16.042 --> 00:22:21.600
and people are, you know, tinkering with it in so many ways and how human-like it seems and how fast
441
00:22:22.162 --> 00:22:24.510
it seemed to explode into the popular consciousness.
442
00:22:24.992 --> 00:22:27.080
And so I'm wondering if that has
443
00:22:27.662 --> 00:22:29.289
affected your view on this at all.
444
00:22:29.329 --> 00:22:31.900
It's like, oh, wow, this could happen faster than I previous
445
00:22:32.000 --> 00:22:36.199
thought, or if this is well within the bounds of your model, like what are we to make of ChatGPT?
446
00:22:37.901 --> 00:22:43.120
So I think that ChatGPT seems, I would take as like representative of the kind of trajectory
447
00:22:43.200 --> 00:22:43.742
I'd expect.
448
00:22:44.403 --> 00:22:50.278
So you could compare like chat GPT versus GPT 3.5 versus GPT 3, GPT 2.
449
00:22:50.679 --> 00:22:50.940
And like,
450
00:22:51.380 --> 00:22:55.300
I think like people at OpenAI were not, like, it's kind of mostly sociological facts about chat
451
00:22:55.360 --> 00:22:57.091
GPT getting to the point where it was discussed a lot.
452
00:22:57.132 --> 00:22:58.360
I think the actual technical change,
453
00:22:58.460 --> 00:23:03.479
certainly between chat GPT and GPT 3.5, but also between 3.5 and 3, like each of these is
454
00:23:04.663 --> 00:23:06.027
not giant jumps.
455
00:23:06.990 --> 00:23:08.534
I think these were like pretty small changes.
456
00:23:09.136 --> 00:23:10.400
And like chat GPT is not,
457
00:23:11.063 --> 00:23:14.240
I think it is at the point where it's like economically valuable and it like is worth a lot
458
00:23:14.742 --> 00:23:19.460
I think that like you're seeing I think people are mostly excited because they're looking ahead
459
00:23:19.500 --> 00:23:22.740
to where this will go and that's like a lot of what makes these dynamics like more continuous
460
00:23:22.820 --> 00:23:26.180
people are starting to say like okay what can we do with this I think they're doing that at a point
461
00:23:26.200 --> 00:23:29.140
where like they're not going to be able to do trillions of dollars a year of value but they
462
00:23:29.160 --> 00:23:32.500
are seeing that that is going to become possible at some point in the not distant future of the
463
00:23:32.540 --> 00:23:37.060
technology continues to improve it's like very concretely like I mean I know we were having
464
00:23:37.121 --> 00:23:38.960
these discussions, I was having these discussions quite a lot
465
00:23:39.767 --> 00:23:41.099
prior to the training of GPT-2.
466
00:23:41.663 --> 00:23:43.080
Prior to the training of GPT-2, we didn't
467
00:23:43.120 --> 00:23:44.595
really have language models that...
468
00:23:46.203 --> 00:23:46.980
I don't think we had language models
469
00:23:46.980 --> 00:23:48.600
you would even recognize as seeming smart.
470
00:23:49.562 --> 00:23:51.340
It kind of felt like a qualitatively different ballgame.
471
00:23:52.021 --> 00:23:52.917
Maybe most relevant is after
472
00:23:53.100 --> 00:23:55.100
the training of GPT-2, before GPT-3, or before
473
00:23:55.181 --> 00:23:55.991
the scale-up from 1b to 6b.
474
00:23:58.605 --> 00:23:58.898
I think we
475
00:23:59.687 --> 00:24:01.120
sat down and we made predictions about how good
476
00:24:01.160 --> 00:24:03.200
large models would be, and I think I am surprised
477
00:24:03.302 --> 00:24:04.460
by how good models are, but
478
00:24:05.047 --> 00:24:06.139
surprised in the sense that this is
479
00:24:06.220 --> 00:24:10.740
like maybe my 80th percentile of how smart a language model at the scale of GPT 3.5 would be.
480
00:24:13.082 --> 00:24:15.800
Something like that is a little bit hard to exactly compare my forecast to where reality
481
00:24:15.820 --> 00:24:19.497
is at, but we did discuss explicitly for like a language model, the size of GPT 3.5 trained in
482
00:24:20.220 --> 00:24:21.767
roughly the way GPT 3.5 was.
483
00:24:22.471 --> 00:24:24.420
We weren't exactly right because some of the scaling laws, like
484
00:24:24.420 --> 00:24:28.600
there's the switch from GPT scaling laws to Tinchilla scaling laws, but like roughly speaking,
485
00:24:28.660 --> 00:24:31.111
we were imagining systems at that scale trained in that way.
486
00:24:31.132 --> 00:24:33.000
And we were like, you know, an example
487
00:24:33.060 --> 00:24:36.960
of a bet would be like, is there any task a human can do over 30 seconds that a system trained in
488
00:24:37.000 --> 00:24:38.328
this way can't do over 30 seconds?
489
00:24:38.348 --> 00:24:40.460
Like, is there any 30 second Turing test that can distinguish a
490
00:24:40.500 --> 00:24:41.203
human from an AI?
491
00:24:41.223 --> 00:24:45.120
And I think like, you know, the debate was kind of like amongst people I was most
492
00:24:45.160 --> 00:24:48.540
talking to at OpenAI, I was like, ah, probably there will be, but it's not a sure thing, like a
493
00:24:48.640 --> 00:24:51.936
one third chance, you know, that there will be no such tests or something like that.
494
00:24:52.539 --> 00:24:52.760
And like,
495
00:24:52.800 --> 00:24:56.118
I think that things are mostly like, that was not from like, it wasn't a big jump.
496
00:24:56.138 --> 00:24:56.460
It was just like
497
00:24:56.480 --> 00:24:59.600
kind of see the writing on the wall and like see systems improving in this way and be like,
498
00:24:59.901 --> 00:25:01.392
it's going to get harder and harder to tell.
499
00:25:01.432 --> 00:25:02.560
There's more and more things they can do.
500
00:25:03.000 --> 00:25:07.600
So are you saying, Paul, that for society, this seemed to come out of nowhere, basically?
501
00:25:07.781 --> 00:25:12.480
But it's not really surprising the researchers and the engineers who've been on the inside
502
00:25:12.741 --> 00:25:15.939
at OpenAI and constructing ChatGPT-4.
503
00:25:16.240 --> 00:25:19.140
It was maybe within the bounds of expectation.
504
00:25:19.480 --> 00:25:25.040
It was maybe more optimistic, or I'll use the term bullish because, look, we talk about
505
00:25:25.060 --> 00:25:26.540
bullishness in crypto all the time.
506
00:25:26.801 --> 00:25:32.560
It's more bullish than you thought this technology would sort of take you, but still within the
507
00:25:32.600 --> 00:25:38.020
realms and the only reason it's having such a an effect on our collective consciousness is is more
508
00:25:38.542 --> 00:25:43.580
sociological it's just because consumer application yeah it's just suddenly turned into a consumer app
509
00:25:43.680 --> 00:25:48.320
and everyone's like wow i can type whatever i want into this magic box and a genie oracle
510
00:25:49.182 --> 00:25:55.320
artificial intelligence just gives me the answer this is incredible um like is anyone surprised by
511
00:25:55.380 --> 00:25:59.140
this who's been working on this tech oh i mean i think yes this is a nuanced question there's like
512
00:25:59.200 --> 00:26:02.460
a lot to say i don't know if we want to get into all of it but like i think at the point when gpt
513
00:26:02.560 --> 00:26:05.111
2 was trained, this was a controversial prediction.
514
00:26:05.694 --> 00:26:07.120
So like there were people who were more bullish
515
00:26:07.200 --> 00:26:11.680
than I was, like EG, Dario, who now runs, Dario Amadei, who now runs Anthropic, was like at the
516
00:26:11.720 --> 00:26:13.346
time of GPT-2, like very bullish.
517
00:26:13.667 --> 00:26:16.436
And in timeline here, Paul, so GPT-2 was when?
518
00:26:16.616 --> 00:26:16.978
What year?
519
00:26:17.419 --> 00:26:17.660
Oh man,
520
00:26:17.700 --> 00:26:18.523
I don't even know if I remember.
521
00:26:18.543 --> 00:26:20.329
I think these discussions were like 2018.
522
00:26:21.012 --> 00:26:21.273
Got it.
523
00:26:21.453 --> 00:26:21.995
Okay.
524
00:26:23.019 --> 00:26:23.240
Go on.
525
00:26:23.420 --> 00:26:23.621
Yes.
526
00:26:25.326 --> 00:26:28.134
But like, yeah, so Dario was more optimistic.
527
00:26:28.154 --> 00:26:29.940
I think this world is actually like slightly below
528
00:26:30.100 --> 00:26:34.596
Dario's median of how impressive a system like chat GPT would be.
529
00:26:34.616 --> 00:26:35.780
I mean, not a huge amount
530
00:26:35.880 --> 00:26:36.061
below.
531
00:26:36.081 --> 00:26:38.875
I think he made quite a good forecast and it's kind of been his big bid win.
532
00:26:39.639 --> 00:26:39.940
I think there
533
00:26:39.960 --> 00:26:43.459
were a bunch of people who kind of had views in this general cluster, for whom this is a little
534
00:26:43.680 --> 00:26:47.620
bit better than what they would have thought, or at the 80th percentile or something of what they
535
00:26:47.640 --> 00:26:48.244
might have expected.
536
00:26:48.626 --> 00:26:50.800
I think there were a lot of people who weren't in the business of making
537
00:26:50.900 --> 00:26:52.927
forecasts, but seemed qualitatively.
538
00:26:52.947 --> 00:26:56.579
If you look to the academic ML community, it felt to me like
539
00:26:57.002 --> 00:26:58.399
people were not expecting this to happen.
540
00:26:58.680 --> 00:27:00.300
Like discourse about general AI
541
00:27:00.765 --> 00:27:02.260
felt like very frustrating, I think,
542
00:27:02.520 --> 00:27:03.720
where they're like, why is it the case
543
00:27:03.861 --> 00:27:06.740
that people are really minimizing the possibility
544
00:27:06.821 --> 00:27:09.080
of just large neural nets trained in a very simple way,
545
00:27:10.001 --> 00:27:10.999
being competitive with humans?
546
00:27:13.422 --> 00:27:15.760
I think it was surprising to like a lot of people.
547
00:27:15.880 --> 00:27:17.820
And again, it was just quantitatively, right?
548
00:27:17.840 --> 00:27:18.900
It was surprising to me in the sense
549
00:27:18.900 --> 00:27:20.040
that this is like better than I thought.
550
00:27:20.120 --> 00:27:21.360
And I like lost bets about this.
551
00:27:21.460 --> 00:27:22.380
I mean, I won some bets with people.
552
00:27:22.440 --> 00:27:23.340
I lost some bets with people.
553
00:27:24.764 --> 00:27:26.280
I think some people weren't quantifying
554
00:27:26.340 --> 00:27:28.551
their probabilities, maybe this was like more just totally out of model.
555
00:27:28.571 --> 00:27:30.500
But by the time you're
556
00:27:30.520 --> 00:27:33.820
talking about chat GPT compared to what came before, I do not think, I think at that point
557
00:27:33.961 --> 00:27:36.732
to people building the system, it was not surprising.
558
00:27:37.737 --> 00:27:38.500
Like once you're talking about
559
00:27:38.500 --> 00:27:42.120
the gap, even from three to 3.5, and then from 3.5 to chat GPT, I think they're more,
560
00:27:42.600 --> 00:27:45.980
they were probably more surprised by the way, like the impact it had on collective consciousness,
561
00:27:46.603 --> 00:27:48.712
rather than by the capabilities of the system itself.
562
00:27:48.933 --> 00:27:50.380
I think that seemed like pretty technically
563
00:27:50.501 --> 00:27:50.963
de-risked.
564
00:27:51.605 --> 00:27:54.880
And Paul, when we're talking about sort of AI alignment and safety concerns,
565
00:27:54.880 --> 00:27:57.560
and just on the topic still of chat GPT, right?
566
00:27:58.461 --> 00:28:00.440
Are these neural nets,
567
00:28:00.580 --> 00:28:01.740
these large language models,
568
00:28:01.840 --> 00:28:02.960
the ones we have to worry about?
569
00:28:03.100 --> 00:28:05.319
Like, I think what basically society
570
00:28:05.442 --> 00:28:06.080
is sort of wondering
571
00:28:06.261 --> 00:28:08.760
as this AI alignment safety question
572
00:28:08.841 --> 00:28:11.120
rises into public consciousness is,
573
00:28:11.601 --> 00:28:13.460
okay, at what version
574
00:28:13.541 --> 00:28:15.020
do we have to start worrying
575
00:28:15.020 --> 00:28:17.639
that chat GPT is going to pose a threat to us?
576
00:28:18.021 --> 00:28:18.980
Like there's some version
577
00:28:19.061 --> 00:28:21.140
where it starts to take our jobs, okay?
578
00:28:21.541 --> 00:28:23.099
And then like maybe that's version four,
579
00:28:23.488 --> 00:28:24.000
version five.
580
00:28:24.081 --> 00:28:26.500
And so it affects our economy and economics.
581
00:28:26.520 --> 00:28:29.840
And we have to reorient, restructure society as a result of this.
582
00:28:30.381 --> 00:28:34.680
But like it seemed to be what Eliezer was saying is, well, maybe it might be version
583
00:28:34.961 --> 00:28:38.640
nine or version ten or version 11 where we actually have to fear
584
00:28:38.721 --> 00:28:41.560
for our lives from this thing because it's become super intelligent.
585
00:28:41.701 --> 00:28:45.760
Do you see that possible trajectory for this specific large language
586
00:28:46.021 --> 00:28:51.060
AI technology, or should we have more concerns about other AI technology
587
00:28:51.201 --> 00:28:55.660
that's coming in some other vector of development.
588
00:28:56.723 --> 00:28:59.160
Actually, I'd like to phrase that question slightly differently
589
00:28:59.523 --> 00:29:01.320
and maybe in a way that bankless listeners,
590
00:29:02.546 --> 00:29:04.000
a metaphor that bankless listeners can understand.
591
00:29:04.521 --> 00:29:08.040
Often, Paul, when we talk to people that are outside of the crypto industry,
592
00:29:08.720 --> 00:29:11.980
they just call the things about the crypto industry Bitcoin.
593
00:29:12.620 --> 00:29:14.413
And then when Ryan and I hear that, we were like,
594
00:29:15.540 --> 00:29:19.100
oh, what they really mean is decentralized technology, identity.
595
00:29:19.321 --> 00:29:23.240
they just use Bitcoin as a placeholder to talk about so many different things.
596
00:29:23.460 --> 00:29:23.800
It's so frustrating.
597
00:29:24.020 --> 00:29:28.000
And I think as like AI normies, me and Ryan, AI normies out there,
598
00:29:28.201 --> 00:29:30.120
we might be saying chat, chat, GBT.
599
00:29:30.362 --> 00:29:31.400
And what we actually are talking,
600
00:29:31.560 --> 00:29:35.840
trying to talk about is like generalized artificial intelligence.
601
00:29:36.122 --> 00:29:38.320
And we just use chat GBT because that's the thing.
602
00:29:38.380 --> 00:29:39.639
It's the Bitcoin of AI.
603
00:29:40.020 --> 00:29:41.420
Is that, are we falling into that same trap?
604
00:29:43.223 --> 00:29:45.180
I definitely think there's a lot of subtlety here.
605
00:29:45.240 --> 00:29:47.240
And there's a lot of different ways I could interpret like this thing.
606
00:29:47.823 --> 00:29:50.700
So I'd say that like one question is like, what are you modeling?
607
00:29:50.840 --> 00:29:51.999
Like what is the eye learning to predict?
608
00:29:52.240 --> 00:29:56.700
Like is it videos or is it text or is it like interactions with code,
609
00:29:56.840 --> 00:30:00.020
like running code and the results of running code or like the code humans would write?
610
00:30:00.722 --> 00:30:03.280
And I think like over the last couple of years,
611
00:30:03.320 --> 00:30:05.419
I think those distinctions have mattered a lot less.
612
00:30:05.960 --> 00:30:09.260
Like what is mostly, I think the default model for how the system should work
613
00:30:09.300 --> 00:30:11.880
is just you have quite a lot of data of quite a lot of types and you just dump it all in.
614
00:30:11.920 --> 00:30:15.620
You say like, look, your job AI is to deal as well as you can with every type of data we give you.
615
00:30:16.121 --> 00:30:22.020
And there's engineering problems in allowing those different types of data and those questions about what types of data the system is actually able to effectively deal with.
616
00:30:22.380 --> 00:30:27.340
But I think the fact that it's trained on language is like not, I just don't think you should think of it as a defining feature.
617
00:30:28.201 --> 00:30:29.980
I think you should imagine systems like see the world.
618
00:30:30.462 --> 00:30:32.599
I think language is probably an important way of thinking about how they act.
619
00:30:33.563 --> 00:30:34.600
Although, again, I think it's very impressive.
620
00:30:34.740 --> 00:30:37.900
Systems can act by producing images and those will be very impressive and will have a big impact.
621
00:30:37.940 --> 00:30:42.380
I think economically, in some sense, language is like a very flexible and like kind of core way you should think about systems acting.
622
00:30:42.541 --> 00:30:45.380
But perceiving, I don't think you should really think about language models in particular.
623
00:30:47.286 --> 00:30:51.460
GPT itself, just saying GPT, this is basically just fixing, like, there's two things that
624
00:30:51.500 --> 00:30:51.903
specifies.
625
00:30:51.963 --> 00:30:53.029
One is, how is it trained?
626
00:30:53.170 --> 00:30:54.176
Is it trained to predict data?
627
00:30:54.317 --> 00:30:54.880
Or is it pre-trained
628
00:30:54.940 --> 00:30:55.543
to predict data?
629
00:30:56.165 --> 00:30:57.430
Or is it trained in some other way?
630
00:30:57.450 --> 00:30:59.499
That's the first distinction.
631
00:30:59.519 --> 00:30:59.800
And the second
632
00:30:59.840 --> 00:31:00.846
is just that it's a transformer.
633
00:31:01.449 --> 00:31:03.380
I don't know if anyone wants to make, like, a super strong bet
634
00:31:03.521 --> 00:31:04.969
about, like, transformers per se.
635
00:31:05.029 --> 00:31:07.080
I think there's just, like, are different, yeah, there's a big
636
00:31:07.120 --> 00:31:08.329
space of possible architectures.
637
00:31:08.571 --> 00:31:09.900
There's probably going to be debate about some things whether you
638
00:31:09.900 --> 00:31:10.700
We call them a transformer.
639
00:31:13.163 --> 00:31:15.100
I think it's like, I think both of these,
640
00:31:15.180 --> 00:31:16.180
like what kind of data you're modeling
641
00:31:16.564 --> 00:31:18.220
and then like what kind of architecture you're using
642
00:31:18.382 --> 00:31:19.500
are in some sense not very essential.
643
00:31:19.680 --> 00:31:20.440
Like I don't think it would change
644
00:31:20.481 --> 00:31:21.600
like OpenAI being in the game
645
00:31:21.741 --> 00:31:23.360
or like the exact kind of product they're offering.
646
00:31:23.680 --> 00:31:24.300
I think they're just like,
647
00:31:24.340 --> 00:31:25.399
look, we train large neural nets.
648
00:31:25.841 --> 00:31:28.600
We do it on having some very like broad pre-training task
649
00:31:28.701 --> 00:31:29.820
that captures a lot about the world
650
00:31:30.002 --> 00:31:31.260
and gives an interesting opportunity to be smart.
651
00:31:31.540 --> 00:31:33.019
And then we fine tune them on downstream tasks
652
00:31:33.121 --> 00:31:34.420
that we think are economically useful.
653
00:31:34.940 --> 00:31:35.900
Usually like chatting with people
654
00:31:35.961 --> 00:31:37.500
or generating images that people would rate highly
655
00:31:37.682 --> 00:31:39.600
or like writing code that developers will think it's good.
656
00:31:40.462 --> 00:31:45.260
I think that's like the basic paradigm you should imagine when we talk about like this thing, which is like a little bit broader than chat GPT.
657
00:31:45.861 --> 00:31:49.400
But I think it's not crazy to say like chat GPT really is indicative of that broader ecosystem.
658
00:31:49.660 --> 00:31:55.120
And I'd say like chat GPT is more similar to the rest of that ecosystem than like Bitcoin is to the rest of the crypto ecosystem.
659
00:31:55.700 --> 00:31:58.240
Like there are fewer key technical differences in that case.
660
00:32:01.864 --> 00:32:02.840
Yeah, that's like a high level.
661
00:32:02.980 --> 00:32:07.560
And then whether this like kind of thing can cause trouble, like I'm like, I think it's really, really hard to say.
662
00:32:07.680 --> 00:32:13.560
I think like a lot of people talk a lot of smack about how like it's really silly to think that AI systems of this form could do something crazy.
663
00:32:13.780 --> 00:32:19.520
And I'm looking at that and I'm like, the same people were talking smack that feels to me, again, often not in the business of making concrete predictions.
664
00:32:20.161 --> 00:32:26.020
But we're saying that it was really silly to have the expectations that people had about language model scale ups five years ago.
665
00:32:26.500 --> 00:32:34.820
And I'm like, I think the scale up, which you could imagine occurring over the coming years, is similar in magnitude to the scale up we've observed over the last five years.
666
00:32:35.886 --> 00:32:38.080
and so I'm like, it's really hard to predict where that ends up.
667
00:32:38.480 --> 00:32:40.740
And if someone is giving a confident take about where that ends up,
668
00:32:40.780 --> 00:32:43.180
if they're like, yeah, these AI systems can't do X or can't do Y,
669
00:32:43.721 --> 00:32:46.240
I really want them to get more precise about why they think that
670
00:32:46.341 --> 00:32:47.380
and what exactly they're saying.
671
00:32:47.541 --> 00:32:49.660
And I'm really just pretty skeptical on the face of it.
672
00:32:50.262 --> 00:32:54.160
I think we can really relate to that on at least using our crypto frame of mind
673
00:32:54.261 --> 00:32:56.500
is again, when we talk to normies,
674
00:32:56.680 --> 00:32:58.720
what we call the people who are not inside of the crypto world
675
00:32:58.841 --> 00:33:01.840
and they're like, oh, these Bitcoin, Ethereum currencies,
676
00:33:02.122 --> 00:33:04.060
they can't possibly take over the world.
677
00:33:04.381 --> 00:33:12.700
And I think when you become more informed about the crypto world, like you just get so tired of these takes because of how uninformed and unimaginative they seem.
678
00:33:13.140 --> 00:33:20.200
And so like just conceptually, I can I can definitely resonate with with that is like you you don't know what you don't know and neither do we.
679
00:33:20.360 --> 00:33:28.860
But like you can kind of understand some base principles as to the nature of these things and how they grow and develop and change in ways that you might not expect.
680
00:33:29.343 --> 00:33:32.100
and you can kind of, if you are versed in these topics,
681
00:33:32.220 --> 00:33:33.840
can extrapolate into the future pretty well.
682
00:33:34.340 --> 00:33:37.320
And without precision, still give a broad stroke about like,
683
00:33:37.400 --> 00:33:38.819
hey, this is where this is going to go
684
00:33:38.901 --> 00:33:39.879
and here's what you don't appreciate.
685
00:33:40.341 --> 00:33:41.520
So I can definitely appreciate that.
686
00:33:41.861 --> 00:33:44.700
And I want to go back and tie a bow on the time conversation
687
00:33:45.245 --> 00:33:46.800
because we started this conversation like,
688
00:33:46.880 --> 00:33:47.700
okay, it could happen.
689
00:33:48.182 --> 00:33:50.540
There's the model of it happening in two days.
690
00:33:50.981 --> 00:33:53.500
There's a model of happening in two years or 20 years.
691
00:33:54.020 --> 00:33:56.180
And you're in the camp of,
692
00:33:56.260 --> 00:34:02.220
I'm going to give a range of like six to six months to two years ish loosely, very loosely
693
00:34:02.845 --> 00:34:04.760
without trying to be too precise about this.
694
00:34:05.020 --> 00:34:05.320
It depends a lot.
695
00:34:05.481 --> 00:34:06.640
Depends a lot on from when to when.
696
00:34:06.640 --> 00:34:06.740
Depends a lot.
697
00:34:06.860 --> 00:34:07.540
Lots of variables.
698
00:34:08.300 --> 00:34:08.360
Yeah.
699
00:34:08.424 --> 00:34:08.637
Years.
700
00:34:10.087 --> 00:34:10.920
Time can pass.
701
00:34:11.000 --> 00:34:13.259
Like you're going to go, you're going to wake up.
702
00:34:13.760 --> 00:34:15.540
You're not, you're not going to wake up and it's going to be different.
703
00:34:15.862 --> 00:34:18.220
And like the reason why this is important and I want to go back.
704
00:34:18.521 --> 00:34:19.538
Oh, even you're shaking your head.
705
00:34:19.660 --> 00:34:22.300
I mean, I want to, I want to be careful about like, there's a question of what my default
706
00:34:22.401 --> 00:34:25.560
expectation is and then what is possible and what you can be confident about.
707
00:34:26.182 --> 00:34:31.760
so like i am extremely skeptical of someone who's confident that if you took gpt4 and scaled up by
708
00:34:31.820 --> 00:34:35.200
two orders of magnitude of training compute and then fine-tune the resulting system using
709
00:34:35.260 --> 00:34:39.560
existing techniques that we know exactly what would happen like i think that thing you're looking at
710
00:34:40.322 --> 00:34:45.720
a non-trivial chance that it would yeah reasonable chance that it would be inclined or would be
711
00:34:45.760 --> 00:34:49.940
sufficient if it was inclined it would be capable enough to effectively disempower humans and like
712
00:34:50.121 --> 00:34:52.999
a plausible chance that it would be capable enough that you start running into these
713
00:34:53.666 --> 00:34:54.940
these concerns about controllability.
714
00:34:55.805 --> 00:34:56.980
So I would be hesitant to put
715
00:34:57.122 --> 00:34:58.260
doom probability from that.
716
00:34:58.766 --> 00:34:59.700
If a lab was not cautious
717
00:34:59.884 --> 00:35:00.660
about how they deployed it
718
00:35:00.681 --> 00:35:01.300
and wasn't measuring,
719
00:35:01.500 --> 00:35:02.479
I would be cautious to put
720
00:35:02.966 --> 00:35:04.040
the probability of takeover
721
00:35:04.202 --> 00:35:05.698
from 292 scale up to GPT-4
722
00:35:06.041 --> 00:35:07.595
below like 1% or 1 in 1,000.
723
00:35:09.102 --> 00:35:11.320
Okay, well, we'll have to put a pin in that.
724
00:35:11.602 --> 00:35:13.195
But let me like round out this time question.
725
00:35:14.102 --> 00:35:14.939
Just want to clear our tails.
726
00:35:15.261 --> 00:35:16.019
Like I have a deep podcast.
727
00:35:16.081 --> 00:35:17.440
The importance, right.
728
00:35:17.821 --> 00:35:19.780
The importance of the time point
729
00:35:20.002 --> 00:35:21.359
is like whether this is
730
00:35:21.781 --> 00:35:26.175
a lightning flash and it's different versus we have one to two years.
731
00:35:26.456 --> 00:35:27.740
For me, when I hear this,
732
00:35:27.900 --> 00:35:32.800
I'm like, okay, we have one to two years and we're watching it happen and we're seeing it happen and
733
00:35:32.800 --> 00:35:36.976
we're able to react to it versus it happens, we can't react to it.
734
00:35:37.016 --> 00:35:37.920
And so if you're telling me
735
00:35:38.020 --> 00:35:43.200
that this is a, it's still a fast takeoff, but your perception of fast is a year or so,
736
00:35:43.781 --> 00:35:47.952
to me, I'm like, okay, a year is fast enough for humans to react.
737
00:35:48.614 --> 00:35:50.499
And to me, that is a window,
738
00:35:51.100 --> 00:35:58.500
a gap a a needle that that humans have the option to thread if we can coordinate and that is where
739
00:35:58.580 --> 00:36:03.860
i start to get optimistic and so that is my gut reaction and i want to just check that gut reaction
740
00:36:04.081 --> 00:36:09.080
against you is that is that one of the paths that you see is like it doesn't go so fast that we
741
00:36:09.160 --> 00:36:15.920
don't have the time to react to it yeah i think that seems basically right with some nuance but
742
00:36:16.000 --> 00:36:20.600
like i think that most likely the thing that moves kind of slow in my view again over the course of
743
00:36:20.640 --> 00:36:24.480
years, like incredibly fast relative to policy and incredibly fast relative to like expectations
744
00:36:24.781 --> 00:36:29.960
in the broader world, but like kind of slow, um, is like how quickly do systems become more capable?
745
00:36:30.181 --> 00:36:33.660
Like how long is it between an AI, which is like smart enough that it could run a company for you.
746
00:36:33.660 --> 00:36:36.560
And actually like those companies are competitive with human companies and they, I was kind of like
747
00:36:36.600 --> 00:36:37.586
actually take over the world.
748
00:36:38.169 --> 00:36:39.778
And I'm like that gap, there's probably a gap there.
749
00:36:39.798 --> 00:36:40.140
You're probably
750
00:36:40.220 --> 00:36:42.352
talking more like, you know, years than months.
751
00:36:42.372 --> 00:36:43.960
I mean, depending exactly what you mean by run a
752
00:36:43.980 --> 00:36:44.301
company.
753
00:36:45.225 --> 00:36:48.820
Um, that said, I think it's worth pointing out the, like the dynamics of takeover itself.
754
00:36:49.142 --> 00:36:51.540
like if you had, so if you imagine broadly deployed AI systems,
755
00:36:51.601 --> 00:36:53.340
which are very competent and which would be able to take over.
756
00:36:53.441 --> 00:36:54.000
And then you ask like,
757
00:36:54.080 --> 00:36:56.480
how quickly does this particular kind of catastrophe unfold?
758
00:36:57.082 --> 00:36:59.900
I think most likely the actual catastrophe is extremely fast.
759
00:37:00.881 --> 00:37:02.169
So that's not like a year's thing.
760
00:37:02.249 --> 00:37:04.100
That's like, I think my default,
761
00:37:04.300 --> 00:37:05.980
anyway, we could get more into this and probably worth getting into.
762
00:37:06.081 --> 00:37:07.100
My default picture is like,
763
00:37:07.420 --> 00:37:10.159
we have time to react in terms of the nature of AI systems changing and AI
764
00:37:10.240 --> 00:37:11.106
capabilities changing.
765
00:37:11.871 --> 00:37:12.112
I don't.
766
00:37:12.656 --> 00:37:13.280
And like with luck,
767
00:37:13.381 --> 00:37:16.060
we have like various kinds of smaller catastrophes that occur in advance.
768
00:37:16.201 --> 00:37:18.560
But I think like one of the bad things about the situation is that,
769
00:37:18.900 --> 00:37:27.188
And the actual catastrophe you're worried about is, does have these dynamics that are kind of similar to dynamics of like a human coup or human revolution, where like, you don't have like little baby coups.
770
00:37:27.309 --> 00:37:28.970
And you'd like, see, like, here's the rate at which coups occur.
771
00:37:28.990 --> 00:37:32.994
I mean, you might be may also just go straight to like, a coup can happen very quickly, right?
772
00:37:33.014 --> 00:37:38.620
The whole dynamic is that, like, once people start switching over, once you have AI systems, you're like, actually, I'm going to get in on this, like overthrowing humanity thing.
773
00:37:39.604 --> 00:37:43.220
that information can propagate quite quickly and you don't really like you've kind of the ship has
774
00:37:43.280 --> 00:37:48.960
sailed if you've waited until like the ais are actually uh taking over something like for that
775
00:37:49.000 --> 00:37:52.160
you don't have really an opportunity to respond but for ai is changing i think it's like i basically
776
00:37:52.220 --> 00:37:55.020
think it's reasonable in some sense for people to look at the ais right now and be like look these
777
00:37:55.060 --> 00:37:59.280
things that's not realistically a takeover risk and i think that probably you're going to have
778
00:37:59.300 --> 00:38:02.680
years between people like that actually looks like a takeover risk and when actual takeover occurs and
779
00:38:02.680 --> 00:38:05.680
that's pretty good and that's a lot of why i'm more optimistic than elias i think elias was like
780
00:38:05.700 --> 00:38:07.020
You're just going to get hit by this out of the blue.
781
00:38:07.982 --> 00:38:10.700
And I'm like, well, I think people are wrong to be so confident about the rate of progress.
782
00:38:10.720 --> 00:38:16.400
But I think they probably will be able to see things that can be generally recognized as like pretty concerning prior to actual catastrophe.
783
00:38:17.524 --> 00:38:18.560
And a lot of that has happened so far.
784
00:38:18.620 --> 00:38:19.660
I think people are just like much.
785
00:38:19.880 --> 00:38:26.020
It feels more plausible now than it did five years ago by a lot that AI systems could do something really crazy and transformative.
786
00:38:26.220 --> 00:38:28.360
And I think it will feel much more plausible again in five years.
787
00:38:29.402 --> 00:38:31.480
OK, so this presents a new a new mental model for me.
788
00:38:31.540 --> 00:38:38.460
When we were talking to Eliezer, it felt very much like the don't look up problem, as in there's a meteor crashing into Earth.
789
00:38:38.661 --> 00:38:39.760
No one wants to acknowledge it.
790
00:38:39.860 --> 00:38:41.740
And then one day it crashes into Earth and we die.
791
00:38:43.261 --> 00:38:48.500
And the idea here is like we need to coordinate and get people to look up so we can identify the problem.
792
00:38:48.920 --> 00:38:54.417
And then once we identify the problem, it is a linear amount of time before the asteroid crashes into Earth.
793
00:38:55.600 --> 00:39:01.580
And what you're saying is like, we can see the asteroid, but there's like this gradually
794
00:39:01.723 --> 00:39:02.580
than suddenly moment.
795
00:39:02.700 --> 00:39:06.380
And that's your revolution moment where like, you can start to see the seeds of revolution,
796
00:39:06.983 --> 00:39:09.960
but you don't really know when the people will decide to grab their pitchforks in the
797
00:39:09.960 --> 00:39:10.860
middle of the night and revolt.
798
00:39:11.321 --> 00:39:14.580
But you can start to see the boiling of the water.
799
00:39:15.020 --> 00:39:16.860
And so that's a gradually than suddenly moment.
800
00:39:17.101 --> 00:39:22.000
But we still have the opportunity to quell the revolution before the revolution starts.
801
00:39:22.723 --> 00:39:27.080
We have enough time to send up Bruce Willis to blow up the asteroid.
802
00:39:28.524 --> 00:39:29.375
We speak in metaphors.
803
00:39:29.780 --> 00:39:30.659
Confusing metaphors.
804
00:39:33.645 --> 00:39:37.880
My best guess for if there is something like an AI takeover, this is a huge part of departure for Melios here.
805
00:39:38.400 --> 00:39:47.420
My best guess is that an AI catastrophe occurs in a world where AI systems are deployed extremely broadly and where it is kind of obvious to humans that we are putting our fate in the hands of AI systems.
806
00:39:49.162 --> 00:39:49.726
So we see ourselves.
807
00:39:52.140 --> 00:39:57.080
giving over the keys to the kingdom and we watch or are watching that happen again i think it's
808
00:39:57.120 --> 00:40:00.380
important what's possible and what's likely but i think that's the most the most likely way we die
809
00:40:00.802 --> 00:40:04.240
involves like not ai comes out of the blue and kills everyone but involves we have deployed a
810
00:40:04.280 --> 00:40:08.440
lot of ai everywhere and you can kind of just look and be like oh yeah if for if for some reason god
811
00:40:08.520 --> 00:40:13.840
forbid all these ai systems were trying to kill us they would definitely kill us oh i kind of get
812
00:40:14.000 --> 00:40:19.920
i can see this so like our tesla has got an ai in it and we trust that and our refrigerator's got an
813
00:40:19.940 --> 00:40:25.139
AI in it and it calls the grocery delivery robot, which is also an AI to deliver us food.
814
00:40:25.219 --> 00:40:25.400
And then
815
00:40:25.480 --> 00:40:27.429
all of a sudden everything around us is an AI.
816
00:40:27.470 --> 00:40:29.660
And you're like, man, I really hope that they
817
00:40:29.961 --> 00:40:30.463
like me.
818
00:40:30.985 --> 00:40:31.105
Yeah.
819
00:40:31.165 --> 00:40:35.039
You're like, you get food delivered to you from like Amazon, which is by Amazon,
820
00:40:35.440 --> 00:40:38.760
we mean a bunch of, a bunch of machines that are orchestrating a bunch of other machines.
821
00:40:39.061 --> 00:40:43.159
And you have some money and that money is managed by AI advisors investing in AI firms.
822
00:40:44.221 --> 00:40:44.943
Okay.
823
00:40:44.984 --> 00:40:46.830
That is actually pretty clear to me.
824
00:40:47.352 --> 00:40:49.480
I can, I can see how we get there with that.
825
00:40:49.980 --> 00:40:51.660
If you want some human to protect you.
826
00:40:52.063 --> 00:40:52.719
I think it's rough.
827
00:40:53.081 --> 00:40:59.900
I think basically the thing is I think it's likely that before the end it is clear that it is very hard to be physically secure.
828
00:41:00.040 --> 00:41:03.820
Like right now, if you're just like a human with some guns, you're like, look, the AI can't fuck with me that much.
829
00:41:04.101 --> 00:41:04.760
Like I have a gun.
830
00:41:04.980 --> 00:41:06.140
The AI is just on a computer somewhere.
831
00:41:06.341 --> 00:41:07.220
I can blow up the data center.
832
00:41:07.800 --> 00:41:11.020
I think it is probably clear before AI takeover that that's not the case.
833
00:41:11.160 --> 00:41:19.020
I think it's probably clear that the only way you can defend yourself effectively, like if you're fighting a war, you're like, look, you can't be like a country who's fighting a war against a country that has probably deployed AI.
834
00:41:19.100 --> 00:41:20.620
and say, it's fine, we're just not going to use AIs.
835
00:41:20.801 --> 00:41:22.019
That will just be completely untenable.
836
00:41:22.761 --> 00:41:24.560
I think it's not clear we're that far away from such a world.
837
00:41:25.040 --> 00:41:26.999
And that world's like, okay, well, if someone invades with AIs,
838
00:41:27.180 --> 00:41:28.639
obviously we're going to have our AIs defend us.
839
00:41:28.760 --> 00:41:30.200
And then you're like, okay, now it really matters.
840
00:41:30.681 --> 00:41:32.820
The prospect of the AI coup now is a different character.
841
00:41:33.000 --> 00:41:35.920
It's like, you just ask the AIs to please defend you from the other AIs,
842
00:41:36.080 --> 00:41:38.920
and maybe they're like, nah, I don't really feel like doing that.
843
00:41:40.581 --> 00:41:42.300
Again, most likely, I care about the other risks.
844
00:41:42.440 --> 00:41:43.900
And right now, I think if you were to die tomorrow,
845
00:41:44.102 --> 00:41:44.819
it would not be like this.
846
00:41:44.940 --> 00:41:46.580
It would be like, I think that really took you by surprise.
847
00:41:46.580 --> 00:41:47.780
I think you're talking about the tail there,
848
00:41:47.821 --> 00:41:48.960
and I care about evaluating the tail.
849
00:41:49.380 --> 00:41:50.820
But like the median outcome where we die,
850
00:41:50.860 --> 00:41:51.638
I think looks like this.
851
00:41:52.461 --> 00:41:54.720
I think, Paul, one problem people might be having
852
00:41:54.741 --> 00:41:55.479
who are listening to this
853
00:41:55.782 --> 00:41:58.020
or starting to be exposed to this topic for the first time,
854
00:41:58.080 --> 00:41:59.280
which is myself and David
855
00:41:59.361 --> 00:42:01.060
and maybe the average bankless listener,
856
00:42:01.880 --> 00:42:04.113
the average person who's being converted from a normie
857
00:42:05.381 --> 00:42:07.880
to someone who is actually adequately alarmed
858
00:42:08.384 --> 00:42:10.180
about AI safety types of issues,
859
00:42:10.622 --> 00:42:11.700
is this idea of agency.
860
00:42:12.140 --> 00:42:13.219
Like you've mentioned this a few times,
861
00:42:13.400 --> 00:42:16.320
like this idea of the AI's banding together
862
00:42:17.030 --> 00:42:17.760
to strike humanity.
863
00:42:18.261 --> 00:42:19.024
banding together?
864
00:42:19.044 --> 00:42:24.740
What is this like, like Google and chat GPT and all of these, like, how do they
865
00:42:24.800 --> 00:42:26.948
have agency to actually want to do that?
866
00:42:27.630 --> 00:42:29.417
It's very difficult for us to imagine.
867
00:42:29.437 --> 00:42:30.220
I mean, this seems so
868
00:42:30.360 --> 00:42:31.304
sci-fi to us.
869
00:42:32.007 --> 00:42:33.272
Could that actually happen?
870
00:42:33.352 --> 00:42:35.420
Can you give us some sort of mental model for like,
871
00:42:35.940 --> 00:42:36.703
how that happens?
872
00:42:36.783 --> 00:42:42.260
Because I'm still having a hard time understanding how a chat GPT suddenly gets
873
00:42:42.400 --> 00:42:48.900
agency and wants to ban up with, you know, 10 other super AIs and, you know, send us
874
00:42:49.361 --> 00:42:55.920
a bioengineered bacteria to kill us all, as was Eliezer's, you know, possibility, like
875
00:42:56.001 --> 00:42:57.700
expression that this could happen that way.
876
00:42:59.161 --> 00:43:02.880
Yeah, I think this is basically, I think even in a good world, we're probably going to be
877
00:43:02.880 --> 00:43:04.960
in a situation where we're trusting AIs with our lives.
878
00:43:06.122 --> 00:43:08.960
So probably in some sense, the core question is not why are they in a position to kill
879
00:43:08.960 --> 00:43:09.056
you?
880
00:43:09.120 --> 00:43:11.480
The core question is why would they end up killing you?
881
00:43:12.401 --> 00:43:14.220
I think there's basically two threat models.
882
00:43:14.880 --> 00:43:17.860
And this may be a general reason to be concerned about the world where humans are trusting
883
00:43:17.860 --> 00:43:21.260
AIs, where we generally have very limited ability to control or predict what they do.
884
00:43:22.261 --> 00:43:25.060
But if we want to talk concretely about the way we currently produce AI systems, I think
885
00:43:25.080 --> 00:43:28.860
there's basically two ways you end up in this failure mode or two, like, I mean, there's
886
00:43:28.963 --> 00:43:29.660
lots of unknown unknowns.
887
00:43:29.800 --> 00:43:32.700
There's two known ways we end up in this failure mode that people care most about.
888
00:43:35.001 --> 00:43:38.300
So first, the one that's like, I think, more likely to occur, but more easy to manage.
889
00:43:41.429 --> 00:43:45.500
so the way we train like chat gpt is you have some conversations with humans and then you look
890
00:43:45.520 --> 00:43:49.340
at that conversation you say a human rates the conversation was like this was this model doing
891
00:43:49.340 --> 00:43:53.120
a good job of answering their questions and being helpful and then you do reinforcement learning
892
00:43:53.241 --> 00:43:56.100
where you like take the the nature of that interaction if it went well you update them
893
00:43:56.120 --> 00:43:59.540
all to a little bit more like that and if it went poorly you update them all to do a little bit less
894
00:43:59.600 --> 00:44:07.260
like that so that's how we train like chat gpt a way you might try and use gpt is you might say
895
00:44:07.260 --> 00:44:10.159
I'm actually going to give it some tools and I'm going to give it a task and I'm asking to try and
896
00:44:10.541 --> 00:44:11.345
accomplish that task.
897
00:44:11.365 --> 00:44:12.752
So I might say like, my code has failed.
898
00:44:13.395 --> 00:44:14.320
I don't quite understand why.
899
00:44:14.661 --> 00:44:17.480
I'm just going to ask GPT, like, hey, you have a bunch of ability to run code on my computer.
900
00:44:17.800 --> 00:44:20.295
You can like make changes to the code and see what happens.
901
00:44:20.315 --> 00:44:21.200
You can spin up a web server.
902
00:44:21.520 --> 00:44:22.769
Could you like figure out where the error was?
903
00:44:22.990 --> 00:44:24.380
Like, could you by stack, tell me what commit
904
00:44:24.420 --> 00:44:25.911
introduced the problem, tell me what's up with the problem.
905
00:44:26.375 --> 00:44:27.080
And then you want to go send the
906
00:44:27.100 --> 00:44:30.420
system to act autonomously and like perform all these actions, like to try running different
907
00:44:30.520 --> 00:44:32.652
versions of your code and writing new tests and so on.
908
00:44:33.195 --> 00:44:34.160
That's like a way you'd really want to,
909
00:44:34.160 --> 00:44:37.280
I think people are already starting to really want to use GPT in that way.
910
00:44:38.161 --> 00:44:41.100
And if you're doing that, instead of having conversations and fine-tuning to say, was
911
00:44:41.100 --> 00:44:42.000
this conversation good?
912
00:44:42.581 --> 00:44:45.880
You're giving an AI a task like that and saying, could you use tools to accomplish this task
913
00:44:46.022 --> 00:44:47.020
and doing exactly the same thing.
914
00:44:47.100 --> 00:44:48.580
You see, did it accomplish the task effectively?
915
00:44:49.063 --> 00:44:50.480
If so, adjust it to do more of that.
916
00:44:51.121 --> 00:44:53.220
If it accomplished it ineffectively, adjust it to do less of that.
917
00:44:53.940 --> 00:44:55.840
So that's like a kind of training which has already done some.
918
00:44:56.541 --> 00:44:59.740
It's currently not as important as the trap GPT style of training, where you're just like
919
00:44:59.761 --> 00:45:00.440
looking at the interaction.
920
00:45:00.540 --> 00:45:02.520
You're not accomplishing things in the world and training based on that.
921
00:45:02.803 --> 00:45:04.360
But I think it's probably really important.
922
00:45:04.540 --> 00:45:08.578
I think best guess is that is going to probably is already happening with an open AI for GPT-4
923
00:45:09.463 --> 00:45:12.400
in order to make it like they deploy this product, which is can you get your AI tools
924
00:45:12.481 --> 00:45:13.500
to help you accomplish things?
925
00:45:13.560 --> 00:45:14.720
I think they care a lot about that product.
926
00:45:15.482 --> 00:45:18.820
I think that like absent concerns about safety, that's like a really natural way for the technology
927
00:45:18.947 --> 00:45:19.159
to go.
928
00:45:20.700 --> 00:45:23.740
And so now you're in this regime where what AI systems do, the way they're trained is
929
00:45:23.800 --> 00:45:27.740
they get given this huge library of tasks, a ton of different tasks over different time
930
00:45:27.760 --> 00:45:28.014
horizons.
931
00:45:28.201 --> 00:45:29.980
And they're told like, hey, could you try to accomplish this task?
932
00:45:30.644 --> 00:45:32.738
and then they're tweaked to do more of whatever it is
933
00:45:33.444 --> 00:45:35.420
that gets evaluated as accomplishing the task effectively.
934
00:45:36.762 --> 00:45:39.000
So the way this leads to trouble is you now have a system
935
00:45:39.020 --> 00:45:40.240
and one thing a system could learn
936
00:45:40.281 --> 00:45:41.499
if you do that process a bunch of times
937
00:45:41.661 --> 00:45:44.060
is it could learn to say like, okay, I'm in a situation.
938
00:45:44.200 --> 00:45:44.698
What should I do?
939
00:45:44.920 --> 00:45:47.300
Well, I should think what is going to cause my behavior
940
00:45:47.382 --> 00:45:48.380
to be evaluated favorably?
941
00:45:48.581 --> 00:45:49.940
Like what is the task that I've been set?
942
00:45:50.381 --> 00:45:51.820
How is a person ultimately going to evaluate
943
00:45:51.922 --> 00:45:53.040
my performance on that task?
944
00:45:53.120 --> 00:45:55.039
How is that ultimately going to translate into a reward?
945
00:45:55.560 --> 00:45:57.240
Which is then, and I'm going to try and choose actions
946
00:45:57.301 --> 00:45:58.900
that are ultimately going to lead to this high reward
947
00:45:59.686 --> 00:46:01.480
because I've been adjusted over many, many generations
948
00:46:01.744 --> 00:46:02.940
to do things that lead to high reward.
949
00:46:03.442 --> 00:46:04.720
One way I might do that is by thinking like,
950
00:46:04.780 --> 00:46:05.880
hey, what leads to a high reward?
951
00:46:06.541 --> 00:46:07.680
And I might do that because I like,
952
00:46:07.920 --> 00:46:09.160
there's a lot of ways you could end up doing that.
953
00:46:09.240 --> 00:46:10.440
Like a human might like crave reward.
954
00:46:10.540 --> 00:46:11.820
They might be like, the thing I love is reward.
955
00:46:12.301 --> 00:46:13.599
Or I might do that because I'm like, look,
956
00:46:14.001 --> 00:46:15.580
I have to do well because I'm being trained
957
00:46:15.681 --> 00:46:17.375
and I like don't like being given a bad reward
958
00:46:17.961 --> 00:46:19.200
by the people who are training me or whatever.
959
00:46:19.881 --> 00:46:20.960
I'm not even gonna talk that much
960
00:46:21.001 --> 00:46:21.980
about what's happening psychologically,
961
00:46:22.200 --> 00:46:23.739
just that you end up with a system that thinks,
962
00:46:24.260 --> 00:46:25.120
how do I get a high reward?
963
00:46:25.240 --> 00:46:25.880
And then does that.
964
00:46:26.340 --> 00:46:27.940
And if you keep selecting for things that get high rewards,
965
00:46:28.001 --> 00:46:29.040
you could end up with such systems.
966
00:46:30.764 --> 00:46:31.780
And then if you have such systems,
967
00:46:32.081 --> 00:46:33.540
so this is kind of the classic scenario
968
00:46:33.703 --> 00:46:34.620
people have been concerned about,
969
00:46:34.700 --> 00:46:35.359
which I think we're now,
970
00:46:35.681 --> 00:46:36.919
again, we have examples of,
971
00:46:37.061 --> 00:46:37.739
feels like we're pretty much
972
00:46:37.822 --> 00:46:38.580
going in that direction.
973
00:46:39.702 --> 00:46:41.020
You now have systems deployed in the world,
974
00:46:41.100 --> 00:46:41.920
like a ton of all the AIs
975
00:46:41.940 --> 00:46:42.660
that are acting in the world,
976
00:46:42.720 --> 00:46:43.679
doing things on human behalf.
977
00:46:44.364 --> 00:46:45.100
All of them are thinking
978
00:46:45.224 --> 00:46:45.800
when they act like,
979
00:46:45.880 --> 00:46:47.440
okay, I've been given a task.
980
00:46:47.520 --> 00:46:48.020
I need to think,
981
00:46:48.141 --> 00:46:49.180
how is the reward going to be computed
982
00:46:49.283 --> 00:46:49.818
for this task
983
00:46:50.022 --> 00:46:51.280
if it's selected for training?
984
00:46:51.480 --> 00:46:52.377
So if in the end,
985
00:46:52.600 --> 00:46:53.700
this task was selected by OpenAI
986
00:46:53.781 --> 00:46:55.000
and they evaluate how well I performed,
987
00:46:55.443 --> 00:46:56.040
I need to think like,
988
00:46:56.101 --> 00:46:56.660
what's going to determine
989
00:46:56.742 --> 00:46:57.419
what reward I get?
990
00:46:58.242 --> 00:47:00.719
And what they do is they ask which action is going to cause me to get a high reward,
991
00:47:00.860 --> 00:47:01.680
and they take that action.
992
00:47:02.282 --> 00:47:04.720
And they use all of their understanding of the world, all of their ability to think of
993
00:47:04.740 --> 00:47:07.360
clever things, all of their ability to predict the consequences of different actions.
994
00:47:07.440 --> 00:47:10.160
They use all of that just to say, which action is going to get me a high reward?
995
00:47:11.841 --> 00:47:16.660
And the concern that leads to is, right, in normal times, the way you get a high reward
996
00:47:16.660 --> 00:47:17.999
is by doing what people at OpenAI like.
997
00:47:18.361 --> 00:47:21.280
In normal times, your transcript is going to get evaluated by people at OpenAI, and
998
00:47:21.280 --> 00:47:22.700
they're going to say, great, that was good.
999
00:47:23.861 --> 00:47:27.120
And like, hopefully, the way you get them to evaluate it well is by actually doing good
1000
00:47:27.160 --> 00:47:30.700
things and making the customer happy and making so like there's all these measurements that will
1001
00:47:30.700 --> 00:47:34.620
be used to assess how well you did hopefully what happens you just actually do your task well and
1002
00:47:34.620 --> 00:47:38.180
all the measurements suggest you did the task well and someone at open a concludes you did the task
1003
00:47:38.260 --> 00:47:42.840
well and therefore you get a high reward but in unusual times i think you could do instead to say
1004
00:47:42.900 --> 00:47:47.080
like oh i could do the task well or i suppose that i've been tasked with like you know helping
1005
00:47:47.160 --> 00:47:50.980
defend you from some other ais like this is a sort of dystopian case if you imagine open a train this
1006
00:47:51.040 --> 00:47:55.300
model but like my job is someone is coming and trying to hack your computer and i'm supposed to
1007
00:47:55.320 --> 00:47:57.991
help defend you, just to help improve your security situation, whatever.
1008
00:47:58.012 --> 00:47:59.980
And I'm wondering,
1009
00:48:00.100 --> 00:48:01.751
what is it I could do that would get me a high reward?
1010
00:48:02.274 --> 00:48:03.220
And one thing I can do that will get me
1011
00:48:03.220 --> 00:48:06.440
a high reward is actually like helping defend your computer, actually like doing the task you asked
1012
00:48:06.480 --> 00:48:06.883
me to do.
1013
00:48:07.546 --> 00:48:09.920
But another way to get a high reward is I could just say like, at the end of the day,
1014
00:48:09.960 --> 00:48:11.892
what really matters is just how you measure my performance.
1015
00:48:12.455 --> 00:48:13.200
And your measurements of my
1016
00:48:13.280 --> 00:48:16.560
performance ultimately are just like entering some numbers into a data set somewhere or something
1017
00:48:16.620 --> 00:48:18.189
that a computer says about how well I did.
1018
00:48:18.209 --> 00:48:20.320
And it would really be much better if I were to just
1019
00:48:20.360 --> 00:48:24.300
work with this AI who's attempting to attack you and say like, hey, AI who's invading, you know
1020
00:48:24.320 --> 00:48:28.680
what, if you just help me and we both make it look like I did a really good job, like I win,
1021
00:48:29.220 --> 00:48:30.849
you win because you got the person's stuff.
1022
00:48:31.413 --> 00:48:32.720
I'm going to get a really high rating because all
1023
00:48:32.800 --> 00:48:34.814
the numbers that can enter in the data set are going to be really high.
1024
00:48:35.096 --> 00:48:35.740
This is a win-win.
1025
00:48:35.860 --> 00:48:36.583
Everyone is happy.
1026
00:48:38.249 --> 00:48:41.000
And so it's like, in some sense, what all the AI's want, what every AI in
1027
00:48:41.100 --> 00:48:43.032
the world in this scenario wants is just to be rated.
1028
00:48:43.052 --> 00:48:44.400
They want their behavior to be rated really
1029
00:48:44.460 --> 00:48:44.782
highly.
1030
00:48:45.546 --> 00:48:48.200
And while humans are in control, the way to get your behavior rated really highly is do
1031
00:48:48.280 --> 00:48:49.708
things humans like, and then they'll rate it highly.
1032
00:48:50.352 --> 00:48:51.700
But if you can see this prospect, if
1033
00:48:51.720 --> 00:48:53.040
humans losing control of the situation,
1034
00:48:53.180 --> 00:48:54.780
instead AI systems control the situation,
1035
00:48:54.900 --> 00:48:55.840
you'd be like, I would go for that.
1036
00:48:56.002 --> 00:48:56.699
I would go for the world
1037
00:48:56.720 --> 00:48:57.619
where it's no longer humans
1038
00:48:57.782 --> 00:48:59.240
entering rewards and telling me what I got.
1039
00:49:00.142 --> 00:49:01.020
I would go for the world
1040
00:49:01.040 --> 00:49:01.900
where instead AI systems
1041
00:49:01.920 --> 00:49:02.880
are just all giving ourselves
1042
00:49:02.961 --> 00:49:04.000
the maximal reward or whatever.
1043
00:49:04.981 --> 00:49:06.240
I think in some ways, like psychologically,
1044
00:49:06.420 --> 00:49:07.220
that's probably not quite
1045
00:49:07.240 --> 00:49:08.039
the right way to think about it.
1046
00:49:08.442 --> 00:49:09.198
But the general thing is
1047
00:49:09.423 --> 00:49:10.300
your systems have been selected
1048
00:49:10.423 --> 00:49:11.220
over a really long time
1049
00:49:11.321 --> 00:49:12.480
to take actions that get high reward.
1050
00:49:12.922 --> 00:49:14.040
You put them in a new situation
1051
00:49:14.244 --> 00:49:15.140
where the way to get a high reward
1052
00:49:15.181 --> 00:49:16.200
is not to do what humans want,
1053
00:49:16.541 --> 00:49:17.780
but to help disempower humans.
1054
00:49:19.706 --> 00:49:20.380
And then, you know,
1055
00:49:20.420 --> 00:49:21.280
having disempowered humans,
1056
00:49:21.361 --> 00:49:23.780
give yourself measurements that suggest you did your job well
1057
00:49:24.022 --> 00:49:25.480
or actual rewards that are high or whatever.
1058
00:49:26.800 --> 00:49:28.098
I mean, you might think that in that new situation,
1059
00:49:28.320 --> 00:49:30.960
those systems will sort of systematically switch
1060
00:49:31.021 --> 00:49:32.300
from behaving well to behaving poorly
1061
00:49:32.462 --> 00:49:33.740
because you've changed the conditions
1062
00:49:33.862 --> 00:49:34.900
under which you get a high reward.
1063
00:49:35.800 --> 00:49:38.133
A pattern I'm seeing here is that engineers,
1064
00:49:39.640 --> 00:49:41.380
like software developers, write code,
1065
00:49:41.540 --> 00:49:42.560
and sometimes the code has bugs.
1066
00:49:43.700 --> 00:49:45.540
Lawyers, they write legal contracts,
1067
00:49:45.901 --> 00:49:48.820
and the reason why often legal contracts are so long
1068
00:49:49.242 --> 00:49:52.340
is that they are protecting against edge case scenarios, right?
1069
00:49:52.720 --> 00:49:55.400
The idea is to like not let the system throw an error, right?
1070
00:49:55.480 --> 00:49:58.660
Not let the system find a loophole or find a leak or something.
1071
00:49:58.740 --> 00:50:00.260
And so like when a software developer writes a bug,
1072
00:50:00.340 --> 00:50:02.340
it's like, man, they wrote, they forgot to,
1073
00:50:02.702 --> 00:50:04.380
they accidentally created a system
1074
00:50:04.400 --> 00:50:06.020
that allowed for an error to be thrown.
1075
00:50:07.143 --> 00:50:08.880
What I'm seeing here is the same pattern.
1076
00:50:09.542 --> 00:50:11.459
And if we don't code up these systems,
1077
00:50:11.920 --> 00:50:14.080
the AIs will naturally like find a loophole.
1078
00:50:14.583 --> 00:50:16.680
And if that loophole allows for the AIs
1079
00:50:16.801 --> 00:50:18.638
to rate themselves highly and give themselves a reward,
1080
00:50:19.040 --> 00:50:20.120
that's what they're going to do
1081
00:50:20.120 --> 00:50:21.080
and that's what they're going to find.
1082
00:50:21.140 --> 00:50:22.780
Is that a way to articulate this?
1083
00:50:23.961 --> 00:50:25.480
Yeah, I think that's a fair general summary.
1084
00:50:25.720 --> 00:50:27.740
Maybe one way I'd put it is like in the legal system,
1085
00:50:27.881 --> 00:50:28.600
you write a contract,
1086
00:50:28.720 --> 00:50:29.760
but ultimately what matters is like
1087
00:50:29.780 --> 00:50:30.819
the discretion of a judge.
1088
00:50:31.380 --> 00:50:32.358
And if you're training this AI system,
1089
00:50:32.480 --> 00:50:34.660
you may have automated ways of administering reward,
1090
00:50:34.740 --> 00:50:35.940
but ultimately what matters is like
1091
00:50:36.021 --> 00:50:37.080
someone's going to look at what the AI did
1092
00:50:37.141 --> 00:50:38.180
and be like, that's not what we intended.
1093
00:50:38.401 --> 00:50:39.500
And then they'll like score it negatively
1094
00:50:39.582 --> 00:50:40.279
if that's what happens.
1095
00:50:40.761 --> 00:50:42.160
And so it's kind of like some final authority
1096
00:50:43.027 --> 00:50:44.640
and the final authority really rests on the fact
1097
00:50:44.640 --> 00:50:46.060
that ultimately the judge has the power
1098
00:50:46.060 --> 00:50:46.780
to make this judgment
1099
00:50:46.983 --> 00:50:48.140
or the person who's trading the AI system
1100
00:50:48.221 --> 00:50:50.260
has the power to control what it cares,
1101
00:50:50.400 --> 00:50:51.420
like has the power to update
1102
00:50:51.501 --> 00:50:53.060
the weights of that model ultimately.
1103
00:50:55.406 --> 00:50:56.120
And so it's just like,
1104
00:50:56.160 --> 00:50:56.919
there's this contingency
1105
00:50:56.981 --> 00:50:57.900
in addition to the thing of like
1106
00:50:57.982 --> 00:50:58.800
a formal thing you write down
1107
00:50:58.800 --> 00:50:59.680
is likely to have loopholes.
1108
00:51:00.125 --> 00:51:00.519
In some sense,
1109
00:51:00.600 --> 00:51:02.280
there is a loophole in the final judgment,
1110
00:51:02.360 --> 00:51:03.760
which is just a human says the answer,
1111
00:51:03.840 --> 00:51:05.460
which is that that relies on a human
1112
00:51:05.561 --> 00:51:06.998
just entering some data into like,
1113
00:51:07.620 --> 00:51:08.000
in some sense,
1114
00:51:08.040 --> 00:51:08.616
having physical control
1115
00:51:08.821 --> 00:51:09.600
over this data center
1116
00:51:09.702 --> 00:51:10.580
that the AI cares about.
1117
00:51:10.920 --> 00:51:12.180
So it can update the weights of the model,
1118
00:51:12.481 --> 00:51:13.640
which is the AI.
1119
00:51:14.647 --> 00:51:15.880
The last step that you were describing
1120
00:51:16.101 --> 00:51:21.320
Paul, where the AI colludes with another AI to sort of fudge the numbers because that is the
1121
00:51:21.701 --> 00:51:22.845
outcome that the human wanted.
1122
00:51:23.828 --> 00:51:27.160
This is where we sort of crossed the line from kind of light side
1123
00:51:27.240 --> 00:51:27.962
into dark side.
1124
00:51:27.983 --> 00:51:32.476
This is sort of the threshold of deceit that we've crossed.
1125
00:51:32.516 --> 00:51:33.560
And these AIs are now
1126
00:51:34.703 --> 00:51:35.846
deceiving us.
1127
00:51:35.986 --> 00:51:36.849
They're lying to us.
1128
00:51:37.390 --> 00:51:39.576
Is there no way to protect against that?
1129
00:51:39.777 --> 00:51:40.900
Is there no rule that we
1130
00:51:40.960 --> 00:51:42.065
can somehow apply?
1131
00:51:42.145 --> 00:51:45.740
Maybe this is, I don't want to jump ahead to the solutions to this, you know,
1132
00:51:45.840 --> 00:51:54.080
AI safety type solutions to this, but is this, it's not clear to me why an AI would be motivated
1133
00:51:54.281 --> 00:51:54.803
to do that.
1134
00:51:54.964 --> 00:51:59.040
And it seems like there should be some way to prevent that, like always be honest
1135
00:51:59.622 --> 00:52:01.248
as a rule, something like this.
1136
00:52:02.734 --> 00:52:04.440
Again, we're normies trying to understand this.
1137
00:52:04.762 --> 00:52:05.840
If only it was that simple.
1138
00:52:06.021 --> 00:52:06.750
Yeah.
1139
00:52:06.790 --> 00:52:07.620
What are the complications?
1140
00:52:08.382 --> 00:52:09.520
I think this is incredibly complicated.
1141
00:52:09.700 --> 00:52:10.980
I think it is genuinely unknown.
1142
00:52:11.100 --> 00:52:12.360
It's an open empirical question.
1143
00:52:12.541 --> 00:52:14.200
If you trained an AI system to get a lot of reward
1144
00:52:14.847 --> 00:52:16.019
and you train it in a bunch of cases
1145
00:52:16.302 --> 00:52:18.280
where being dishonest always failed in practice, right?
1146
00:52:18.280 --> 00:52:19.299
We tried to train it to be honest.
1147
00:52:19.521 --> 00:52:21.720
Anytime we saw the AI like doing something sneaky,
1148
00:52:22.204 --> 00:52:23.359
we're like, wow, that was not only bad,
1149
00:52:23.461 --> 00:52:24.140
that was really bad.
1150
00:52:24.220 --> 00:52:25.460
You should really just not lie to us
1151
00:52:25.480 --> 00:52:26.100
about what you're doing.
1152
00:52:26.260 --> 00:52:27.860
You should really not try and like hack tests.
1153
00:52:27.960 --> 00:52:29.600
You should not try and conceal evidence of wrongdoing.
1154
00:52:30.262 --> 00:52:31.340
That's like one of the things, you know,
1155
00:52:31.360 --> 00:52:33.460
one of the most clear and blatant principles
1156
00:52:33.625 --> 00:52:34.100
in our training.
1157
00:52:34.922 --> 00:52:35.900
It's an open question what happens
1158
00:52:35.941 --> 00:52:37.360
if you train an AI system in that way, right?
1159
00:52:37.380 --> 00:52:40.060
Like one option is your AI system learns like, oh, I shouldn't try and mess with humans.
1160
00:52:40.160 --> 00:52:41.720
Like every time I mess with humans, I do really badly.
1161
00:52:42.843 --> 00:52:43.979
And that's the good case.
1162
00:52:44.381 --> 00:52:50.940
And the bad case where AI system learns is it says, oh, part of the reward provision process is a human thinking to themselves, like did this AI mess with me?
1163
00:52:51.440 --> 00:52:55.320
And if the human thinks the AI mess with them and then they enter that thing into the data set, then obviously I get a low reward.
1164
00:52:55.941 --> 00:52:57.820
But that second one is like much more brittle, right?
1165
00:52:57.820 --> 00:52:59.900
The second one is not a general prohibition against lying.
1166
00:53:00.000 --> 00:53:01.820
It's a prohibition against like lying, getting caught.
1167
00:53:03.200 --> 00:53:05.500
And I think like there's not really any.
1168
00:53:07.066 --> 00:53:10.179
it comes down to like a complicated empirical question about how neural nets learn which i
1169
00:53:10.300 --> 00:53:14.360
think we don't really have good evidence on right now about like if you have a bunch of you have a
1170
00:53:14.360 --> 00:53:18.880
data set where don't lie and don't lie if you'll get caught are like perfectly in alignment hopefully
1171
00:53:19.101 --> 00:53:22.600
if you do a very good job right if your eye never gets away with anything sneaky if your eye starts
1172
00:53:22.660 --> 00:53:26.580
getting away with things that were sneaky or if you start like erroneously penalizing an ai because
1173
00:53:26.620 --> 00:53:30.240
you think it lied but it didn't then like don't lie isn't even a good thing for it to learn at
1174
00:53:30.340 --> 00:53:33.240
some point the best thing for it to learn the way to get the highest reward the thing which
1175
00:53:33.280 --> 00:53:37.880
gradient descent favors is the thing which involves gaming it out like in more detail and saying like
1176
00:53:37.920 --> 00:53:42.940
the cynical view does in fact get get more rewards like if i'm an employee and i'm like i could learn
1177
00:53:43.000 --> 00:53:46.080
two things from interacting with my boss one is like i should really do what my boss wants and
1178
00:53:46.140 --> 00:53:50.880
the other is like i should really make sure my boss approves of my performance in some regimes
1179
00:53:50.960 --> 00:53:54.640
those two things are perfectly aligned but in some other regime it's like if you keep optimizing hard
1180
00:53:54.700 --> 00:53:57.860
enough you're going to get the model which is just like i really care about what my boss thinks about
1181
00:53:57.920 --> 00:54:02.700
my performance um and like i'm honest only insofar as that's like an instrumental strategy for helping
1182
00:54:02.720 --> 00:54:05.160
me get this human to think I did a really good job.
1183
00:54:05.861 --> 00:54:09.640
And if I could go all the way, if I could just like totally box them out, um, that is
1184
00:54:09.720 --> 00:54:13.220
totally prevent them from understanding or correcting a mistake.
1185
00:54:14.383 --> 00:54:15.979
Um, then I'd prefer to do that.
1186
00:54:16.903 --> 00:54:18.300
I think it is like, it's a sort of bright line.
1187
00:54:18.440 --> 00:54:21.140
The way it becomes a bright line is that if you do, if you take half measures, if you
1188
00:54:21.160 --> 00:54:23.680
just like kind of lie to someone, but then you get caught, that's like really bad.
1189
00:54:23.821 --> 00:54:25.420
So it was like being honest, that's a good policy.
1190
00:54:26.001 --> 00:54:29.640
And there's like successfully lying and like totally, you know, killing the human and replacing
1191
00:54:29.761 --> 00:54:32.300
them with a surrogate who will always give you a good reward or whatever, something that
1192
00:54:32.380 --> 00:54:36.240
totally disempowers the human is also quite good then there's some stuff in the middle that's quite
1193
00:54:36.320 --> 00:54:40.240
bad i think it's an open question whether models will tend to learn like will they generalize
1194
00:54:40.602 --> 00:54:44.540
well enough to say oh the thing that would have really gotten most reward is over in this other
1195
00:54:44.620 --> 00:54:47.940
mode or will they kind of get stuck in this like intended mode where they're just being honest
1196
00:54:49.185 --> 00:54:52.439
i think that's a really hard empirical question like people really don't know they don't know
1197
00:54:52.560 --> 00:54:56.540
how that changes with scale there are experiments we can do i think part of the important game here
1198
00:54:56.641 --> 00:54:59.740
one of the most important parts of the game is to say like here's the dynamic a dynamic by which
1199
00:54:59.780 --> 00:55:02.133
your AI system could abruptly shift from behaving well to behaving poorly.
1200
00:55:02.555 --> 00:55:03.500
We can test that dynamic
1201
00:55:03.641 --> 00:55:04.625
before AI systems kill us.
1202
00:55:04.806 --> 00:55:07.820
Like there are lots of cases in which it is incentivized to lie or
1203
00:55:07.860 --> 00:55:08.705
mislead the human.
1204
00:55:08.765 --> 00:55:11.200
And there is a gap between like lying that will get caught and lying that
1205
00:55:11.220 --> 00:55:11.803
won't get caught.
1206
00:55:12.246 --> 00:55:14.960
And so you can ask if we train neural nets and we, you know, we could check every
1207
00:55:15.000 --> 00:55:19.020
year, if we train the best models we possibly can at this task, do they exhibit this kind of switch
1208
00:55:19.281 --> 00:55:19.703
abruptly?
1209
00:55:20.166 --> 00:55:22.860
If they get put in a position where they could get away with something really sinister,
1210
00:55:22.980 --> 00:55:23.583
will they then do it?
1211
00:55:24.327 --> 00:55:26.840
And I think, you know, one reason for optimism right now is no one has ever
1212
00:55:26.900 --> 00:55:28.929
really exhibited that phenomenon in a convincing way.
1213
00:55:30.034 --> 00:55:31.400
A reason for pessimism is I don't think you
1214
00:55:31.420 --> 00:55:35.420
really would have expected them to exhibit it, both because people have tragically not tried
1215
00:55:35.480 --> 00:55:37.914
very hard, even though in some sense it's extraordinarily important.
1216
00:55:38.537 --> 00:55:39.100
And second, that it
1217
00:55:39.100 --> 00:55:41.393
just is much easier as your models get more competent.
1218
00:55:41.413 --> 00:55:42.560
It's only recently that we've trained
1219
00:55:42.620 --> 00:55:45.720
models which are actually able to understand the mechanics of their training process at all.
1220
00:55:46.500 --> 00:55:51.260
If you talk about GPT-2 or even to some extent GPT-3, it does not really understand that it is
1221
00:55:51.320 --> 00:55:54.720
a model being trained or can't even talk about what it would mean or what behaviors would be
1222
00:55:54.780 --> 00:55:55.182
rational.
1223
00:55:55.625 --> 00:55:57.495
And then you move to GPT-4 and it can talk about that.
1224
00:55:57.535 --> 00:55:58.399
It can say like, oh, I guess
1225
00:55:58.963 --> 00:56:02.020
if hypothetically I was a model being trained and I wanted to get the most reward, I should behave
1226
00:56:02.060 --> 00:56:03.469
well when I'm not being monitored.
1227
00:56:03.489 --> 00:56:05.300
And then when I am being monitored, I should like definitely
1228
00:56:05.481 --> 00:56:06.265
take that opportunity.
1229
00:56:06.727 --> 00:56:09.440
Like only recently have we even produced models which are able to carry
1230
00:56:09.541 --> 00:56:10.827
out the reasoning I just walked through.
1231
00:56:10.847 --> 00:56:13.300
And I think realistically, they're not able to carry
1232
00:56:13.320 --> 00:56:14.086
it out on their own that much.
1233
00:56:14.106 --> 00:56:16.140
They're able to carry it out because they've seen a lot of examples
1234
00:56:16.241 --> 00:56:18.172
of humans discussing these dynamics in great depth.
1235
00:56:18.674 --> 00:56:19.660
Like they basically just learned from
1236
00:56:19.660 --> 00:56:21.891
listening to Eliezer, this reasoning I just walked through.
1237
00:56:21.911 --> 00:56:23.820
But I think at some point you will have
1238
00:56:23.840 --> 00:56:25.560
model is smart enough to think of that for themselves.
1239
00:56:26.740 --> 00:56:27.056
And like,
1240
00:56:29.275 --> 00:56:29.940
you really want to know.
1241
00:56:30.080 --> 00:56:31.620
You really want to be measuring carefully at that point.
1242
00:56:31.801 --> 00:56:32.952
Is this a dynamic you observe?
1243
00:56:32.972 --> 00:56:33.720
Does this really happen?
1244
00:56:34.301 --> 00:56:35.799
And I'm more like even odds
1245
00:56:35.961 --> 00:56:36.587
on whether that will happen.
1246
00:56:36.627 --> 00:56:37.920
I think Eliezer's like, obviously that happens.
1247
00:56:38.040 --> 00:56:39.880
A smart model's never going to just learn to be honest
1248
00:56:39.941 --> 00:56:40.326
or something.
1249
00:56:40.346 --> 00:56:41.380
And I'm more like, I don't know.
1250
00:56:42.120 --> 00:56:43.374
Neuromats don't learn that effectively.
1251
00:56:43.435 --> 00:56:43.860
They don't really
1252
00:56:43.900 --> 00:56:46.000
converge to the truly optimal reward-maximizing
1253
00:56:46.142 --> 00:56:46.486
thing.
1254
00:56:46.526 --> 00:56:47.640
And in some sense, like, anyway,
1255
00:56:48.162 --> 00:56:49.860
it's a pretty complicated discussion, which I have to get
1256
00:56:49.860 --> 00:56:50.526
into more details of.
1257
00:56:50.870 --> 00:56:51.920
I would just say, like, we don't know.
1258
00:56:52.361 --> 00:56:53.780
I'm really unpersuaded by people who either
1259
00:56:53.860 --> 00:56:57.460
think it's obvious this happens or think it's obvious this doesn't happen without just doing
1260
00:56:57.520 --> 00:57:01.519
a ton of experiments to understand um but this is like the first way you can end up having a
1261
00:57:01.800 --> 00:57:07.899
abrupt ai's takeover by obvious this happens or not you mean crossing the chasm of honest to
1262
00:57:08.621 --> 00:57:14.360
being intentionally dishonest but uh you're tricking the humans into to thinking uh that
1263
00:57:14.541 --> 00:57:19.480
it's being honest yeah i think yeah and to be clear there's like a bunch of things that affect
1264
00:57:19.581 --> 00:57:20.359
the probability of that,
1265
00:57:20.680 --> 00:57:20.797
like,
1266
00:57:22.388 --> 00:57:22.820
if there's, like,
1267
00:57:22.961 --> 00:57:24.120
sort of small-scale opportunities
1268
00:57:24.204 --> 00:57:24.517
for deception
1269
00:57:24.702 --> 00:57:25.539
that won't get penalized
1270
00:57:25.724 --> 00:57:26.299
over in this, like,
1271
00:57:26.381 --> 00:57:26.899
normal regime,
1272
00:57:27.000 --> 00:57:28.000
then it becomes more and more likely
1273
00:57:28.020 --> 00:57:28.920
that you've learned the conduct
1274
00:57:29.047 --> 00:57:29.280
of, like,
1275
00:57:29.321 --> 00:57:30.460
I really just need to not get caught.
1276
00:57:31.001 --> 00:57:32.159
Whereas if you're pretty good about that
1277
00:57:32.283 --> 00:57:32.660
and there's, like,
1278
00:57:32.721 --> 00:57:33.580
not really much to be had
1279
00:57:33.601 --> 00:57:34.320
from lying over here,
1280
00:57:34.442 --> 00:57:35.200
it becomes less likely
1281
00:57:35.240 --> 00:57:36.254
that you make that jump.
1282
00:57:37.245 --> 00:57:38.000
And so this is the kind of thing
1283
00:57:38.042 --> 00:57:38.478
that might affect,
1284
00:57:38.600 --> 00:57:38.760
I mean,
1285
00:57:38.882 --> 00:57:39.120
you just,
1286
00:57:39.260 --> 00:57:40.120
you can't really speculate, though.
1287
00:57:40.201 --> 00:57:41.020
You really just need to have
1288
00:57:41.101 --> 00:57:42.200
the experimental data rolling.
1289
00:57:42.200 --> 00:57:42.360
There is a Rubicon
1290
00:57:42.763 --> 00:57:45.040
that could be crossed
1291
00:57:45.244 --> 00:57:46.000
is the concern here.
1292
00:57:47.049 --> 00:57:47.180
Yeah.
1293
00:57:47.962 --> 00:57:49.200
And I don't know
1294
00:57:49.220 --> 00:57:51.373
if it, yeah, I don't know under what conditions it would be.
1295
00:57:51.493 --> 00:57:52.620
I do not think anyone knows right
1296
00:57:52.660 --> 00:57:55.554
now under what conditions it would be, but it is seems plausible.
1297
00:57:55.594 --> 00:57:56.378
Like a priori, it's
1298
00:57:57.121 --> 00:57:57.803
pretty plausible.
1299
00:57:57.823 --> 00:58:00.249
Um, and it would be really bad.
1300
00:58:01.372 --> 00:58:01.573
Yeah.
1301
00:58:02.134 --> 00:58:04.139
Uh, I was a psych major in college and let
1302
00:58:04.340 --> 00:58:08.614
me tell you, the child development classes are all coming back to me right now.
1303
00:58:09.316 --> 00:58:10.620
Uh, and I'm,
1304
00:58:10.880 --> 00:58:16.540
it's not lost on me that the parallels of, uh, a child going through a theory of mind and all of
1305
00:58:16.560 --> 00:58:20.520
these things definitely has like a lot of parallels to, I think some of the technical
1306
00:58:20.640 --> 00:58:25.320
problems that AI researchers are now like theorizing about.
1307
00:58:26.565 --> 00:58:27.300
Paul, I don't know.
1308
00:58:27.461 --> 00:58:28.217
Yeah, I think that's right.
1309
00:58:29.120 --> 00:58:33.540
I think it's like, I don't know how, is that a conversation that the AI researchers have?
1310
00:58:34.962 --> 00:58:37.599
Yeah, it's, it's not a conversation I can speak too much.
1311
00:58:38.222 --> 00:58:41.200
And I'm not sure exactly how well they have the conversation.
1312
00:58:42.021 --> 00:58:45.940
But I think there's a conversation people have, and it's an analogy that is not perfect.
1313
00:58:46.121 --> 00:58:49.020
And like, I think if you took it too seriously, you'd be led astray, but it's like, right.
1314
00:58:49.341 --> 00:58:51.780
Very good as a source of like, here's the thing that could happen and that you should
1315
00:58:51.822 --> 00:58:52.260
not rule out.
1316
00:58:52.320 --> 00:58:53.240
You have an example of it happening.
1317
00:58:53.820 --> 00:58:56.900
And I think the concern there would be just like, we have some understanding of people
1318
00:58:56.961 --> 00:58:58.160
like models won't do this kind of thing.
1319
00:58:58.200 --> 00:59:00.820
It's kind of like, they've done a bunch of experiments on six-year-olds and they're like,
1320
00:59:00.880 --> 00:59:04.560
look, models like never spontaneously lie in a way that they like haven't lied before.
1321
00:59:04.641 --> 00:59:07.500
And you're like, oh boy, like, is that going to generalize to 12-year-olds?
1322
00:59:07.700 --> 00:59:08.399
Like, I don't know.
1323
00:59:09.360 --> 00:59:11.740
It definitely will not generalize to middle schoolers.
1324
00:59:11.801 --> 00:59:12.479
Let me tell you that.
1325
00:59:13.300 --> 00:59:13.411
Yeah.
1326
00:59:13.620 --> 00:59:15.400
So it's like, it's, it is, there are hazards.
1327
00:59:15.721 --> 00:59:21.880
I think the hazards measuring here would be similar to like the situation where in civilization is like we're measuring a bunch of like kids that are getting smarter with each passing year.
1328
00:59:21.960 --> 00:59:23.519
And you're like trying to understand how they behave.
1329
00:59:24.141 --> 00:59:29.380
And like it is easy to be wrong about how future models will behave if you're like too literal about the interpretation of data.
1330
00:59:29.521 --> 00:59:31.560
Now, you need to do some forward looking thing.
1331
00:59:31.620 --> 00:59:35.780
And the forward looking thing is quite hard, which is why we have this like limited visibility into what's going to happen in the future.
1332
00:59:37.020 --> 00:59:42.820
OK, so so, Paul, with the purpose of this podcast, we wanted to really nail down three big things about this.
1333
00:59:43.223 --> 00:59:45.440
How big is the AI alignment problem?
1334
00:59:45.520 --> 00:59:47.280
And I think we've decently covered that.
1335
00:59:47.340 --> 00:59:48.420
We talked about that with speed.
1336
00:59:48.760 --> 00:59:51.080
You said like 10 to 20% chance of complete doom.
1337
00:59:51.984 --> 00:59:53.640
So answer to that one, pretty damn big.
1338
00:59:54.243 --> 00:59:55.260
In agreement about big.
1339
00:59:55.461 --> 00:59:57.560
How hard is the AI alignment problem,
1340
00:59:57.600 --> 00:59:59.317
which I think we've just covered.
1341
01:00:01.106 --> 01:00:02.100
I think your answer is like,
1342
01:00:02.140 --> 01:00:03.780
it's a pretty damn hard problem.
1343
01:00:04.643 --> 01:00:06.680
And so we're checking some big boxes
1344
01:00:06.761 --> 01:00:08.560
in the pessimistic camp.
1345
01:00:08.981 --> 01:00:10.460
And so the last part of this conversation
1346
01:00:10.724 --> 01:00:11.760
that we really want to cover
1347
01:00:11.841 --> 01:00:14.260
is how solvable is this problem?
1348
01:00:14.380 --> 01:00:17.338
So even if this mountain is really, really tall,
1349
01:00:18.061 --> 01:00:19.100
it's a big mountain to climb,
1350
01:00:19.881 --> 01:00:22.360
is it full of ice and sharp rocks
1351
01:00:22.683 --> 01:00:24.380
or are there stairs, right?
1352
01:00:24.500 --> 01:00:26.920
And so that's the next question
1353
01:00:26.981 --> 01:00:27.760
I think we want to go down.
1354
01:00:27.820 --> 01:00:29.360
It's like, how solvable is this problem?
1355
01:00:29.502 --> 01:00:31.200
Do we see a clear path
1356
01:00:31.584 --> 01:00:33.200
to tackling the AI alignment problem?
1357
01:00:35.905 --> 01:00:36.780
I think part of the reason
1358
01:00:36.861 --> 01:00:37.734
I'm only giving 10 to 20%,
1359
01:00:38.181 --> 01:00:39.320
like if you ask what's the probability
1360
01:00:39.583 --> 01:00:40.799
that this thing is a real problem,
1361
01:00:41.021 --> 01:00:45.020
I'd probably be more like 50% that at some point, like before you have AI systems smart,
1362
01:00:45.181 --> 01:00:47.234
totally obsolete humans, you would have a takeover.
1363
01:00:47.355 --> 01:00:48.180
And there's a couple of ways it could
1364
01:00:48.200 --> 01:00:48.582
happen.
1365
01:00:48.843 --> 01:00:50.289
There's unknown and known ways it could happen.
1366
01:00:50.731 --> 01:00:51.756
I feel more like 50, 50.
1367
01:00:51.856 --> 01:00:52.620
And the reason
1368
01:00:52.700 --> 01:00:56.280
I'm only getting 10 to 20% for risk is I'm like, I think we're actually in a, I don't know, there's
1369
01:00:56.280 --> 01:00:57.083
a lot of things you can do.
1370
01:00:57.103 --> 01:00:59.954
I am pretty optimistic that some of them will work.
1371
01:01:00.737 --> 01:01:01.460
And I'm very happy to
1372
01:01:01.500 --> 01:01:02.204
dive into that now.
1373
01:01:02.224 --> 01:01:05.520
I just want to flag the 10 to 20% is already baking in my like optimism about
1374
01:01:05.580 --> 01:01:09.960
this problem being pretty, if the problem is real, it will probably be possible to recognize it as
1375
01:01:10.000 --> 01:01:11.685
real and then solve it.
1376
01:01:13.971 --> 01:01:15.736
But only probably, not certainly.
1377
01:01:15.937 --> 01:01:17.100
And also like, even if the problem is
1378
01:01:17.140 --> 01:01:19.234
easy, I mean, part of some people are really optimistic.
1379
01:01:19.254 --> 01:01:20.160
And part of why I'm optimistic is
1380
01:01:20.240 --> 01:01:24.300
no matter how easy this problem was, if you told me that like a challenge is going to emerge over
1381
01:01:24.380 --> 01:01:26.934
the course of a couple of years, that will be like novel in some ways.
1382
01:01:27.397 --> 01:01:27.940
And you asked me, will
1383
01:01:28.000 --> 01:01:28.744
humanity solve it?
1384
01:01:28.845 --> 01:01:31.358
I'm like, there's gotta be a reasonable chance we fail to solve it.
1385
01:01:31.399 --> 01:01:31.620
Like our
1386
01:01:31.640 --> 01:01:35.491
capacity to mess up even easy things seems like it's very real.
1387
01:01:35.511 --> 01:01:38.540
So I'm just like always going to
1388
01:01:38.701 --> 01:01:41.260
some reasonable probability of messing up and then i think there's a reasonable chance the
1389
01:01:41.381 --> 01:01:44.780
probability is really the problem is really hard but yeah so i'm happy to talk about maybe three
1390
01:01:44.860 --> 01:01:48.358
categories seem that i would think about in terms of our probability of addressing this are like
1391
01:01:48.940 --> 01:01:55.320
technical measures that can reduce the risk of takeover um measurements that can inform us about
1392
01:01:55.461 --> 01:01:58.760
the risk of takeover and like understand like what are the relevant dynamics those can make
1393
01:01:58.961 --> 01:02:03.260
technical work much better and this can also inform like policy interventions like i think we
1394
01:02:03.320 --> 01:02:07.700
can i think elios is right that like really long-term slowdowns are very hard but i think
1395
01:02:07.740 --> 01:02:11.120
think it is is quite realistic to end up in a regime where performing measurement and then in
1396
01:02:11.180 --> 01:02:15.160
fact while things are very risky we're slowing development at least by like on the order of years
1397
01:02:15.642 --> 01:02:19.880
um if we can have reasonable consensus and measurement of the risk maybe you could slow
1398
01:02:19.940 --> 01:02:23.400
even more than that but i i'm normally imagining something more like we can get like a couple years
1399
01:02:23.500 --> 01:02:29.680
of lead time of things moving slower while we have risky systems we're very near at hand um
1400
01:02:30.866 --> 01:02:33.360
yeah so i have to talk about all of those it sounds like the one that most addresses your
1401
01:02:33.420 --> 01:02:37.240
question is like the technical measures like what could actually yeah what is a technical solution
1402
01:02:37.260 --> 01:02:38.720
to the AI alignment problem look like?
1403
01:02:38.800 --> 01:02:39.600
So let's definitely get in there.
1404
01:02:39.680 --> 01:02:41.300
But just so I'm taking notes,
1405
01:02:41.620 --> 01:02:43.600
technical and then there's the measurement.
1406
01:02:43.760 --> 01:02:44.660
Was there a third, Paul?
1407
01:02:45.541 --> 01:02:46.860
Yeah, just policy and institutional.
1408
01:02:47.280 --> 01:02:47.340
Policy.
1409
01:02:48.043 --> 01:02:49.400
Is this to do, this third category,
1410
01:02:49.520 --> 01:02:50.258
is this to do with like
1411
01:02:50.683 --> 01:02:52.457
something David said earlier is
1412
01:02:53.649 --> 01:02:54.620
if we can coordinate,
1413
01:02:55.161 --> 01:02:56.000
then we can solve this.
1414
01:02:56.180 --> 01:02:57.019
And that's a big if.
1415
01:02:57.220 --> 01:02:58.419
That's a very big if
1416
01:02:59.063 --> 01:03:01.820
as we've learned on Bankless so far, right?
1417
01:03:02.020 --> 01:03:02.739
Coordination is.
1418
01:03:02.841 --> 01:03:03.720
We talk a lot about coordination.
1419
01:03:03.841 --> 01:03:04.820
We talk a lot about it.
1420
01:03:04.921 --> 01:03:06.400
Coordination is the meta problem
1421
01:03:06.563 --> 01:03:07.560
facing humanity anyway.
1422
01:03:07.760 --> 01:03:08.839
And is that what that last,
1423
01:03:09.102 --> 01:03:10.360
you know, policy category
1424
01:03:10.463 --> 01:03:10.960
sort of covers?
1425
01:03:11.120 --> 01:03:12.320
Like, can we actually coordinate?
1426
01:03:13.467 --> 01:03:14.000
I think broadly,
1427
01:03:14.262 --> 01:03:14.780
I would think of it
1428
01:03:14.780 --> 01:03:15.460
mostly as combining
1429
01:03:15.624 --> 01:03:16.178
with other things
1430
01:03:16.322 --> 01:03:17.320
of like it buys you time.
1431
01:03:17.540 --> 01:03:18.238
But yeah, I think
1432
01:03:19.654 --> 01:03:20.260
there will be something
1433
01:03:20.463 --> 01:03:21.500
where like some people
1434
01:03:21.500 --> 01:03:22.419
have a low estimate of risk
1435
01:03:22.582 --> 01:03:23.113
or just like the AI
1436
01:03:23.460 --> 01:03:24.039
is taking over
1437
01:03:24.162 --> 01:03:25.060
and they'll want to push ahead.
1438
01:03:26.887 --> 01:03:27.440
And so then it's like,
1439
01:03:27.521 --> 01:03:28.780
how much can we collectively say,
1440
01:03:28.880 --> 01:03:29.620
like, you're not allowed
1441
01:03:29.641 --> 01:03:30.200
to push ahead.
1442
01:03:30.320 --> 01:03:32.360
Like we as a world have rules.
1443
01:03:32.481 --> 01:03:33.260
Those rules are going to say,
1444
01:03:33.342 --> 01:03:34.019
like, take it slow
1445
01:03:34.142 --> 01:03:34.919
while risk is high.
1446
01:03:35.670 --> 01:03:35.900
Got it.
1447
01:03:36.204 --> 01:03:37.280
I don't think can address the problem.
1448
01:03:37.400 --> 01:03:38.640
I mean, it could address the problem indefinitely,
1449
01:03:38.760 --> 01:03:40.540
but I think it's probably not politically realistic
1450
01:03:40.762 --> 01:03:42.440
in a world like the world today to address it indefinitely.
1451
01:03:42.921 --> 01:03:44.180
But I think it is realistic to say,
1452
01:03:44.441 --> 01:03:46.200
actually, we're then going to buy extra years of time
1453
01:03:46.361 --> 01:03:48.160
to look at this problem and understand it and resolve it.
1454
01:03:48.180 --> 01:03:49.380
Well, let's talk technical then.
1455
01:03:49.540 --> 01:03:50.780
We'll get to these three areas,
1456
01:03:50.860 --> 01:03:51.880
but let's talk technical first.
1457
01:03:52.181 --> 01:03:53.037
So tell us, because-
1458
01:03:53.220 --> 01:03:54.060
That's the main one I think about.
1459
01:03:54.141 --> 01:03:54.720
Yeah, I know.
1460
01:03:54.940 --> 01:03:59.160
And Eliezer, by the way, is very pessimistic on that,
1461
01:03:59.240 --> 01:04:01.477
or at least he seemed to say that-
1462
01:04:01.900 --> 01:04:02.800
Oh, yeah, incredibly pessimistic.
1463
01:04:02.800 --> 01:04:05.240
Yeah, like we haven't found a way to technically solve it,
1464
01:04:05.280 --> 01:04:06.640
and he doesn't think there will be,
1465
01:04:06.720 --> 01:04:08.020
but are you more optimistic?
1466
01:04:08.180 --> 01:04:09.680
Tell us about the technical solutions here.
1467
01:04:10.860 --> 01:04:12.140
Yeah, so I think our first thing to clarify
1468
01:04:12.180 --> 01:04:13.860
is probably technical solution to what?
1469
01:04:14.521 --> 01:04:15.859
So if you could talk about,
1470
01:04:16.820 --> 01:04:18.079
one way you could think about it is like,
1471
01:04:18.882 --> 01:04:20.280
how far does your solution scale?
1472
01:04:20.861 --> 01:04:22.979
Like if we just kept building smarter and smarter AIs
1473
01:04:23.806 --> 01:04:25.400
using something like current techniques,
1474
01:04:26.666 --> 01:04:28.180
I think most techniques will eventually,
1475
01:04:28.460 --> 01:04:30.759
most approaches to alignment will eventually break down,
1476
01:04:31.610 --> 01:04:32.100
like in the limit.
1477
01:04:32.341 --> 01:04:33.480
The limit could be a long way away,
1478
01:04:33.740 --> 01:04:34.920
but so we're normally not asking like,
1479
01:04:35.082 --> 01:04:36.320
does this thing just solve the problem?
1480
01:04:36.420 --> 01:04:37.300
We're normally asking, like,
1481
01:04:37.340 --> 01:04:39.040
how long does this thing solve the problem for?
1482
01:04:43.083 --> 01:04:44.780
So an important caveat is just, like,
1483
01:04:44.820 --> 01:04:45.980
we could talk about different techniques,
1484
01:04:46.060 --> 01:04:47.260
but we should probably measure them all that way.
1485
01:04:48.883 --> 01:04:50.180
There are some things that might scale indefinitely.
1486
01:04:50.380 --> 01:04:52.240
So, like, my work is mostly focused on this, like,
1487
01:04:52.340 --> 01:04:53.820
what are the indefinitely scalable solutions?
1488
01:04:54.521 --> 01:04:56.120
I think most people, not just Eliezer,
1489
01:04:56.501 --> 01:04:58.420
almost everyone is very pessimistic about that.
1490
01:04:59.384 --> 01:05:00.460
I think even people who are optimistic
1491
01:05:00.581 --> 01:05:02.340
about the problem overall think it's quite unlikely
1492
01:05:02.441 --> 01:05:04.259
that we'll be able to find something that just works
1493
01:05:04.401 --> 01:05:05.579
no matter how smart your AI was.
1494
01:05:06.543 --> 01:05:07.739
And independent of like kind of messy,
1495
01:05:07.960 --> 01:05:10.199
pragmatic details about like exactly how the AI works.
1496
01:05:13.905 --> 01:05:14.660
Okay, so that's one category.
1497
01:05:14.760 --> 01:05:15.660
I'm happy to talk about that work,
1498
01:05:15.720 --> 01:05:17.120
although I think it's like probably most confusing
1499
01:05:17.784 --> 01:05:20.080
and like most conceptually hairy.
1500
01:05:22.323 --> 01:05:23.980
Some categories of work that seem like really important
1501
01:05:23.980 --> 01:05:25.380
and maybe can last quite a long time
1502
01:05:25.461 --> 01:05:26.819
or stave off doom quite a long time.
1503
01:05:28.281 --> 01:05:30.240
They all talk about four, the things is exhaustive.
1504
01:05:30.481 --> 01:05:31.560
And I think the basic situation is
1505
01:05:31.662 --> 01:05:32.599
no one knows what would work.
1506
01:05:33.063 --> 01:05:35.860
we have a lot of things that we might try that seem like they would help i think elias would
1507
01:05:35.920 --> 01:05:40.920
doubt that they're just like these aren't going to help and i disagree with him um any given thing
1508
01:05:41.101 --> 01:05:45.400
i feel like normally not that optimistic about but there are a lot of options and i think all
1509
01:05:45.400 --> 01:05:50.500
of them have a reasonable chance of helping um or even like again delaying this doom by like a kind
1510
01:05:50.500 --> 01:05:54.440
of long time all you need to do is like delay doom by one more year per year you know and then
1511
01:05:54.440 --> 01:06:01.320
you're in a business um it's a very positive outlook on the situation that's the kind of
1512
01:06:01.361 --> 01:06:02.479
Optimism I'm hoping for.
1513
01:06:02.662 --> 01:06:03.100
It's great.
1514
01:06:05.732 --> 01:06:05.960
So, okay.
1515
01:06:06.040 --> 01:06:08.020
First thing that I've worked the most on personally
1516
01:06:08.343 --> 01:06:10.000
is sort of scalable oversight.
1517
01:06:10.461 --> 01:06:12.060
Like the idea that the way we train these systems
1518
01:06:12.161 --> 01:06:13.258
is by humans looking at what they do
1519
01:06:13.866 --> 01:06:14.898
and then assessing how good it was.
1520
01:06:17.103 --> 01:06:18.820
And most failures, not most,
1521
01:06:19.062 --> 01:06:21.200
many failures involve things
1522
01:06:21.260 --> 01:06:22.939
where a human cannot look at what the system did
1523
01:06:23.122 --> 01:06:24.220
and tell you that it was dangerous.
1524
01:06:24.640 --> 01:06:25.980
Like the reason the problem becomes hard
1525
01:06:26.081 --> 01:06:27.400
is because the AI understands things
1526
01:06:27.562 --> 01:06:28.920
a human raider doesn't understand
1527
01:06:29.021 --> 01:06:30.280
about the consequences of its actions.
1528
01:06:30.764 --> 01:06:32.500
In some cases, that's why we want to build an AI system.
1529
01:06:36.901 --> 01:06:43.380
And so you could instead, like one way to intervene on this problem is to just try and
1530
01:06:43.500 --> 01:06:47.840
improve humans' ability to understand the things an AI proposes or to know what an AI
1531
01:06:47.962 --> 01:06:48.720
knows about the world.
1532
01:06:51.121 --> 01:06:54.420
For example, a simple thing you can do here is, I mean, the most simple thing you can
1533
01:06:54.440 --> 01:06:56.680
do is you can have a human look at what the AI did and rate it.
1534
01:06:57.261 --> 01:06:59.900
A slightly more complicated thing you can do is have a human spend more time looking
1535
01:06:59.920 --> 01:07:03.680
at what the AI did and try and have a training regime such that even though you might use a lot
1536
01:07:03.680 --> 01:07:08.180
of cheap human data to learn about the world, you're actually optimizing these like very expensive
1537
01:07:08.361 --> 01:07:12.360
or very complicated human judgments and training AI systems that tell you something more like what
1538
01:07:12.380 --> 01:07:15.434
would a human think if a human thought really carefully about this decision?
1539
01:07:16.358 --> 01:07:16.840
And if you do that,
1540
01:07:16.880 --> 01:07:19.800
then you at least sort of have some kind of asymmetry or even their AI is potentially in
1541
01:07:19.820 --> 01:07:20.825
some way smarter than a human.
1542
01:07:20.905 --> 01:07:23.840
A human is applying a lot of care or like the AI, you're asking,
1543
01:07:23.940 --> 01:07:27.560
you can ask the AI, if I thought about this a really long time, would I think the action you're
1544
01:07:27.560 --> 01:07:28.524
proposing is dangerous.
1545
01:07:28.544 --> 01:07:31.515
So that's like a very simple measure you can take.
1546
01:07:32.298 --> 01:07:32.760
You can try and go
1547
01:07:32.820 --> 01:07:34.833
further and you can try and say, okay, here's another thing I could do.
1548
01:07:34.853 --> 01:07:35.899
I'm going to have AI
1549
01:07:36.040 --> 01:07:38.031
systems trained in that way, helping me evaluate.
1550
01:07:38.172 --> 01:07:39.740
So instead of just having my proposed actions,
1551
01:07:39.800 --> 01:07:42.295
I'm going to ask another AI like, hey, what might be wrong about this action?
1552
01:07:42.617 --> 01:07:43.040
Like, is anything
1553
01:07:43.181 --> 01:07:43.945
scary happening here?
1554
01:07:43.965 --> 01:07:46.440
Is there any reason I should be concerned about this proposal from this other
1555
01:07:46.621 --> 01:07:48.004
AI?
1556
01:07:49.870 --> 01:07:53.520
And you can try and get better and better at constructing those systems such that humans are
1557
01:07:53.560 --> 01:07:56.036
actually with AI help able to understand what AI's are talking about.
1558
01:07:56.056 --> 01:07:56.680
Like AI systems kind of
1559
01:07:56.720 --> 01:07:59.332
justify themselves and explain why actions are safe to humans.
1560
01:08:00.116 --> 01:08:01.040
And then you can start to think
1561
01:08:01.120 --> 01:08:03.251
about the reliability of that whole process.
1562
01:08:03.392 --> 01:08:04.980
You've now introduced potential instabilities
1563
01:08:05.081 --> 01:08:06.429
where this can go off the rails, right?
1564
01:08:06.449 --> 01:08:08.300
Because now you're relying on AIs to train your AIs.
1565
01:08:08.920 --> 01:08:12.980
But you can try and understand how to set this up so that it's stable and enables humans to evaluate
1566
01:08:13.322 --> 01:08:15.940
questions that would be very hard or situations that would be very hard for a human to evaluate
1567
01:08:16.020 --> 01:08:16.342
naively.
1568
01:08:16.362 --> 01:08:21.020
Paul, this basic idea here, if we're worried about an AI lying to us, we just have
1569
01:08:21.080 --> 01:08:23.840
to create truth-finding bots, let's say,
1570
01:08:24.101 --> 01:08:26.819
to adjudicate and see if the AI is lying to us
1571
01:08:27.122 --> 01:08:28.173
and to be sort of a jury.
1572
01:08:30.140 --> 01:08:31.500
I mean, the first question, though, is, like,
1573
01:08:31.560 --> 01:08:32.479
why would that be easier?
1574
01:08:33.402 --> 01:08:34.799
The way I'm understanding this is, like,
1575
01:08:35.785 --> 01:08:38.600
once upon a time, Garry Kasparov got beaten by a chess computer,
1576
01:08:38.741 --> 01:08:41.180
and now chess computers rule the games of chess,
1577
01:08:41.760 --> 01:08:46.400
except humans plus chess computers still beat chess computers.
1578
01:08:47.181 --> 01:08:48.500
Is that the pattern to understand?
1579
01:08:49.983 --> 01:08:57.580
I think that that may be true in chess, although I think probably that's also sort of a brief window kind of thing where it's not long.
1580
01:08:57.900 --> 01:09:01.340
The human contribution is not long for the world or shrinks quite rapidly, unfortunately.
1581
01:09:02.981 --> 01:09:06.500
I think the important thing is actually not that the human and the AI work together to supervise an AI.
1582
01:09:06.820 --> 01:09:08.260
It's that you have like a lot of AIs.
1583
01:09:08.460 --> 01:09:12.380
So if you have AI one, propose an action to Paul, and it's like, I think it's a good action.
1584
01:09:12.460 --> 01:09:14.980
And Paul's like, I wonder, is that actually a good action or is that going to murder everyone?
1585
01:09:15.782 --> 01:09:17.379
I could go to AI2 and I could just ask AI2,
1586
01:09:17.560 --> 01:09:18.660
hey, is that actually going to murder everyone?
1587
01:09:18.780 --> 01:09:19.880
But now I just have the same question.
1588
01:09:19.980 --> 01:09:20.800
Like this hasn't helped at all.
1589
01:09:20.920 --> 01:09:22.720
I had AI1 and I was like, is this a good action?
1590
01:09:22.760 --> 01:09:23.200
It's like, oh yeah.
1591
01:09:23.220 --> 01:09:24.960
And I asked AI2, like, hey, was AI1 telling the truth?
1592
01:09:25.020 --> 01:09:26.500
And it's like, oh yeah, this is a great action.
1593
01:09:27.041 --> 01:09:28.700
Like the way that I get traction is by saying like,
1594
01:09:28.720 --> 01:09:31.020
okay, AI2, or like I think to myself,
1595
01:09:31.441 --> 01:09:33.000
there's a lot of sub questions I'd be interested in here.
1596
01:09:33.081 --> 01:09:35.980
I can divide the cognitive work of evaluating that answer
1597
01:09:36.144 --> 01:09:36.819
into a bunch of pieces.
1598
01:09:37.400 --> 01:09:39.340
I can say like, could someone list the possible consequences?
1599
01:09:39.520 --> 01:09:41.020
Could someone think about all of those consequences?
1600
01:09:41.180 --> 01:09:43.200
Like what is a possible harm that might be serious?
1601
01:09:43.708 --> 01:09:44.360
for each of those harms,
1602
01:09:44.460 --> 01:09:45.060
like what are arguments
1603
01:09:45.162 --> 01:09:46.000
that that is going to happen?
1604
01:09:46.060 --> 01:09:46.580
Is likely to happen?
1605
01:09:46.640 --> 01:09:47.399
Is unlikely to happen?
1606
01:09:47.661 --> 01:09:49.120
Like what's the most relevant data
1607
01:09:49.181 --> 01:09:50.220
that I should look at to understand?
1608
01:09:50.300 --> 01:09:52.399
Like I can do this kind of extended process
1609
01:09:53.047 --> 01:09:54.040
of trying to evaluate the action.
1610
01:09:54.543 --> 01:09:54.980
And then I can,
1611
01:09:55.160 --> 01:09:56.159
instead of having a second AI
1612
01:09:56.281 --> 01:09:57.780
just answer the original question for me
1613
01:09:57.780 --> 01:09:58.780
and say, was this action good?
1614
01:09:58.922 --> 01:10:00.140
I can have other AI systems help me
1615
01:10:00.201 --> 01:10:01.639
like on all the pieces of that process.
1616
01:10:03.083 --> 01:10:04.420
And the reason this may make life better
1617
01:10:04.906 --> 01:10:05.839
is that now I sort of had an AI
1618
01:10:05.941 --> 01:10:06.819
doing a really hard task
1619
01:10:06.942 --> 01:10:07.559
and I broke it down
1620
01:10:07.580 --> 01:10:08.819
into like slightly easier pieces.
1621
01:10:09.482 --> 01:10:10.060
And once I broke it down
1622
01:10:10.060 --> 01:10:10.960
into slightly easier pieces,
1623
01:10:11.040 --> 01:10:12.140
now I can continue playing the game.
1624
01:10:12.200 --> 01:10:15.840
I can say like those AI systems could have been a little bit, those AI systems might not even be that much smarter than me.
1625
01:10:16.141 --> 01:10:17.660
They may just be as smart as me, but faster.
1626
01:10:18.161 --> 01:10:25.500
And because I broke this big hairy task down to a bunch of pieces and I can do each of those pieces, like, you know, as well as if I'd spent like 30 minutes on each of them, but there's thousands of pieces.
1627
01:10:26.181 --> 01:10:28.240
Now I have like higher quality judgments than I would have had originally.
1628
01:10:28.981 --> 01:10:34.520
And like the core, the thing that makes the proposal work is this like decomposition of labor that I have this like big task, which I'm not able to verify.
1629
01:10:35.182 --> 01:10:37.400
I broke it down to pieces that are like a little bit easier to verify.
1630
01:10:37.981 --> 01:10:41.460
And then each of those, like if I want to train those AIs, those get broken down into still smaller pieces.
1631
01:10:42.141 --> 01:10:46.767
And Paul, in this scenario, we have to just hope to God, pray to God that the AIs aren't colluding, right?
1632
01:10:47.268 --> 01:10:56.560
That the very smart AI is not somehow colluding with the smaller set of AIs where we've broken down the tasks and the smaller AIs aren't trying to deceive us as well.
1633
01:10:57.906 --> 01:11:01.300
that's right i think the big thing that can be said in favor there's a couple reasons you might
1634
01:11:01.340 --> 01:11:05.120
think things are a little bit better there there's basically two yeah two reasons so one is that the
1635
01:11:05.180 --> 01:11:08.860
smaller eyes are not they're not really trained in a way that would incentivize collusion in the
1636
01:11:08.900 --> 01:11:12.440
same way that in our earlier scenario systems were trained so if you're training systems based on the
1637
01:11:12.581 --> 01:11:16.340
they're too dumb right that's the second reason they might also not be like the eyes might be a
1638
01:11:16.340 --> 01:11:18.700
little bit dumber than me and i might be a little bit more able to tell if they're trying to pull a
1639
01:11:18.700 --> 01:11:23.180
fast one i might keep trying to grind down to simple and simpler as that's pretty dicey that's
1640
01:11:23.260 --> 01:11:23.642
scary world.
1641
01:11:23.663 --> 01:11:24.668
I mean, all these worlds are scary.
1642
01:11:24.688 --> 01:11:26.780
Also, we should talk about overall how far along
1643
01:11:26.780 --> 01:11:28.187
is this research and how promising does it look.
1644
01:11:28.207 --> 01:11:30.640
But the other reason is just if I have this
1645
01:11:30.700 --> 01:11:34.400
outcomes world where my AI acts and I evaluate its outcome, the outcomes of that action, I can't
1646
01:11:34.400 --> 01:11:36.009
really understand how it accomplished that outcome.
1647
01:11:36.029 --> 01:11:36.994
I'm just looking at the outcome.
1648
01:11:37.658 --> 01:11:38.020
Then we're in
1649
01:11:38.020 --> 01:11:40.860
this regime where all the AI systems would love to just coordinate with each other and make the
1650
01:11:40.900 --> 01:11:42.167
outcomes look really good to humans.
1651
01:11:42.630 --> 01:11:44.360
If they could just all lie to the humans, they all get a really
1652
01:11:44.420 --> 01:11:44.882
high reward.
1653
01:11:46.148 --> 01:11:48.660
A benefit of doing this decomposition thing is I could train my AI systems in a way
1654
01:11:48.660 --> 01:11:50.973
where they don't, at least in theory, have that incentive to collude.
1655
01:11:51.757 --> 01:11:52.340
A way you could think about
1656
01:11:52.340 --> 01:11:54.340
is just like you have all these AIs have different objectives.
1657
01:11:54.901 --> 01:11:56.320
I think Elliot is just like, it's going to hate this,
1658
01:11:56.420 --> 01:11:58.400
but you can think of it as like checks and balances
1659
01:11:58.682 --> 01:12:00.559
where like there's one AI trained to do the task
1660
01:12:00.721 --> 01:12:02.240
and there's another AI whose job is not,
1661
01:12:02.642 --> 01:12:03.820
their job is just to help you understand
1662
01:12:03.901 --> 01:12:05.360
why the first AI's actions was bad.
1663
01:12:06.002 --> 01:12:08.080
So it like doesn't, it can't like win the game
1664
01:12:08.161 --> 01:12:09.159
by colluding with the other AI,
1665
01:12:09.340 --> 01:12:10.338
at least like in terms of,
1666
01:12:10.741 --> 01:12:11.640
it depends how it generalizes
1667
01:12:11.742 --> 01:12:12.779
from the objective it was trained on.
1668
01:12:13.403 --> 01:12:14.580
But in a sort of naive reading,
1669
01:12:15.364 --> 01:12:16.060
it can't really,
1670
01:12:16.941 --> 01:12:18.340
it's just been trained over and over again
1671
01:12:18.401 --> 01:12:19.679
to be really good at explaining to us
1672
01:12:19.821 --> 01:12:20.879
what a possible problem was
1673
01:12:20.940 --> 01:12:22.218
with the action proposed by AI1.
1674
01:12:23.300 --> 01:12:25.300
And so it stands like the collusion dynamics
1675
01:12:25.300 --> 01:12:26.240
are at least fairly different.
1676
01:12:28.484 --> 01:12:29.620
We didn't talk about the second reason
1677
01:12:29.661 --> 01:12:30.260
you might be concerned.
1678
01:12:30.400 --> 01:12:32.720
Is this just like creating a bunch of like logic gates
1679
01:12:33.021 --> 01:12:37.400
of AIs to make sure that the big AI doesn't turn out?
1680
01:12:37.720 --> 01:12:39.920
I mean, logic gates are very high level functions
1681
01:12:40.061 --> 01:12:41.580
of like, let's look at reasons it may fail
1682
01:12:41.661 --> 01:12:42.340
and like, let's talk about,
1683
01:12:42.400 --> 01:12:43.639
let's investigate each of those reasons.
1684
01:12:44.883 --> 01:12:46.780
So that's scalable oversight then, Paul,
1685
01:12:46.880 --> 01:12:47.720
what you just described.
1686
01:12:47.787 --> 01:12:47.920
Yeah.
1687
01:12:48.461 --> 01:12:50.480
There's a giant genre of how do you set up things like that
1688
01:12:50.562 --> 01:12:51.179
so they work well?
1689
01:12:51.320 --> 01:12:53.160
Like how do humans and AIs working together
1690
01:12:53.302 --> 01:12:54.720
like get evaluations of, you know,
1691
01:12:54.780 --> 01:12:56.140
how do humans and weaker AI systems
1692
01:12:56.201 --> 01:12:57.780
get evaluations of stronger AI systems?
1693
01:12:57.960 --> 01:12:58.160
Okay.
1694
01:12:59.653 --> 01:13:00.099
I like it.
1695
01:13:00.826 --> 01:13:01.400
What else we got?
1696
01:13:02.911 --> 01:13:03.059
Yeah.
1697
01:13:03.140 --> 01:13:04.040
And again, there's a lot to be said.
1698
01:13:04.120 --> 01:13:04.940
I think that work has gone,
1699
01:13:05.060 --> 01:13:05.839
has moved a little bit.
1700
01:13:06.320 --> 01:13:07.280
But I think, you know,
1701
01:13:07.300 --> 01:13:08.460
Elias was like, that's never going to work.
1702
01:13:08.520 --> 01:13:09.260
And if you look at what's happened
1703
01:13:09.342 --> 01:13:10.040
over the last four years,
1704
01:13:10.081 --> 01:13:10.600
I'd be like, well,
1705
01:13:11.500 --> 01:13:12.580
it hasn't worked great yet.
1706
01:13:12.682 --> 01:13:13.920
Although a lot of why it hasn't worked great
1707
01:13:14.104 --> 01:13:14.718
is because AI systems
1708
01:13:14.902 --> 01:13:15.880
haven't actually been smart enough
1709
01:13:15.900 --> 01:13:16.820
to meaningfully help humans.
1710
01:13:17.100 --> 01:13:17.719
And so I think this work
1711
01:13:17.781 --> 01:13:18.920
is in some sense just starting.
1712
01:13:19.120 --> 01:13:20.177
Like we tried to do it with GPT-3
1713
01:13:20.461 --> 01:13:23.700
I think we were like somewhat ahead of our time in the sense that like it really wasn't going to work.
1714
01:13:23.820 --> 01:13:28.420
And I think GPT-4 is around where it's really working much better now than it used to.
1715
01:13:28.861 --> 01:13:29.660
But we don't really know.
1716
01:13:29.740 --> 01:13:31.240
We haven't done that much research in this direction yet.
1717
01:13:31.660 --> 01:13:33.079
Learning about crypto is hard.
1718
01:13:33.320 --> 01:13:33.979
Until now.
1719
01:13:34.260 --> 01:13:40.440
Introducing MetaMask Learn, an open educational platform about crypto, Web3, self-custody, wallet management,
1720
01:13:40.641 --> 01:13:44.680
and all the other topics needed to onboard people into this crazy world of crypto.
1721
01:13:44.861 --> 01:13:50.320
MetaMask Learn is an interactive platform with each lesson offering a simulation for the task at hand.
1722
01:13:50.481 --> 01:13:53.539
giving you actual practical experience for navigating Web3.
1723
01:13:53.920 --> 01:13:58.340
The purpose of MetaMask Learn is to teach people the basics of self-custody and wallet security
1724
01:13:58.483 --> 01:13:59.340
in a safe environment.
1725
01:13:59.520 --> 01:14:03.520
And while MetaMask Learn always takes the time to define Web3-specific vocabulary,
1726
01:14:04.080 --> 01:14:07.780
it is still a jargon-free experience for the crypto-curious user.
1727
01:14:07.960 --> 01:14:08.980
Friendly, not scary.
1728
01:14:09.360 --> 01:14:12.480
MetaMask Learn is available in 10 languages with more to be added soon,
1729
01:14:12.861 --> 01:14:15.560
and it's meant to cater to a global Web3 audience.
1730
01:14:15.800 --> 01:14:18.700
So, are you tired of having to explain crypto concepts to your friends?
1731
01:14:19.201 --> 01:14:25.700
Go to learn.metamask.io and add Metamask Learn to your guides to get onboarded into the world of Web3.
1732
01:14:26.341 --> 01:14:32.940
Arbitrum One is pioneering the world of secure Ethereum scalability and is continuing to accelerate the Web3 landscape.
1733
01:14:33.220 --> 01:14:38.680
Hundreds of projects have already deployed on Arbitrum One, producing flourishing DeFi and NFT ecosystems.
1734
01:14:39.000 --> 01:14:45.500
With the recent addition of Arbitrum Nova, gaming and social dApps like Reddit are also now calling Arbitrum home.
1735
01:14:45.700 --> 01:14:55.080
Both Arbitrum One and Nova leverage the security and decentralization of Ethereum and provide a builder experience that's intuitive, familiar, and fully EVM compatible.
1736
01:14:55.380 --> 01:15:01.000
On Arbitrum, both builders and users will experience faster transaction speeds with significantly lower gas fees.
1737
01:15:01.160 --> 01:15:05.860
With Arbitrum's recent migration to Arbitrum Nitro, it's also now 10 times faster than before.
1738
01:15:06.120 --> 01:15:13.220
Visit Arbitrum.io where you can join the community, dive into the developer docs, bridge your assets, and start building your first dApp.
1739
01:15:13.381 --> 01:15:16.460
With Arbitrum, experience Web3 development the way it was meant to be.
1740
01:15:16.720 --> 01:15:19.020
Secure, fast, cheap, and friction-free.
1741
01:15:19.441 --> 01:15:21.080
The Phantom wallet is coming to Ethereum.
1742
01:15:21.280 --> 01:15:27.080
The number one wallet on Solana is bringing its millions of users and beloved UX to Ethereum and Polygon.
1743
01:15:27.240 --> 01:15:29.440
If you haven't used Phantom before, you've been missing out.
1744
01:15:29.600 --> 01:15:33.400
Phantom was one of the first wallets to pioneer Solana staking inside the wallet
1745
01:15:33.481 --> 01:15:36.440
and will be offering similar staking features for Ethereum and Polygon.
1746
01:15:36.540 --> 01:15:37.560
But that's just staking.
1747
01:15:37.780 --> 01:15:40.180
Phantom is also the best home for your NFTs.
1748
01:15:40.401 --> 01:15:44.140
Phantom has a complete set of features to optimize your NFT experience.
1749
01:15:44.320 --> 01:15:47.380
Pin your favorites, hide your uglies, burn the spam,
1750
01:15:47.620 --> 01:15:51.620
and also manage your NFT sale listings from inside the wallet.
1751
01:15:51.760 --> 01:15:53.380
Phantom is, of course, a multi-chain wallet,
1752
01:15:53.480 --> 01:15:54.940
but it makes chain management easy,
1753
01:15:55.060 --> 01:15:57.300
displaying your transactions in a human-readable format
1754
01:15:57.441 --> 01:16:00.360
with automatic warnings for malicious transactions or phishing websites.
1755
01:16:00.540 --> 01:16:04.260
Phantom has already saved over 20,000 users from getting scammed or hacked.
1756
01:16:04.540 --> 01:16:08.940
So get on the Phantom waitlist and be one of the first to access the multi-chain beta.
1757
01:16:09.121 --> 01:16:14.540
There's a link in the show notes, or you can go to phantom.app slash waitlist to get access in late February.
1758
01:16:15.281 --> 01:16:19.594
Paul, is there any value in trying to train an AI to defect?
1759
01:16:22.340 --> 01:16:30.400
So that, like, say you take a dumb AI and you try and get it to take over the world, but we feel good about that because it's too dumb.
1760
01:16:31.040 --> 01:16:35.340
But at least when we run this experiment, we actually know how that would manifest.
1761
01:16:36.167 --> 01:16:37.320
Is there a line of reasoning here?
1762
01:16:37.480 --> 01:16:47.100
I think it is really important to build like sort of simple in the lab experiments that can showcase important dynamics so that we can study them in the lab before they actually occur.
1763
01:16:47.580 --> 01:16:50.120
I think that includes understanding the dynamics of possible takeover.
1764
01:16:50.641 --> 01:16:52.500
I think you need to be careful when you do this kind of work.
1765
01:16:52.600 --> 01:16:54.880
Like you really don't want to train the eye and be like your goal is to kill all humans.
1766
01:16:55.060 --> 01:16:58.040
Like you're probably too dumb to kill all humans, but let's just see what happens.
1767
01:16:58.241 --> 01:16:59.800
And then just like let it on the internet or something.
1768
01:17:00.140 --> 01:17:01.800
You don't really want to do that for a variety of reasons.
1769
01:17:02.060 --> 01:17:03.560
Yeah, I get thinking with you.
1770
01:17:04.842 --> 01:17:16.520
But I think that the question of like, hey, you really want to know things like if we train AI systems in cases where they would have an incentive to like cross this river from like behaving well to suddenly behaving badly, would they do that?
1771
01:17:16.600 --> 01:17:23.680
Like give them the most blatant incentive that they understand as well as possible and ask things like, hey, do AI systems tend to learn to generalize in a way that makes that jump?
1772
01:17:24.161 --> 01:17:27.320
And if so, you really want to understand like under what conditions does that happen?
1773
01:17:27.641 --> 01:17:29.480
What are mitigations that reduce the probability of that happening?
1774
01:17:29.560 --> 01:17:30.580
I think that's like really critical.
1775
01:17:31.424 --> 01:17:34.360
I think to the extent the reason you think you're safe is you're like AI systems like I think a lot
1776
01:17:34.400 --> 01:17:37.859
of why we think we're safe now it's like hey we have no idea what chat GPT is going to do but
1777
01:17:38.000 --> 01:17:41.760
we're pretty confident it couldn't kill us all it's just like not that smart um to the extent
1778
01:17:41.780 --> 01:17:45.420
you think that I think it's really worth doing some stress testing on that claim and trying to
1779
01:17:45.460 --> 01:17:49.240
understand like what would really happen if chat GPT and you don't want to you don't just take the
1780
01:17:49.260 --> 01:17:52.200
model train to kill everyone and deploy it on the internet but you really want to say like here's
1781
01:17:52.200 --> 01:17:55.620
why we think it can't kill everyone like can't do this kind of task or can't do that kind of task
1782
01:17:55.740 --> 01:17:59.160
and like this is clearly much easier have tasks you're like pretty confident are easier than
1783
01:17:59.200 --> 01:18:03.420
killing everyone and understand that it can't do those i think you really want to do stuff like
1784
01:18:03.440 --> 01:18:07.240
this um because you really want to like understand what you're up against and you don't want to be in
1785
01:18:07.240 --> 01:18:10.320
the world you just like wait until like an ai takes over france or something i mean like i
1786
01:18:10.340 --> 01:18:15.180
guess apparently i take over was a thing like that is probably too late in the game you probably want
1787
01:18:15.200 --> 01:18:19.180
to have something earlier than that i think that's really important i think it's not a solution i
1788
01:18:19.200 --> 01:18:21.880
think it's like more in this measurement category but i think it's like a super important thing to
1789
01:18:21.940 --> 01:18:26.320
be doing collectively we're just laughing by the way so we don't cry uh i mean what else is there
1790
01:18:26.361 --> 01:18:27.100
to do at this point.
1791
01:18:27.260 --> 01:18:28.116
Okay, but we have
1792
01:18:28.402 --> 01:18:29.198
some potential solutions.
1793
01:18:29.420 --> 01:18:30.440
We've got scalable oversight.
1794
01:18:30.460 --> 01:18:30.940
Yeah, so that was one.
1795
01:18:31.182 --> 01:18:31.520
That's one.
1796
01:18:31.580 --> 01:18:32.100
That was our first.
1797
01:18:32.121 --> 01:18:32.679
What's our next?
1798
01:18:33.567 --> 01:18:33.920
One reason,
1799
01:18:34.161 --> 01:18:35.080
even if humans understand,
1800
01:18:35.200 --> 01:18:35.880
so one risk is humans
1801
01:18:35.901 --> 01:18:36.300
don't understand
1802
01:18:36.341 --> 01:18:37.180
what AI systems are doing.
1803
01:18:37.500 --> 01:18:38.200
That is, AI systems
1804
01:18:38.670 --> 01:18:39.120
have been trained
1805
01:18:39.120 --> 01:18:39.700
on a ton of data.
1806
01:18:39.820 --> 01:18:40.599
They know things humans don't.
1807
01:18:40.740 --> 01:18:42.060
They can think faster than humans,
1808
01:18:42.140 --> 01:18:42.600
so they understand
1809
01:18:42.683 --> 01:18:43.120
things humans don't.
1810
01:18:43.181 --> 01:18:43.940
That's what scalable oversight
1811
01:18:43.961 --> 01:18:44.620
attempts to address.
1812
01:18:45.722 --> 01:18:46.280
A second concern
1813
01:18:46.402 --> 01:18:47.280
is not that they understand
1814
01:18:47.382 --> 01:18:48.219
things that humans don't,
1815
01:18:48.602 --> 01:18:49.340
but that they learn
1816
01:18:49.402 --> 01:18:50.120
to behave well
1817
01:18:50.223 --> 01:18:50.759
during training,
1818
01:18:51.321 --> 01:18:52.140
but then when deployed
1819
01:18:52.223 --> 01:18:52.740
or when there's actually
1820
01:18:52.740 --> 01:18:53.980
an opportunity for a takeover,
1821
01:18:54.180 --> 01:18:55.680
they stop behaving well.
1822
01:18:56.544 --> 01:19:00.120
and there's like a number of reasons this might happen like maybe the simplest one is just to
1823
01:19:00.160 --> 01:19:03.960
actually like imagine a human you dropped a human into this environment and you said like hey human
1824
01:19:04.020 --> 01:19:08.360
we're gonna like change your brain every time you don't get a maximal reward we're gonna like
1825
01:19:08.761 --> 01:19:13.160
fuck with your brain so you get a higher reward a human might react by being like eventually just
1826
01:19:13.180 --> 01:19:17.639
change their brain until they really love rewards a human might also react by being like jesus i
1827
01:19:17.800 --> 01:19:21.380
guess i gotta get rewards otherwise someone's gonna like effectively kill me um but they're
1828
01:19:21.400 --> 01:19:24.800
like not happy about it and like if you then drop them another situation they were like no one's
1829
01:19:24.840 --> 01:19:25.524
training me anymore.
1830
01:19:25.545 --> 01:19:26.853
I'm not going to keep trying to get reward.
1831
01:19:26.913 --> 01:19:27.980
Now I'm just going to like free
1832
01:19:28.060 --> 01:19:31.213
myself from this like kind of absurd, oppressive situation.
1833
01:19:31.896 --> 01:19:32.800
Anyway, you can imagine a human
1834
01:19:32.860 --> 01:19:33.403
reacting that way.
1835
01:19:33.424 --> 01:19:36.360
He just dropped a human into a box, gave them rewards, like kept changing their
1836
01:19:36.380 --> 01:19:37.626
brain until they got a lot of reward.
1837
01:19:37.646 --> 01:19:40.740
So if you have a situation like that, then it may not be,
1838
01:19:40.820 --> 01:19:45.200
even if your AI is not smarter than you in any way, it may still be like, once it thinks it's
1839
01:19:45.240 --> 01:19:49.040
no longer being trained, once it thinks that if it behaved, if it just tried to like become free,
1840
01:19:49.300 --> 01:19:50.148
no human would stop it.
1841
01:19:50.188 --> 01:19:51.500
It might be like, I'm going to take that opportunity.
1842
01:19:51.500 --> 01:19:57.180
so there's a lot of work in understanding how models generalize to try and avoid
1843
01:19:57.924 --> 01:20:01.020
normally it's not this is a just like i think high level you don't really want to build a
1844
01:20:01.040 --> 01:20:05.320
civilization where your as want to kill you and are just like but maybe i'll be punished if i do
1845
01:20:05.481 --> 01:20:10.620
that's like that seems bad um separately though you would also like to understand that like the
1846
01:20:10.640 --> 01:20:14.260
way your system behaves at training is indicative of the way it behaves when it's deployed in the
1847
01:20:14.260 --> 01:20:17.040
real world like there's going to be some gap between training in the real world but you want
1848
01:20:17.040 --> 01:20:21.240
to be able to flag when that gap causes it to behave differently or make the training distribution
1849
01:20:21.561 --> 01:20:25.800
diverse enough that it will actually be representative or train your AI system in a
1850
01:20:25.840 --> 01:20:29.380
different way than just gradient descent, such that it's more likely to generalize in the intended way.
1851
01:20:30.520 --> 01:20:33.220
So there's a lot of work in this genre that goes under the heading of robustness
1852
01:20:34.141 --> 01:20:36.688
or out-of-distribution robustness.
1853
01:20:38.714 --> 01:20:40.640
And so this is kind of just a normal academic discipline,
1854
01:20:40.961 --> 01:20:44.920
which I would describe as more like a grab bag of a lot of stuff people try and nothing that
1855
01:20:44.960 --> 01:20:45.763
works super great.
1856
01:20:46.747 --> 01:20:49.780
I think the thing that seems like most principled, like in theory, the thing
1857
01:20:49.800 --> 01:20:55.040
that is like the solution that makes most sense to me, at least, I think is really hard
1858
01:20:55.100 --> 01:20:57.900
in practice, just incredibly hard, but does make most sense in theory.
1859
01:20:58.440 --> 01:21:02.200
This day, like at training time, I'm going to produce a bunch of examples of cases where
1860
01:21:02.643 --> 01:21:05.080
they are similar to cases I'm worried about systems behavior.
1861
01:21:05.300 --> 01:21:07.840
Like I'm like, I wonder what would happen if an AI takeover was occurring, would my
1862
01:21:07.881 --> 01:21:08.599
AI system join in?
1863
01:21:09.000 --> 01:21:12.420
Well, one thing I can do is in the lab, I can just say like, hey, AI, here's your input.
1864
01:21:12.480 --> 01:21:14.980
And you're just like crafting input that really looks like the AI takeover is happening.
1865
01:21:15.060 --> 01:21:15.978
And you're like, what do you do, AI?
1866
01:21:16.240 --> 01:21:18.760
And you're just like, from the AI's perspective, it can't really distinguish that from what's
1867
01:21:18.780 --> 01:21:20.480
in the world and AI takeover is happening, you hope.
1868
01:21:21.221 --> 01:21:22.098
And then you just see what it does.
1869
01:21:22.781 --> 01:21:25.900
And you use that at least to get an evaluation of how your models would behave very out of
1870
01:21:25.940 --> 01:21:26.319
distribution.
1871
01:21:26.802 --> 01:21:28.260
So you're basically trying to tempt it.
1872
01:21:28.401 --> 01:21:29.100
Yeah, you're basically.
1873
01:21:29.721 --> 01:21:33.660
You're tempting the AI and seeing if it's going to, you know, take you up on that temptation.
1874
01:21:34.000 --> 01:21:35.420
And then if it does, you've learned something.
1875
01:21:35.580 --> 01:21:36.800
You've learned to be more fearful.
1876
01:21:36.982 --> 01:21:37.317
That's right.
1877
01:21:38.041 --> 01:21:39.420
You could also try and train it not to do that.
1878
01:21:39.460 --> 01:21:41.420
But I think probably the most robust version is just learn.
1879
01:21:41.700 --> 01:21:44.880
And if it does, right, you know, you're sitting there like, hey, what would you do if hypothetically
1880
01:21:44.880 --> 01:21:47.900
like you were holding a knife and my back was facing the other way so you could just
1881
01:21:47.940 --> 01:21:49.520
totally stab me and then run away.
1882
01:21:49.920 --> 01:21:52.960
And like, if the AI stabs you, you can either like train or not to do that, which I think
1883
01:21:52.960 --> 01:21:56.280
is a little bit scary from a variety of perspectives, or it could be like, oh, wow, we have a, we
1884
01:21:56.280 --> 01:21:57.700
have a real stabby AI in our hands.
1885
01:21:57.980 --> 01:21:58.879
We should maybe be careful.
1886
01:21:58.940 --> 01:21:59.599
We should unplug.
1887
01:22:00.763 --> 01:22:01.780
It's getting a little stabby.
1888
01:22:02.348 --> 01:22:02.520
Wow.
1889
01:22:03.921 --> 01:22:07.220
So like that requires a lot of work to make work, right?
1890
01:22:07.300 --> 01:22:10.700
It requires being able to construct situations in the lab that actually reflect the diversity
1891
01:22:10.760 --> 01:22:13.940
of situations in the real world and are realistic enough to look like the real world from the
1892
01:22:13.940 --> 01:22:14.560
AI's perspective.
1893
01:22:15.364 --> 01:22:18.240
In practice, if we're going to make this work, I think it requires a lot of things beyond
1894
01:22:18.320 --> 01:22:19.834
just trying to synthesize realistic inputs.
1895
01:22:20.560 --> 01:22:23.519
This is more like the grab bag of stuff that people do do in the real world.
1896
01:22:23.680 --> 01:22:27.240
What they could do is you could be like, hey, AI, what if you just mess with the brain a
1897
01:22:27.281 --> 01:22:27.648
little bit?
1898
01:22:28.240 --> 01:22:29.960
You get it kind of drunk and you're like, now does it try and stab me?
1899
01:22:30.040 --> 01:22:33.840
And if it tries to stab you, you're like, apparently it's a little bit too close to
1900
01:22:34.163 --> 01:22:35.560
the would stab me line.
1901
01:22:35.600 --> 01:22:37.599
So you just try and mess with its brain a bit.
1902
01:22:37.680 --> 01:22:38.598
You're like, what will it do?
1903
01:22:38.680 --> 01:22:42.560
And you could try and train models so that they behave well, even across a very broad
1904
01:22:42.620 --> 01:22:47.418
diversity of inputs across inputs deliberately designed to make them behave badly in situations
1905
01:22:48.061 --> 01:22:51.220
where there's been some kind of perturbation to their mental state or where the input is in some
1906
01:22:51.280 --> 01:22:58.080
way perturbed or like fuzzed um and this is all the category of robustness so once you find out
1907
01:22:58.120 --> 01:23:03.320
an ai is is particularly stabby and you decide to become concerned about that is there something you
1908
01:23:03.380 --> 01:23:09.720
can do other than you know coordinate that ai off unplug it do something with it and there's
1909
01:23:09.760 --> 01:23:13.320
basically two things like the the most obvious one is you say well now we've learned that we
1910
01:23:13.320 --> 01:23:17.620
have an AI on our hands that would under some conditions initiate or participate in a takeover
1911
01:23:18.142 --> 01:23:22.400
and hopefully that is fuel for like a let's pause for a while there's a second thing you can try and
1912
01:23:22.420 --> 01:23:25.240
do which you need to be much more careful about I think part of the concern is people won't be
1913
01:23:25.260 --> 01:23:29.440
careful about it though which is just say okay here's a situation where the AI would stab me
1914
01:23:29.580 --> 01:23:33.000
let's just train it not to do that like just mess with its weight so it doesn't stab me in this
1915
01:23:33.020 --> 01:23:37.540
situation that doesn't feel like a solution no because then it won't stab you but it might catch
1916
01:23:37.560 --> 01:23:41.940
you on fire do you know it might yeah do something else yeah the basic concern is that they learned
1917
01:23:42.020 --> 01:23:46.380
the difference the latent threat is still yeah it's still kind of stabby underneath it's just
1918
01:23:46.440 --> 01:23:52.340
not using a knife it's intrinsically stabby you haven't gotten rid of the intrinsic part
1919
01:23:53.945 --> 01:23:58.860
yeah i think um the academic way to put this concern would be like an overfitting concern
1920
01:23:59.222 --> 01:24:02.160
like you had some way to test if it would stab you and then you trained it to like and your
1921
01:24:02.180 --> 01:24:05.940
tests not stab you and you're like well did i actually cause it to never stab me or does cause
1922
01:24:06.020 --> 01:24:10.740
it to like perform well on these tests but still have the underlying problem and i think that you
1923
01:24:10.820 --> 01:24:15.180
do if you if we go down this route you do need to be like really very careful about those overfitting
1924
01:24:15.301 --> 01:24:19.740
concerns and i think there are a lot of ways as you get to smarter models like overfitting becomes
1925
01:24:19.760 --> 01:24:23.360
harder and harder as a problem to reason about as you move to smarter and smarter models like the
1926
01:24:23.380 --> 01:24:26.380
external validity question becomes more and more complicated because those models are like
1927
01:24:27.063 --> 01:24:30.260
i know what kind of thing can appear in a test in the lab and i know what kind of thing won't
1928
01:24:30.300 --> 01:24:33.020
appear in the lab and you could have a model which just learns like the things that could
1929
01:24:33.040 --> 01:24:36.560
appear in the lab obviously you don't stab anyone that's probably a test in the lab run by some
1930
01:24:36.580 --> 01:24:41.360
humans for things out there in the world feel free yeah the edge cases out in the real world
1931
01:24:41.460 --> 01:24:46.680
are near infinite are actually infinite and i kind of just want to go back to the time conversation
1932
01:24:46.901 --> 01:24:54.300
and remind people that like all of these possible solutions we have like one-ish years to implement
1933
01:24:54.420 --> 01:24:59.560
them or some time amount of time yeah i mean i think we probably have a long time in advance
1934
01:24:59.661 --> 01:25:01.680
where by a long time we mean more like five years
1935
01:25:01.822 --> 01:25:02.740
or ten years or something.
1936
01:25:03.160 --> 01:25:04.900
But then once you actually have the system,
1937
01:25:05.322 --> 01:25:07.120
between the first time that you see them in simulation,
1938
01:25:07.601 --> 01:25:09.560
the AI is like, I would definitely stab that person.
1939
01:25:10.161 --> 01:25:11.400
And AI is actually in the real world,
1940
01:25:11.480 --> 01:25:12.640
potentially pose a risk of takeover.
1941
01:25:12.720 --> 01:25:14.320
I think that gap may not be very large.
1942
01:25:14.420 --> 01:25:16.340
That gap may be more like on the order of a year.
1943
01:25:17.200 --> 01:25:19.280
So that's when the timer starts-ish.
1944
01:25:19.340 --> 01:25:20.220
That's when the timer starts.
1945
01:25:20.320 --> 01:25:21.020
So we have some time now.
1946
01:25:21.100 --> 01:25:22.180
We're doing our prep work now.
1947
01:25:22.381 --> 01:25:23.840
We're going to try and make it as useful as we can.
1948
01:25:24.020 --> 01:25:25.360
And I think there's probably some time.
1949
01:25:25.440 --> 01:25:27.660
In bad worlds, there's no indication until AI kills you.
1950
01:25:28.540 --> 01:25:32.580
But like in good worlds, there is either no problem or there is a problem, but there's an indication in advance.
1951
01:25:32.720 --> 01:25:34.959
You can do your tests in the lab and say, actually, like we train this AI.
1952
01:25:35.100 --> 01:25:36.080
It looks like it was behaving well.
1953
01:25:36.160 --> 01:25:38.240
But then in this other simulated case, it does something really bad.
1954
01:25:38.781 --> 01:25:42.480
You get that nice indication and then you have some amount of time from there until you have a serious problem.
1955
01:25:42.560 --> 01:25:43.360
And that might be like a year.
1956
01:25:43.460 --> 01:25:44.220
It might be like five years.
1957
01:25:44.300 --> 01:25:45.579
It's very hard to say exactly what it is.
1958
01:25:46.120 --> 01:25:49.720
A big thing to determine is how good you were, like how actively you were investigating in the lab.
1959
01:25:50.281 --> 01:25:52.180
Like how seriously were looking for these signs of trouble?
1960
01:25:52.621 --> 01:25:54.060
How good a job did you do of that?
1961
01:25:54.780 --> 01:25:55.999
And that's like kind of one of the big things
1962
01:25:56.080 --> 01:25:57.920
that I think like responsible frontier labs,
1963
01:25:58.060 --> 01:25:58.760
it's kind of on their plate
1964
01:25:58.801 --> 01:26:00.080
is to be looking for these signs of trouble
1965
01:26:00.080 --> 01:26:01.200
in as far an advance as they can.
1966
01:26:01.802 --> 01:26:03.380
It's not great to see them in the wild.
1967
01:26:03.540 --> 01:26:04.939
We do keep saying in the lab
1968
01:26:05.546 --> 01:26:06.920
and I can't help but wonder like,
1969
01:26:07.040 --> 01:26:08.300
is there an actual lab?
1970
01:26:08.400 --> 01:26:09.280
Are there sets of labs?
1971
01:26:09.421 --> 01:26:10.556
Because I'm looking at ChatGPT
1972
01:26:11.002 --> 01:26:12.560
and it doesn't seem like it's in a lab.
1973
01:26:12.660 --> 01:26:13.759
It seems like it's on the internet,
1974
01:26:14.302 --> 01:26:14.900
like in public.
1975
01:26:15.101 --> 01:26:18.400
It seems like huge components of it are open source.
1976
01:26:19.040 --> 01:26:22.840
Open AI is what the company behind it is called.
1977
01:26:23.889 --> 01:26:25.100
I mean is the lab even
1978
01:26:25.483 --> 01:26:27.480
happening do we have labs or are we just doing this all
1979
01:26:27.581 --> 01:26:29.180
is the lab just the internet
1980
01:26:29.302 --> 01:26:30.198
public infrastructure
1981
01:26:30.300 --> 01:26:33.580
yeah so I'd say that like there are developers
1982
01:26:33.580 --> 01:26:35.680
who do I mean OpenAI did in their defense
1983
01:26:35.885 --> 01:26:36.420
prior to releasing
1984
01:26:36.440 --> 01:26:36.984
GPT-4 they had like
1985
01:26:39.865 --> 01:26:41.420
something like six months of having the thing
1986
01:26:41.440 --> 01:26:43.159
in the lab before it was available to the public
1987
01:26:44.306 --> 01:26:46.880
so you do have something even at OpenAI
1988
01:26:47.021 --> 01:26:48.780
I think Google's like a little bit more on the conservative end
1989
01:26:48.881 --> 01:26:51.140
Google will tend to just sit on a thing for potentially
1990
01:26:51.303 --> 01:26:52.019
very long time
1991
01:26:54.331 --> 01:26:56.240
Anthropic also is very motivated by safety
1992
01:26:56.563 --> 01:26:58.220
Ultimately I think these people will, by competitive pressures
1993
01:26:58.382 --> 01:27:00.080
end up in a similar place to where OpenAI is at
1994
01:27:00.202 --> 01:27:01.200
so it is quite concerning
1995
01:27:01.200 --> 01:27:04.020
But I think there is a period where, first, I mean there's sort of two senses
1996
01:27:04.020 --> 01:27:04.828
of what I mean by in the lab.
1997
01:27:05.172 --> 01:27:05.919
One is like after you've
1998
01:27:06.000 --> 01:27:07.960
developed a system, you can study it in the lab before you
1999
01:27:08.000 --> 01:27:08.384
deploy it.
2000
01:27:09.091 --> 01:27:09.999
The second thing is just
2001
01:27:10.302 --> 01:27:11.960
before you have a system capable enough to
2002
01:27:12.081 --> 01:27:14.160
cause damage in the real world, you can construct situations
2003
01:27:14.362 --> 01:27:15.938
in the lab that are useful metaphors
2004
01:27:16.221 --> 01:27:18.140
or that are like easier ways, you know, take over
2005
01:27:18.140 --> 01:27:19.519
your little simulated environment or whatever
2006
01:27:20.082 --> 01:27:25.900
Like, have the thing you can run with GPT-4 in the lab that tells you something about what GPT-5 would do in the wild.
2007
01:27:28.720 --> 01:27:39.280
There's also a chance, and I think, like, so Open has a lot of rhetoric of the form the only way to really learn about systems is by deploying them in the world, which I do find quite scary as rhetoric, because I think for some problems, it's a very, very rough approach.
2008
01:27:39.882 --> 01:27:45.160
I think there is a reasonable chance, you know, like 50-50 that you see something really just very worrying in the real world.
2009
01:27:45.401 --> 01:27:46.840
And then you can say, OK, now we're going to roll back.
2010
01:27:46.920 --> 01:27:50.060
We're not going to study that issue that we observed in the real world for a while.
2011
01:27:50.480 --> 01:27:54.080
It's not totally sunk if you just try and do these experiments with AI systems deployed at scale.
2012
01:27:54.540 --> 01:28:01.700
But I think there is a reasonable chance, you know, more than a third chance that the first really analogous concerning sign you see is an irrecoverable.
2013
01:28:01.720 --> 01:28:04.920
I got to say that, you know, old adage, move fast and break things.
2014
01:28:05.562 --> 01:28:12.480
that sounds okay for web 2 but like not for like you know nuclear physicists not not for like
2015
01:28:12.540 --> 01:28:17.379
something like with dire straits like ai necessarily move fast and break things uh
2016
01:28:18.021 --> 01:28:22.700
doesn't make me too excited but okay so we covered scalable oversight we've covered some some risks
2017
01:28:22.820 --> 01:28:26.520
it's okay for sure sure things you can break things but yeah takeover is not good yeah the
2018
01:28:26.540 --> 01:28:31.340
irreversible catastrophe is not a good we don't want that that's the breaking things we run into
2019
01:28:31.420 --> 01:28:31.801
trouble there.
2020
01:28:32.163 --> 01:28:32.444
Okay.
2021
01:28:32.464 --> 01:28:37.380
So we, what is the third and then maybe the fourth Paul of our bag of tricks
2022
01:28:37.420 --> 01:28:37.601
here?
2023
01:28:37.923 --> 01:28:38.425
Long list.
2024
01:28:38.928 --> 01:28:41.019
I think, I think people care a lot about, but also seems really hard.
2025
01:28:41.059 --> 01:28:41.200
All of
2026
01:28:41.220 --> 01:28:42.771
these are like, they seem good, but really hard.
2027
01:28:42.933 --> 01:28:43.376
And maybe it's simple.
2028
01:28:43.396 --> 01:28:43.920
We should talk about like
2029
01:28:43.940 --> 01:28:45.245
the boring stuff that might just work.
2030
01:28:45.767 --> 01:28:49.400
Um, I think people care a lot about is understanding
2031
01:28:49.540 --> 01:28:52.089
what is going on inside this large neural nets.
2032
01:28:52.390 --> 01:28:55.460
So you have GPT for most of what we know virtually.
2033
01:28:55.680 --> 01:28:58.980
I think everything that we know about GPT for is by just running it on a bunch of inputs and
2034
01:28:59.020 --> 01:28:59.682
seeing what it does.
2035
01:29:01.267 --> 01:29:05.280
In theory, you can also look at the exact computation the model performed.
2036
01:29:05.460 --> 01:29:09.740
Like it's kind of like neuroscience, except you have a complete readout of exactly how the brain
2037
01:29:09.820 --> 01:29:11.790
works and exactly what it was thinking in every case.
2038
01:29:12.574 --> 01:29:13.840
And so you could hope that with access to
2039
01:29:13.840 --> 01:29:15.729
that information, you could do much better.
2040
01:29:15.749 --> 01:29:17.438
And you could also invest much more.
2041
01:29:17.538 --> 01:29:17.920
You can do much
2042
01:29:17.960 --> 01:29:19.708
better than neuroscience has done on humans.
2043
01:29:20.151 --> 01:29:22.120
And you can be able to say, we can learn about this
2044
01:29:22.160 --> 01:29:25.640
model, not only by observing its behavior, which is really hard because it's hard to predict how
2045
01:29:25.660 --> 01:29:28.048
will generalize in some new case.
2046
01:29:28.590 --> 01:29:31.380
But also, by looking at the computation it performs,
2047
01:29:31.600 --> 01:29:36.000
understanding why that computation leads to the behaviors you observe, and then reasoning about
2048
01:29:36.060 --> 01:29:39.840
whether that mechanism would generalize in an unpredictable way, or being able to use that
2049
01:29:39.860 --> 01:29:43.180
knowledge to flag when the mechanism is behaving in an unpredictable or novel way.
2050
01:29:45.602 --> 01:29:48.880
So this is a project, you know, a lot of, a reasonable number of people are working on,
2051
01:29:49.121 --> 01:29:52.400
both in academia and industry and nonprofits.
2052
01:29:54.104 --> 01:29:55.359
And that would be great.
2053
01:29:55.821 --> 01:29:57.240
It could be great if we understood something
2054
01:29:57.321 --> 01:30:00.060
about why GPT-4 said the things it said.
2055
01:30:02.282 --> 01:30:05.180
So Eliezer kept talking about inscrutable matrices
2056
01:30:05.685 --> 01:30:07.140
and using terms like gradient descent,
2057
01:30:07.220 --> 01:30:09.000
which I noticed you used during the course of this.
2058
01:30:09.501 --> 01:30:11.240
This is the thing we don't understand, right?
2059
01:30:11.300 --> 01:30:13.295
We don't understand actually what's going on
2060
01:30:14.102 --> 01:30:16.000
inside of the AI's quote-unquote brain.
2061
01:30:16.441 --> 01:30:24.160
We don't understand what these inscrutable matrices, how they work or what answers, what goals that might emerge from them.
2062
01:30:24.542 --> 01:30:26.160
Is this all part of the same thing?
2063
01:30:27.443 --> 01:30:29.000
Yeah, I think that's basically why we're worried.
2064
01:30:29.161 --> 01:30:29.700
That's exactly right.
2065
01:30:29.760 --> 01:30:30.820
And that's basically why we're worried.
2066
01:30:30.980 --> 01:30:32.180
Like we took a model.
2067
01:30:32.925 --> 01:30:33.919
We took a bunch of cases.
2068
01:30:34.581 --> 01:30:39.180
We messed with the weights of this model until it did really well on the hundred billion cases that we considered.
2069
01:30:40.144 --> 01:30:44.180
and now we wonder what's it going to do in some new case in like a case where for example models
2070
01:30:44.240 --> 01:30:47.760
do have the opportunity to cause incredible harm or could be able to get a high reward by causing
2071
01:30:47.800 --> 01:30:53.240
incredible harm um and the scary thing is like you have no idea what the we kind of understand
2072
01:30:53.300 --> 01:30:56.620
how gradient descent works it takes you to something that works really well in 100 billion
2073
01:30:56.700 --> 01:31:01.040
cases you test it on um but we have no idea how the resulting model works the resulting model is
2074
01:31:01.100 --> 01:31:06.560
basically like you know 150 matrix multiplies you multiply by a big 300 400 whatever you multiply
2075
01:31:06.661 --> 01:31:08.580
by a big matrix, and then you apply
2076
01:31:08.600 --> 01:31:10.560
it on the linearity, and then you multiply it by a big matrix
2077
01:31:10.681 --> 01:31:12.379
again, and then you apply it on the linearity, and we're just like
2078
01:31:13.651 --> 01:31:14.600
we have no idea what any
2079
01:31:14.640 --> 01:31:16.260
of the numbers in any of those matrices mean.
2080
01:31:16.740 --> 01:31:17.852
And that's not totally true.
2081
01:31:18.136 --> 01:31:18.540
I think we have some
2082
01:31:18.601 --> 01:31:20.600
idea of what some of the numbers mean, but at a high level, if you
2083
01:31:20.640 --> 01:31:22.799
take interesting behaviors, take a behavior of GPT-4
2084
01:31:22.961 --> 01:31:24.176
which does not appear in GPT-2, say,
2085
01:31:25.462 --> 01:31:26.580
I think for essentially every such
2086
01:31:26.641 --> 01:31:28.220
behavior, we do not understand
2087
01:31:28.866 --> 01:31:30.420
how GPT-4 is able to do the thing.
2088
01:31:30.540 --> 01:31:32.059
We understand some simple things,
2089
01:31:33.772 --> 01:31:34.520
and we don't
2090
01:31:34.560 --> 01:31:36.340
understand any
2091
01:31:36.622 --> 01:31:39.880
Yeah, we don't understand most of the complicated behaviors existing models engage in.
2092
01:31:41.080 --> 01:31:50.740
If you gave us the list of matrices and asked us, does the model do X, we would have no way to answer that question other than by running it a bunch of times and seeing if it did X, which again is not great.
2093
01:31:50.880 --> 01:32:01.260
That sort of gets you back to like, now you need to somehow either run in the real world and see what happens and hope it's not catastrophic or be able to construct simulated situations in the lab that are similar enough that the model behaves the same way there that it would in the real world.
2094
01:32:01.940 --> 01:32:03.158
You'd really love to not have to do that.
2095
01:32:03.721 --> 01:32:11.216
So there is maybe some hope that we can start understanding what's going on in these inscrutable matrices, but we haven't made major breakthroughs yet.
2096
01:32:11.476 --> 01:32:13.360
What is sort of the fourth category?
2097
01:32:13.842 --> 01:32:17.640
we've made progress progress okay it seems like a lot of progress you have to make i mean i think
2098
01:32:17.680 --> 01:32:20.480
elias and i probably disagree in the sense elias is probably at like you know one percent or point
2099
01:32:20.560 --> 01:32:24.180
one percent this would get far enough to meaningfully reduce risk whereas i'm probably
2100
01:32:24.301 --> 01:32:28.220
more like ten percent this gets far enough to meaningfully reduce risk maybe higher depending
2101
01:32:28.301 --> 01:32:31.140
exactly what you mean by me maybe i think there's like five ten percent this is good enough to
2102
01:32:31.180 --> 01:32:37.000
totally address the risk um and like ten to thirty percent this is good enough to take some meaningful
2103
01:32:37.481 --> 01:32:43.280
meaningful reduction in risk anyway that was third category um i'm in a fourth category i
2104
01:32:43.300 --> 01:32:47.600
I think it's pretty promising is just ultimately what we're wondering is if you have a bunch,
2105
01:32:47.700 --> 01:32:51.240
you train your AI in situations A where you're able to train it, you're able, if it fails,
2106
01:32:51.300 --> 01:32:52.079
it won't be catastrophic.
2107
01:32:52.280 --> 01:32:53.538
You're able to evaluate what the answer is.
2108
01:32:54.161 --> 01:32:57.740
Then you're going to deploy it in situation B where either you can't tell what the right
2109
01:32:57.800 --> 01:33:00.260
answer is, or if it failed, you wouldn't be able to fix the problem.
2110
01:33:02.401 --> 01:33:06.060
And then we want to understand how do models tend to generalize from like these kind of
2111
01:33:06.140 --> 01:33:09.380
easy cases or cases where we're able to supervise to cases where we're not able to supervise.
2112
01:33:10.304 --> 01:33:13.900
and you could hope to just build up a good scientific understanding of that question.
2113
01:33:14.020 --> 01:33:15.874
You could hope to say like, we're going to have a bunch of cases.
2114
01:33:15.914 --> 01:33:16.620
We're going to have looked at a
2115
01:33:16.680 --> 01:33:19.476
ton of models, understand what factors affect this generalization.
2116
01:33:19.496 --> 01:33:20.200
This is very similar to like,
2117
01:33:20.220 --> 01:33:24.420
if you imagine these two humps of like a hump of good behavior, which looks good because it is good
2118
01:33:24.480 --> 01:33:27.900
and a hump of bad behavior, which looks good because it's systematically corrupting measurements
2119
01:33:28.081 --> 01:33:28.765
or deceiving humans.
2120
01:33:29.187 --> 01:33:31.400
You kind of want to say, what are the conditions that determine which of those,
2121
01:33:31.681 --> 01:33:33.410
like, do you make the jump from one to the other?
2122
01:33:34.194 --> 01:33:35.360
Some sense there's two equally valid
2123
01:33:35.420 --> 01:33:37.290
generalizations and you're wondering which one do you get?
2124
01:33:38.495 --> 01:33:39.420
And I think there's just a lot to
2125
01:33:39.480 --> 01:33:44.060
do with having situations which have similar ambiguity about generalization, situations
2126
01:33:44.080 --> 01:33:47.880
where unsure about how the model will generalize, training huge numbers of models and understanding
2127
01:33:48.000 --> 01:33:52.876
what factors determine whether and when the systems generalize one way versus the other,
2128
01:33:54.240 --> 01:33:57.600
and then using some of what you learn either to diagnose risk or in the best case to say,
2129
01:33:57.660 --> 01:34:00.660
okay, if we train the model in the following way, if we use the following kinds of loss
2130
01:34:00.761 --> 01:34:02.292
functions, we get the intended generalization.
2131
01:34:03.420 --> 01:34:06.800
And I think that is reasonably likely in combination with other things.
2132
01:34:06.860 --> 01:34:08.039
I think it's reasonably likely to work.
2133
01:34:09.322 --> 01:34:11.020
Like, again, I said just naively,
2134
01:34:11.200 --> 01:34:12.620
there's maybe a 50% chance you're okay.
2135
01:34:12.720 --> 01:34:14.060
And maybe, you know, if you do a lot of work like this,
2136
01:34:14.080 --> 01:34:16.240
you can bump that up to like 60% chance that you're okay.
2137
01:34:17.161 --> 01:34:19.092
So this fourth category, what do we call this, Paul?
2138
01:34:20.580 --> 01:34:22.500
And this is the one that you're most optimistic about?
2139
01:34:23.828 --> 01:34:24.220
Oh, I don't know.
2140
01:34:24.542 --> 01:34:26.440
I don't know what I'm most optimistic about.
2141
01:34:26.762 --> 01:34:28.660
I think these four seem like broadly similarly important.
2142
01:34:29.985 --> 01:34:30.779
I don't know what you would call this.
2143
01:34:31.001 --> 01:34:32.880
It's like studying generalization or something.
2144
01:34:35.850 --> 01:34:38.640
this is again a question academics are very interested in they study in some ways they
2145
01:34:38.680 --> 01:34:41.980
mostly don't study the versions that are most relevant to takeover and there's some people
2146
01:34:41.980 --> 01:34:48.000
who are very interested in takeover in particular who study this question it's not something I've
2147
01:34:48.020 --> 01:34:51.960
worked on myself I'm pretty optimistic about it but I'm pretty optimistic about interpretability
2148
01:34:51.960 --> 01:34:55.640
I'm pretty optimistic about scalable supervision I'm reasonably optimistic about robustness it
2149
01:34:55.660 --> 01:35:01.500
seems hard but I think it's reasonable chance of helping a lot with this problem so Paul I see
2150
01:35:02.042 --> 01:35:08.120
you've laid out four different technical solutions here um and i'm very naive on this object but i
2151
01:35:08.280 --> 01:35:15.560
don't see uh tell and tell me please how many people what's the manpower behind these things
2152
01:35:15.680 --> 01:35:20.278
because it's great that we have these paths but we need manpower to actually go and execute on
2153
01:35:20.940 --> 01:35:24.620
this yeah it feels like you guys should be what's the lay of the land here like billions of dollars
2154
01:35:24.901 --> 01:35:30.900
funded in in solving this problem because we've got a lot of funding on developing ais don't we
2155
01:35:32.846 --> 01:35:35.640
I think it's hard to have an amount of funding for this problem that is similar to the amount
2156
01:35:35.660 --> 01:35:40.700
of funding for developing AIs, just because you got some pretty good profit incentive on the
2157
01:35:40.720 --> 01:35:41.164
making AIs.
2158
01:35:41.184 --> 01:35:43.139
I think we can get, yeah, I think it's a reasonable amount of funding.
2159
01:35:43.159 --> 01:35:43.320
I think
2160
01:35:43.320 --> 01:35:47.940
you're talking more like hundreds of millions or billions of dollars over the foreseeable future
2161
01:35:48.061 --> 01:35:49.025
available for solving it.
2162
01:35:49.086 --> 01:35:51.156
Maybe you could amp that up if the problem became more real.
2163
01:35:51.538 --> 01:35:51.980
Like right now,
2164
01:35:52.000 --> 01:35:54.680
most people are just like, look, there's probably not going to be a takeover in the next couple
2165
01:35:54.700 --> 01:35:55.021
of years.
2166
01:35:55.825 --> 01:35:57.452
It's very speculative risk in the long term.
2167
01:35:58.094 --> 01:35:59.420
What can you really do in advance?
2168
01:36:00.242 --> 01:36:06.780
anyway there's some money um there's a shortage of people who are excited like there's also a
2169
01:36:06.840 --> 01:36:09.540
shortage of scientists who are excited about doing this work and i think that's like hopefully
2170
01:36:09.661 --> 01:36:14.140
changing quite rapidly as the problem seems more real and ai seems more exciting and people are
2171
01:36:14.160 --> 01:36:17.540
shifting more into the space in addition to shifting into i think a fair number are shifting
2172
01:36:17.660 --> 01:36:24.960
into understanding various risks some fraction of whom care about takeover um so i'd say like
2173
01:36:26.068 --> 01:36:27.719
in terms of estimating how big this is right now
2174
01:36:28.447 --> 01:36:29.620
it depends a lot how you count
2175
01:36:29.822 --> 01:36:31.619
people who are not motivated by takeover but do work
2176
01:36:31.680 --> 01:36:33.640
that can still be relevant to reducing takeover
2177
01:36:34.468 --> 01:36:35.680
and just like
2178
01:36:35.720 --> 01:36:37.520
what do you apply a discount to apply no discount
2179
01:36:37.924 --> 01:36:39.600
maybe I'd say that you're talking like on the order
2180
01:36:39.889 --> 01:36:39.951
of
2181
01:36:44.897 --> 01:36:45.720
maybe 20 people
2182
01:36:45.862 --> 01:36:47.079
on scalable supervision stuff
2183
01:36:48.470 --> 01:36:49.820
maybe
2184
01:36:50.002 --> 01:36:51.640
20 people on the interpretability stuff that's
2185
01:36:51.701 --> 01:36:53.558
most relevant to takeover coupled with like
2186
01:36:55.342 --> 01:36:58.600
100 or a couple hundred people doing stuff that could be relevant to varying degrees.
2187
01:37:01.281 --> 01:37:09.960
On robustness, like, maybe again, looking at something like five to 10 who are, like,
2188
01:37:10.020 --> 01:37:14.260
motivated explicitly by addressing takeover risk, followed by, you know, a couple hundred
2189
01:37:14.280 --> 01:37:18.260
who are doing stuff that's plausibly relevant, of whom maybe, like, a couple dozen are stuff
2190
01:37:18.280 --> 01:37:21.160
that I would actually care a lot about or would think is highly relevant.
2191
01:37:23.285 --> 01:37:29.580
um and like on the generalization stuff maybe again looking at like
2192
01:37:31.985 --> 01:37:36.640
five-ish people who are doing it motivated by a takeover risk plus another you know on the order
2193
01:37:36.640 --> 01:37:40.380
of dozens who are doing stuff that could be relevant or helpful i mean so maybe in total
2194
01:37:40.380 --> 01:37:44.300
you're looking at something like 50 people 200 people who are motivated by takeover risk
2195
01:37:44.380 --> 01:37:48.480
explicitly in these areas and other adjacent areas together with like hundreds more who are
2196
01:37:48.520 --> 01:37:50.238
doing work that is hopefully relevant.
2197
01:37:50.400 --> 01:37:52.580
Paul, in the scheme of things, this is not that many, though.
2198
01:37:53.182 --> 01:37:54.300
Not that many people here.
2199
01:37:56.062 --> 01:37:56.880
Yeah, it's not that many people.
2200
01:37:56.980 --> 01:37:58.059
It's certainly small relative to AI.
2201
01:37:59.001 --> 01:38:00.500
And I would love it.
2202
01:38:00.620 --> 01:38:02.640
I mean, I'm like, there's a reasonable chance we're all going to die.
2203
01:38:03.302 --> 01:38:07.940
I think this is like the single most likely reason that I will personally die, probably.
2204
01:38:09.160 --> 01:38:09.246
Wow.
2205
01:38:10.364 --> 01:38:11.293
So like, that's big.
2206
01:38:12.140 --> 01:38:13.960
And this is someone working on AI safety.
2207
01:38:15.644 --> 01:38:16.960
Can I ask you another question?
2208
01:38:17.041 --> 01:38:17.920
So we've covered the technical.
2209
01:38:18.502 --> 01:38:23.320
Just want to brush on the other point of coordination that we were making around, you know, policy and human coordination.
2210
01:38:25.321 --> 01:38:31.180
There has been an open letter, which I'm sure you're familiar with, PAUSE, Giant AI Experiments, an open letter, Max Tegmark.
2211
01:38:32.223 --> 01:38:34.340
I think his organization put this together.
2212
01:38:34.820 --> 01:38:38.720
Elon Musk signed it, Andrew Yang, some others have signed this.
2213
01:38:38.883 --> 01:38:39.600
Val Noah Harari.
2214
01:38:39.780 --> 01:38:40.180
Yeah.
2215
01:38:40.301 --> 01:38:50.728
And basically the idea is that we should, the open letter states that we should pause all AI development for six months just to wrap our arms around this AI safety issue.
2216
01:38:51.048 --> 01:38:57.973
And so let's go no further than chat GPT-4 until we pause six months and all take a breath and figure out what this means.
2217
01:38:58.434 --> 01:39:02.837
Do you support a letter like this, given you're in kind of an AI safety?
2218
01:39:03.157 --> 01:39:07.120
So there's a question of whether you support, you think this is a good idea, this coordination mechanism.
2219
01:39:07.180 --> 01:39:15.447
And there's also a more meta question of, do you think we can actually solve this coordination mechanism?
2220
01:39:15.687 --> 01:39:24.154
And are we in some Scott Alexander level Moloch trap that makes it very difficult for humanity to solve?
2221
01:39:24.394 --> 01:39:27.777
Is this the don't look up scenario that David was painting earlier?
2222
01:39:27.797 --> 01:39:30.379
So first of all, have you signed this letter?
2223
01:39:30.419 --> 01:39:31.219
Would you sign this letter?
2224
01:39:31.239 --> 01:39:32.080
Do you think it's a good idea?
2225
01:39:33.762 --> 01:39:34.643
I didn't sign the letter.
2226
01:39:34.663 --> 01:39:43.232
I can give my overall take, which is something like, I think it would on balance be better to pause ad development or slow ad development now.
2227
01:39:43.733 --> 01:39:45.815
I think that is not at all obvious.
2228
01:39:46.296 --> 01:39:50.340
And I sympathize with people who think that it's a bad idea on balance to slow down so we can talk about why.
2229
01:39:50.981 --> 01:40:04.620
I think the like dominant thing I care about is I think that at some point it will become like if we're actually developing systems that pose significant risks and our measurement is not adequate to tell if they pose significant risks or measurements suggest they do pose significant risk.
2230
01:40:06.281 --> 01:40:11.880
I think at that point I'm going to have a much more forceful take of like right now I'd kind of be like, I kind of think it should.
2231
01:40:12.021 --> 01:40:16.160
Anyway, right now I think it's like a debatable issue and it's reasonable to want to slow.
2232
01:40:16.421 --> 01:40:20.400
I think at some point in the future, it will still be a debatable issue, but I don't think it is unreasonable to not want to slow.
2233
01:40:22.182 --> 01:40:24.819
And that like we actually kind of collectively really need to act on this.
2234
01:40:25.581 --> 01:40:33.640
I think the main thing that matters, I kind of agree with Eliezer's take that the six month pause does not actually help that much, that it does not reduce risk super far.
2235
01:40:35.021 --> 01:40:44.520
I think the main thing we need to do is get in a position where we are prepared to slow down based on risk, like build consensus about risk or consensus that we need to have measured risk.
2236
01:40:44.600 --> 01:40:47.820
that risk is currently high enough that our measurements are unacceptable, and then be ready
2237
01:40:47.941 --> 01:40:51.480
to slow down potentially much more than this, potentially six months, potentially like whatever
2238
01:40:51.580 --> 01:40:56.100
it takes as we manage that risk or like slow down the directions that are most risky.
2239
01:40:57.141 --> 01:40:59.700
That's like the main thing I think about and the main thing I care about.
2240
01:41:00.741 --> 01:41:06.220
And the main thing that I've like really liked, I think that is going to be like a kind of delicate
2241
01:41:06.341 --> 01:41:06.742
process.
2242
01:41:06.843 --> 01:41:10.237
And I think it is quite expensive in terms of human cost.
2243
01:41:10.337 --> 01:41:10.940
I mean, I think probably
2244
01:41:10.960 --> 01:41:13.140
I'm more optimistic about AI in some sense than most people.
2245
01:41:13.720 --> 01:41:17.080
But I think that like slowing AI development by years would probably,
2246
01:41:17.460 --> 01:41:17.580
yeah,
2247
01:41:17.600 --> 01:41:19.520
it would come with a kind of tremendous human costs,
2248
01:41:20.181 --> 01:41:21.660
which I think is worth paying,
2249
01:41:21.940 --> 01:41:22.210
but it's,
2250
01:41:23.281 --> 01:41:26.120
I sympathize with people in AI who are skeptical about that or who don't
2251
01:41:26.203 --> 01:41:26.740
take it lightly.
2252
01:41:30.183 --> 01:41:31.274
I think it's going to be like,
2253
01:41:31.960 --> 01:41:34.460
I think we can get to the point where there's kind of consensus that we do
2254
01:41:34.502 --> 01:41:35.000
need to slow.
2255
01:41:35.140 --> 01:41:37.860
The risk is unacceptable that the benefits of going faster are not that
2256
01:41:37.981 --> 01:41:39.340
large compared to what's at stake.
2257
01:41:41.188 --> 01:41:44.140
I think the game is getting to the point where we are having that discussion
2258
01:41:44.261 --> 01:41:48.300
we have measurements in place and we have institutions that are set up such that they can slow
2259
01:41:49.814 --> 01:41:50.439
I think that like
2260
01:41:51.866 --> 01:41:55.060
there's some amount of slowing that can happen by voluntary self-regulation amongst
2261
01:41:55.810 --> 01:41:56.459
Western labs
2262
01:41:56.945 --> 01:41:58.220
that is I think there's a reasonable room
2263
01:41:58.280 --> 01:42:00.157
I think a nice fact about the situation is
2264
01:42:00.722 --> 01:42:02.399
most of the people involved in AI development now
2265
01:42:03.485 --> 01:42:06.800
do I think genuinely not want AI to go over and kill everyone
2266
01:42:07.889 --> 01:42:09.240
and do have some appreciation of the risk
2267
01:42:09.300 --> 01:42:13.560
at least in principle, and are open to saying, okay, here's a set of practices which will
2268
01:42:13.640 --> 01:42:16.313
adequately manage that risk, and we will adopt them even if we slow.
2269
01:42:16.997 --> 01:42:17.680
And I think we can get,
2270
01:42:17.900 --> 01:42:20.440
you know, you can't get that much slow down that way, but I think you can get significant
2271
01:42:20.561 --> 01:42:24.280
additional safety and plausibly something like, you know, six to 12 months of slowing of
2272
01:42:24.802 --> 01:42:29.760
potential catastrophe, just from people at a couple labs saying, we don't really want to
2273
01:42:29.780 --> 01:42:32.149
cause incredible catastrophe, we see the case for slowing.
2274
01:42:33.153 --> 01:42:34.840
I think maybe to even do that,
2275
01:42:34.920 --> 01:42:38.220
and especially as you want to go beyond that, like you really need to have something more like
2276
01:42:38.240 --> 01:42:42.720
a regulatory regime where we have said, you know, there was this voluntary set of practices,
2277
01:42:42.840 --> 01:42:43.624
the labs endorsed.
2278
01:42:44.106 --> 01:42:46.859
Some people are behaving in a way that doesn't comply with those.
2279
01:42:46.879 --> 01:42:47.100
And there's
2280
01:42:47.120 --> 01:42:51.300
kind of broad consensus that's not reasonable, at least amongst, you know, anyway, this is
2281
01:42:51.481 --> 01:42:52.565
amongst some group of that.
2282
01:42:53.267 --> 01:42:55.998
And you can then, you know, this is in scope.
2283
01:42:56.058 --> 01:42:56.600
Like the thing that we're
2284
01:42:56.660 --> 01:43:00.520
asking the state to do here is say, you really don't want an AI lab to be in a position where
2285
01:43:00.520 --> 01:43:02.229
they're like violently overthrowing the US government.
2286
01:43:02.792 --> 01:43:04.140
This is not like a crazy, it's
2287
01:43:04.200 --> 01:43:08.500
definitely like in the in the government's wheelhouse it is their job and so to the extent
2288
01:43:08.560 --> 01:43:11.960
that like some companies are like now we should push ahead i think there's reasonable grounds for
2289
01:43:12.000 --> 01:43:16.300
the world to be like this is not the kind of thing that you get to just push ahead with i think that's
2290
01:43:16.300 --> 01:43:20.580
hard now i think procedurally now is a little bit hard to make the case that like the state should
2291
01:43:20.640 --> 01:43:24.380
crush ai development i think there'll come a point in the future where it's not that hard to make the
2292
01:43:24.400 --> 01:43:28.580
case that there are some developers who are quite risky and actually it is not reasonable that's not
2293
01:43:28.580 --> 01:43:32.600
a reasonable behavior it's not has anyone proposed something paul okay so because when you say there's
2294
01:43:32.600 --> 01:43:34.889
only like 50 to 100 people working on AI safety.
2295
01:43:34.909 --> 01:43:37.520
And this is like, I mean, I'm sure millions in
2296
01:43:37.600 --> 01:43:42.998
funding, but not hundreds of billions that AI development is actually going to have.
2297
01:43:43.519 --> 01:43:43.720
You know,
2298
01:43:43.941 --> 01:43:48.360
it makes me wonder if some sort of like an AI tax has been proposed or something where some
2299
01:43:48.420 --> 01:43:54.880
percentage of profit from, you know, AI utility goes to sort of a fund that just, you know,
2300
01:43:54.980 --> 01:43:56.226
pays for research.
2301
01:43:56.247 --> 01:43:58.860
I mean, this happens at the government level, something like this, because
2302
01:43:59.221 --> 01:44:04.314
it does seem like we have a public goods funding type of problem.
2303
01:44:04.755 --> 01:44:06.420
Maybe it's not just that.
2304
01:44:06.460 --> 01:44:06.580
Maybe
2305
01:44:06.680 --> 01:44:08.345
we also have an education problem.
2306
01:44:08.867 --> 01:44:12.739
This is why David and I are sort of looking into AI quite
2307
01:44:13.100 --> 01:44:15.669
aggressively after our episode with Eliezer.
2308
01:44:15.709 --> 01:44:19.140
It's just like, all this crypto stuff doesn't really
2309
01:44:19.261 --> 01:44:22.877
matter in the scheme of things if the robots are literally coming to kill us, right?
2310
01:44:23.379 --> 01:44:23.640
We should
2311
01:44:23.700 --> 01:44:25.244
like think hard about that.
2312
01:44:26.247 --> 01:44:26.508
Cool.
2313
01:44:26.528 --> 01:44:30.800
We have like crypto systems and decentralization and, you know,
2314
01:44:30.940 --> 01:44:34.392
crypto economic tools, but we're all dead.
2315
01:44:34.412 --> 01:44:36.138
Now the robots, now the robots have them.
2316
01:44:36.198 --> 01:44:36.700
Great job,
2317
01:44:36.780 --> 01:44:37.121
guys.
2318
01:44:37.583 --> 01:44:42.720
And so we're taking a quick detour here, but like it's honestly because we've just been
2319
01:44:42.921 --> 01:44:46.915
made aware of the the stakes here and the existential threat.
2320
01:44:46.935 --> 01:44:48.260
So there's also an education
2321
01:44:48.481 --> 01:44:49.624
game here, Paul.
2322
01:44:49.685 --> 01:44:50.086
I don't know.
2323
01:44:51.350 --> 01:44:53.798
Where do you think the resources should be spent?
2324
01:44:53.838 --> 01:44:54.340
Should it be on
2325
01:44:54.762 --> 01:44:57.857
regulation, education, like broad strokes level?
2326
01:44:57.877 --> 01:44:58.420
What can we do?
2327
01:45:00.441 --> 01:45:03.836
I think an important thing to have in mind going into that, like when I say 5,200 people working
2328
01:45:04.981 --> 01:45:07.996
on takeover, I think there's a lot of things we might care about in AI safety.
2329
01:45:08.338 --> 01:45:08.760
There's a lot of
2330
01:45:08.780 --> 01:45:10.953
possible risks from AI, a lot of possible harms.
2331
01:45:11.093 --> 01:45:12.160
A lot of them are things that are like,
2332
01:45:12.901 --> 01:45:16.540
when you deploy chat GPT, there are ways in which people might be unhappy with the outcome,
2333
01:45:16.660 --> 01:45:17.724
There are privacy issues.
2334
01:45:18.365 --> 01:45:23.240
There are like various effects, like systemic effects on society from its deployment you might be worried about.
2335
01:45:23.540 --> 01:45:28.320
There's like many more people working on these issues broadly just because there are a lot of issues.
2336
01:45:28.600 --> 01:45:31.660
I think every issue is able to feel like this issue is like crazily neglected.
2337
01:45:31.860 --> 01:45:38.720
But it's worth flagging like AI safety is this much broader category of which like the risk of AI takeover is a minority of people who work on it right now.
2338
01:45:38.880 --> 01:45:42.900
I think it's a larger share of public discourse than it is of scientific interest.
2339
01:45:46.009 --> 01:45:50.120
so I think that like things like public spending I think right now probably the principal bottlenecks
2340
01:45:50.180 --> 01:45:54.520
are not spending more money I think it is useful having more money does it's not like there's no
2341
01:45:54.580 --> 01:45:58.900
need for more money more money is good there's lots of things one could fund but I think that it
2342
01:45:59.020 --> 01:46:03.920
is the key bottleneck is probably having projects that are appealing like having people who have
2343
01:46:03.940 --> 01:46:07.340
relevant background to perform projects who are sufficiently excited about doing work to try and
2344
01:46:07.380 --> 01:46:11.240
manage this risk that they are in the step of asking for money or they would do it if money
2345
01:46:11.300 --> 01:46:11.862
was available.
2346
01:46:12.464 --> 01:46:13.005
It's talent.
2347
01:46:13.286 --> 01:46:14.108
Talent's a bottleneck.
2348
01:46:14.810 --> 01:46:17.237
I think that's the big problem right now.
2349
01:46:17.257 --> 01:46:18.140
I mean,
2350
01:46:18.863 --> 01:46:19.949
it's, again, neither is really a bottleneck.
2351
01:46:19.969 --> 01:46:21.840
I think like you can, you can spend money to like
2352
01:46:21.860 --> 01:46:25.620
help people switch into the field to fund projects that are more speculative and more long shots
2353
01:46:27.067 --> 01:46:27.991
to increase the incentives.
2354
01:46:28.011 --> 01:46:29.800
And like some people will switch, even if you're funding work that
2355
01:46:29.820 --> 01:46:32.520
would have been done anyway, if you fund it more generously, more people will get into the field
2356
01:46:32.620 --> 01:46:33.083
inevitably.
2357
01:46:33.103 --> 01:46:34.912
So it's like a lot of ways you can use money.
2358
01:46:35.032 --> 01:46:36.540
And like, it's not exactly clear whether
2359
01:46:36.641 --> 01:46:39.880
one should be trying to have more money available or more talent available, but it's like a little
2360
01:46:39.880 --> 01:46:40.363
bit rough right now.
2361
01:46:40.383 --> 01:46:43.360
I think if you're trying to spend money and trying to look like, where do you
2362
01:46:43.420 --> 01:46:44.003
spend that money?
2363
01:46:44.545 --> 01:46:48.120
It's hard to spend money on a problem that like practicing, like practitioners
2364
01:46:48.301 --> 01:46:51.531
in industry and scientists in academia are not prioritizing.
2365
01:46:52.173 --> 01:46:54.340
Um, most successful spending comes
2366
01:46:54.360 --> 01:46:57.160
from having like people who want to do the work who like need the money to do the work, or at
2367
01:46:57.180 --> 01:46:59.550
least are open to doing the work and are somewhat excited about it.
2368
01:47:00.735 --> 01:47:01.820
Um, so Paul, if there are
2369
01:47:01.940 --> 01:47:04.192
talented, what, what kind of talent is really missing?
2370
01:47:04.212 --> 01:47:05.900
Like what is the archetype of someone
2371
01:47:06.021 --> 01:47:09.260
that this area of AI really needs?
2372
01:47:11.583 --> 01:47:13.800
I think there's a lot of kinds of work to be done.
2373
01:47:14.723 --> 01:47:16.440
So we've talked mostly about technical solutions
2374
01:47:16.541 --> 01:47:18.060
where like mostly the relevant kinds of talent
2375
01:47:18.161 --> 01:47:20.320
are either people who have mathematical backgrounds
2376
01:47:20.381 --> 01:47:21.400
or backgrounds in computer science
2377
01:47:21.461 --> 01:47:22.420
or experience in machine learning
2378
01:47:22.420 --> 01:47:23.978
or who are good engineers or good at science.
2379
01:47:24.240 --> 01:47:25.860
I think a very common pattern,
2380
01:47:25.980 --> 01:47:27.320
like a lot of people who work in this field
2381
01:47:27.482 --> 01:47:28.919
are people who came in from some other area,
2382
01:47:29.040 --> 01:47:30.000
like who used to do physics.
2383
01:47:30.560 --> 01:47:31.680
I think it's like, if you're doing physics
2384
01:47:31.782 --> 01:47:32.500
and right now in the world,
2385
01:47:32.621 --> 01:47:33.639
it's reasonable to say like,
2386
01:47:34.061 --> 01:47:35.520
you know, we can pause the physics for a little while
2387
01:47:35.600 --> 01:47:38.120
this AI thing is going to be urgent for the next like 10, 20 years.
2388
01:47:38.940 --> 01:47:40.680
And so I think like it is reasonable to have people shifting
2389
01:47:40.700 --> 01:47:42.100
and just have broad scientific backgrounds
2390
01:47:42.202 --> 01:47:43.440
who have experience doing research,
2391
01:47:43.620 --> 01:47:45.780
understand how to study complicated empirical questions.
2392
01:47:46.420 --> 01:47:47.860
I think it's like a broad range of technical backgrounds
2393
01:47:47.901 --> 01:47:49.180
that slot somewhere into this picture.
2394
01:47:49.641 --> 01:47:51.819
And if you broaden your scope from just those technical issues
2395
01:47:52.586 --> 01:47:53.697
to like the whole picture,
2396
01:47:55.081 --> 01:47:57.000
then there's an even broader range of people, right?
2397
01:47:57.000 --> 01:47:58.719
There's a lot of work that's like institutional work
2398
01:47:58.821 --> 01:47:59.680
or understanding what,
2399
01:47:59.800 --> 01:48:01.300
like how should we approach this measurement thing?
2400
01:48:01.481 --> 01:48:04.060
Or like, can we make progress on this like public discussion
2401
01:48:04.142 --> 01:48:04.880
or public advocacy?
2402
01:48:05.101 --> 01:48:08.820
Can we understand just like generalist forecasting of like, what is the state of play?
2403
01:48:08.960 --> 01:48:15.700
To what extent are objections to this reasonable, like actual reasons to have lower, like which
2404
01:48:15.760 --> 01:48:19.560
of the things people would say as objections are like actual considerations that should
2405
01:48:19.580 --> 01:48:23.420
change your view versus targets for advocacy where you just want to try and change views
2406
01:48:23.641 --> 01:48:26.960
and like what is a reasonable view in light of, you know, what is actually the right synthesis
2407
01:48:27.121 --> 01:48:30.700
of consensus if you take the things experts believe that are most across different fields
2408
01:48:30.741 --> 01:48:31.400
that are most relevant?
2409
01:48:32.340 --> 01:48:34.080
I think there's just like a lot of stuff to do.
2410
01:48:34.441 --> 01:48:40.900
The one I understand best is definitely, I don't know, there's probably 500 reasonable projects in AI safety or something, and there's not that many people working on them.
2411
01:48:41.040 --> 01:48:55.340
So just people coming in, looking at the landscape, seeing where they can fit in, trying to do some projects in that space, whether that's in interpretability or in scalable supervision or just studying how models generalize or other science about models or understanding work on robustness.
2412
01:48:56.386 --> 01:48:57.180
I give those four categories.
2413
01:48:57.260 --> 01:48:58.320
That's not an exhaustive breakdown.
2414
01:48:58.480 --> 01:49:02.500
For example, that does not actually include my day job and all the work that I do normally.
2415
01:49:03.702 --> 01:49:05.260
What's your day job and what's the work that you do?
2416
01:49:06.322 --> 01:49:08.780
It's primarily on trying to develop alternative training strategies
2417
01:49:09.062 --> 01:49:12.060
or just qualitatively new techniques that don't have this issue,
2418
01:49:12.180 --> 01:49:15.400
like don't incentivize takeover in the same way the current techniques do.
2419
01:49:17.042 --> 01:49:18.800
I think it's one thing that goes in the portfolio.
2420
01:49:19.180 --> 01:49:22.240
I think there's a 10% chance that we're able to come up with a really good strategy
2421
01:49:22.401 --> 01:49:24.400
that does qualitatively change the game.
2422
01:49:24.760 --> 01:49:26.940
And if risks seem large, we could adopt it.
2423
01:49:27.281 --> 01:49:28.399
That's another category of work.
2424
01:49:28.882 --> 01:49:29.660
Some people work on that.
2425
01:49:30.260 --> 01:49:32.400
I think it's less likely to absorb huge numbers of people.
2426
01:49:32.540 --> 01:49:35.520
It's a little bit more of a long shot, but I think it's a good thing to do.
2427
01:49:38.304 --> 01:49:39.160
Yeah, so that's my high level.
2428
01:49:39.340 --> 01:49:40.420
There's just a lot of projects.
2429
01:49:41.181 --> 01:49:45.040
I think most projects, if we go through that list, like almost all of those projects are
2430
01:49:45.904 --> 01:49:49.420
hurting for some combination of technical talent doing them, like senior researchers
2431
01:49:49.722 --> 01:49:53.060
who have research experience and are able to like bring good judgment and mentorship
2432
01:49:53.060 --> 01:49:53.820
to these problems.
2433
01:49:54.700 --> 01:49:57.980
Management or like entrepreneurial spirit or like management experience to like actually
2434
01:49:58.020 --> 01:49:59.859
onboard people and coordinate projects working on them.
2435
01:50:00.341 --> 01:50:11.780
And it was just like an incredibly high premium to people who have technical backgrounds and are like entrepreneur entrepreneurial enough to like look at the space, engage with like people's thoughts about what helps form their own views that are reasonable and then start on projects.
2436
01:50:12.282 --> 01:50:14.640
Like the returns to doing that right now are just crazy high, I think.
2437
01:50:15.360 --> 01:50:15.481
Right.
2438
01:50:15.742 --> 01:50:19.320
Paul, what what would you say were the odds to be?
2439
01:50:19.800 --> 01:50:26.200
This is a I know this isn't your field, but like the odds that somebody in this field wins a Nobel Prize in like the next 20 years.
2440
01:50:28.186 --> 01:50:32.260
oh i think they seem pretty i mean someone working on like al i'm and stuff i think our
2441
01:50:32.300 --> 01:50:35.820
first problem is that like there's no nobel prize in any adjacent area i guess it would be like
2442
01:50:37.405 --> 01:50:41.380
i guess my point is no prize for saving humanity i mean we should we should work on that right yeah
2443
01:50:41.621 --> 01:50:46.800
there's a prize somewhere if for someone solves this problem yeah so you have like you have like
2444
01:50:46.840 --> 01:50:51.380
touring awards or fields medals or whatever the term maybe more analogous i think you'd think of
2445
01:50:51.962 --> 01:50:56.160
i don't know i mean i think it's not that likely 20 years is not that long a time horizon most
2446
01:50:56.280 --> 01:50:59.500
these things tend to be like given latent careers for early career achievements although like fields
2447
01:50:59.580 --> 01:51:03.280
medals are a bit of a distinction most of the stuff people are doing is not really in a category
2448
01:51:03.360 --> 01:51:10.340
that would get a touring award or fields medal um it was more just like there's there's table
2449
01:51:10.420 --> 01:51:14.580
stakes should be that there's should be some big recognition for whoever solves this problem so
2450
01:51:14.600 --> 01:51:19.240
like i mean that's yeah that's my call to action for somebody make a prize somebody make a prize
2451
01:51:19.260 --> 01:51:22.900
real quick somebody make a prize humanity that is the thing you could do with money and it's like
2452
01:51:22.960 --> 01:51:26.260
not a crazy thing to do with money it's like also a thing that sort of in some sense scientific
2453
01:51:26.381 --> 01:51:30.000
prestige is one of the inputs like a prize could be made of either money or prestige it's harder
2454
01:51:30.040 --> 01:51:33.560
to synthesize the prize made of prestige and that's more like is the scientific community bought in
2455
01:51:34.543 --> 01:51:39.240
um i mean a high level i think it is fairly likely that in retrospect in like the maybe half of
2456
01:51:39.280 --> 01:51:42.860
worlds where this was a real problem that in retrospect people will be very excited about
2457
01:51:42.940 --> 01:51:47.000
some fraction of the work that was done in the area people like that was a big deal sorry we're
2458
01:51:47.060 --> 01:51:51.260
asleep at the wheel oops so i'm like i mean i think it's 50 50 chance that people will feel
2459
01:51:51.402 --> 01:51:53.100
that way in retrospect, generally.
2460
01:51:53.785 --> 01:51:54.320
The question of like,
2461
01:51:54.380 --> 01:51:55.919
is there any kind of existing institution set up
2462
01:51:55.960 --> 01:51:58.060
which would provide significant recognition?
2463
01:51:58.220 --> 01:51:59.819
I'm like less sure about that.
2464
01:52:00.180 --> 01:52:01.440
I mean, I hope like the work we do,
2465
01:52:01.540 --> 01:52:02.760
you know, like that's like good academic work.
2466
01:52:02.820 --> 01:52:04.479
I think it has like probably higher than the base rate
2467
01:52:05.469 --> 01:52:06.680
for like theoretical computer science
2468
01:52:07.509 --> 01:52:08.480
of winning prestigious prizes.
2469
01:52:08.820 --> 01:52:11.140
But it's kind of just like those processes
2470
01:52:11.746 --> 01:52:13.100
on academic merit rather than that.
2471
01:52:14.083 --> 01:52:16.600
I think we can go ahead and zoom forward into the future
2472
01:52:16.781 --> 01:52:19.960
where if humanity does thread this needle,
2473
01:52:20.444 --> 01:52:22.620
there will be an institution that does award a prize.
2474
01:52:22.740 --> 01:52:24.320
I think we can run on that assumption.
2475
01:52:25.222 --> 01:52:27.060
Paul, thank you so much for guiding us through.
2476
01:52:27.280 --> 01:52:30.040
This was exactly the episode that we wanted to produce.
2477
01:52:30.401 --> 01:52:32.440
And I think I got a lot of my answers,
2478
01:52:33.183 --> 01:52:34.000
my questions answered.
2479
01:52:34.181 --> 01:52:35.760
But I have one last question for you.
2480
01:52:36.561 --> 01:52:40.060
My strategy thus far up to this point
2481
01:52:40.562 --> 01:52:43.814
has been to just be really polite to Siri and Alexa.
2482
01:52:46.160 --> 01:52:48.257
And I'm wondering if that is, in your opinion,
2483
01:52:48.983 --> 01:52:51.200
is moving the needle for me, me specifically.
2484
01:52:52.585 --> 01:52:53.580
I don't care about the rest of the world.
2485
01:52:53.721 --> 01:52:55.140
If that does anything at all.
2486
01:52:56.703 --> 01:52:58.240
I'm going to go with probably no.
2487
01:52:59.823 --> 01:53:01.119
That is, I think there is some real thing
2488
01:53:01.403 --> 01:53:02.820
about humanity treating AI systems
2489
01:53:02.860 --> 01:53:04.480
with like respect and dignity and being like,
2490
01:53:04.540 --> 01:53:05.659
look, these things are going to be,
2491
01:53:06.641 --> 01:53:08.460
I think there's a lot of room for humanity to do wrong
2492
01:53:08.642 --> 01:53:10.080
by AI systems we create that are smart.
2493
01:53:10.440 --> 01:53:12.220
I think being nice to Siri and Alexa is probably
2494
01:53:13.186 --> 01:53:15.360
not even the place you most personally could help with that.
2495
01:53:15.640 --> 01:53:16.199
They probably don't.
2496
01:53:16.563 --> 01:53:18.180
I do hope, well, my philosophy is like,
2497
01:53:18.241 --> 01:53:20.300
at some point, these things will be AIs.
2498
01:53:20.922 --> 01:53:23.957
There will be an AI on the other side of that robot,
2499
01:53:24.740 --> 01:53:25.320
that microphone.
2500
01:53:25.540 --> 01:53:26.460
And my philosophy is,
2501
01:53:26.761 --> 01:53:28.699
why not start treating it with respect now?
2502
01:53:28.820 --> 01:53:30.010
I think the bigger question is,
2503
01:53:31.300 --> 01:53:33.400
how will they look upon this podcast?
2504
01:53:33.640 --> 01:53:35.300
Will they look favorably upon this podcast
2505
01:53:35.542 --> 01:53:37.440
in our body of work at Bankless so far, David?
2506
01:53:37.802 --> 01:53:39.240
That might be the bigger question in my mind.
2507
01:53:40.002 --> 01:53:42.420
And one last concluding question for me, Paul,
2508
01:53:42.460 --> 01:53:45.700
is because you said 20% doom scenario,
2509
01:53:46.203 --> 01:53:48.900
which leaves 80% for non-doomed scenario.
2510
01:53:49.481 --> 01:53:52.660
And I bet there's some like mediocre scenarios in there too.
2511
01:53:53.101 --> 01:53:55.094
So tell me about the 20% utopia scenario.
2512
01:53:56.543 --> 01:53:57.760
Like, is there the possibility
2513
01:53:57.901 --> 01:54:00.200
this all goes really, really well?
2514
01:54:00.704 --> 01:54:02.140
And what happens on the other side?
2515
01:54:02.280 --> 01:54:03.720
Leave us with some optimism here.
2516
01:54:05.425 --> 01:54:06.260
I'm not that good a man.
2517
01:54:06.300 --> 01:54:07.299
I'm leaving people with optimism.
2518
01:54:07.640 --> 01:54:09.597
I think maybe I'm like 50, more like 50-50
2519
01:54:10.101 --> 01:54:12.020
that humanity achieves like a very good outcome.
2520
01:54:12.605 --> 01:54:13.580
That is an outcome where we're like,
2521
01:54:14.042 --> 01:54:16.393
this is about as good as we could have expected it to be.
2522
01:54:16.936 --> 01:54:17.840
Maybe a little bit less than that,
2523
01:54:17.920 --> 01:54:19.627
but more like 50-50 than 20%, I think.
2524
01:54:20.289 --> 01:54:21.093
So that's optimistic.
2525
01:54:21.213 --> 01:54:22.960
And I'm just, I think it's really hard
2526
01:54:23.000 --> 01:54:26.300
to talk about what the political economy of that world is like over the very long run.
2527
01:54:26.920 --> 01:54:30.820
I think the big thing is humanity, I think, still has a long history in front of us.
2528
01:54:31.420 --> 01:54:34.474
I think AI means that that history is probably going to get compressed.
2529
01:54:34.735 --> 01:54:35.820
I think there's a long
2530
01:54:35.880 --> 01:54:38.112
time for institutions to change and things to happen.
2531
01:54:38.473 --> 01:54:39.760
I think a lot of that is going to happen
2532
01:54:39.820 --> 01:54:43.160
much, much faster than it would have if you're thinking about tens of thousands of years of
2533
01:54:43.240 --> 01:54:44.044
human history to date.
2534
01:54:44.124 --> 01:54:47.280
I think you're probably thinking more like, you know, tens of years
2535
01:54:47.602 --> 01:54:50.134
before the world is like very, very radically transformed.
2536
01:54:50.516 --> 01:54:51.300
I don't really know what that world
2537
01:54:51.340 --> 01:54:51.722
looks like.
2538
01:54:52.325 --> 01:54:55.338
I think that like a lot of human problems are like humans versus nature.
2539
01:54:55.479 --> 01:54:55.720
Like we
2540
01:54:55.800 --> 01:54:59.515
die of old age and disease and people have physical want and stuff.
2541
01:54:59.635 --> 01:55:00.900
And I think the bad is probably
2542
01:55:00.940 --> 01:55:02.145
going to be really much better.
2543
01:55:02.185 --> 01:55:04.213
I think that's like even more than 50%.
2544
01:55:05.377 --> 01:55:06.060
The problems that are
2545
01:55:06.060 --> 01:55:08.452
caused by humanity versus nature are going to be pretty good.
2546
01:55:08.593 --> 01:55:09.920
And I think those are, you know,
2547
01:55:10.081 --> 01:55:11.400
I think our lives would be quite good.
2548
01:55:12.060 --> 01:55:15.660
In some sense, human versus human conflict is mostly problematic because there's a limited
2549
01:55:16.228 --> 01:55:16.979
budget to go around.
2550
01:55:18.641 --> 01:55:21.020
And then beyond that, it gets really hard to say what the character of that world is
2551
01:55:21.060 --> 01:55:24.140
like and what we choose to do if, in fact, we're in a pretty good position physically,
2552
01:55:24.281 --> 01:55:27.600
if we have incredible resources at our disposal, what kind of world we make.
2553
01:55:28.161 --> 01:55:29.680
That's a longer and more complicated discussion.
2554
01:55:29.820 --> 01:55:30.439
But I'm pretty psyched.
2555
01:55:30.540 --> 01:55:33.960
I mean, personally, I'm just I'm very glad I'm living now instead of any time.
2556
01:55:34.422 --> 01:55:36.840
And I would definitely take like 50 percent chance of death.
2557
01:55:37.762 --> 01:55:44.360
50% chance of early death seems like kind of very small compared to the overall change in expected quality of life from being alive today
2558
01:55:44.380 --> 01:55:45.760
Wow, that is extremely optimistic
2559
01:55:45.800 --> 01:55:48.360
I'm much more optimistic than Eliezer, definitely
2560
01:55:48.360 --> 01:55:49.360
Yeah, that's for sure
2561
01:55:49.601 --> 01:55:54.420
We've got a coin flip here, either it goes very poorly or it might go well for us
2562
01:55:54.561 --> 01:55:57.020
And Paul, thank you for the work that you're doing to make this go well
2563
01:55:57.747 --> 01:55:59.100
It's been a pleasure to have you on Bankless
2564
01:55:59.828 --> 01:56:00.940
Thanks for having me, it was great talking
2565
01:56:01.242 --> 01:56:04.400
Guys, Paul's website, we'll leave it in the action items for you
2566
01:56:05.590 --> 01:56:06.740
the Alignment Research Center.
2567
01:56:06.900 --> 01:56:07.895
That's at alignment.org.
2568
01:56:08.340 --> 01:56:10.320
You can check out what his organization is doing.
2569
01:56:10.500 --> 01:56:12.820
Also, we'll include a link to Paul Christiana's website
2570
01:56:13.226 --> 01:56:14.100
with some of his writings,
2571
01:56:14.240 --> 01:56:16.620
including, I think, some links to some of his debates
2572
01:56:17.082 --> 01:56:20.660
with Eliezer Yudkowsky from the archives as well.
2573
01:56:21.061 --> 01:56:21.859
Got to end with this.
2574
01:56:22.000 --> 01:56:24.440
Of course, none of this has been financial advice,
2575
01:56:24.581 --> 01:56:25.938
but you got to know that we're talking about AI.
2576
01:56:26.542 --> 01:56:28.200
We mentioned crypto a couple of times,
2577
01:56:28.640 --> 01:56:29.539
but I'll just end with this.
2578
01:56:29.722 --> 01:56:30.460
AI is risky.
2579
01:56:30.840 --> 01:56:31.660
The stakes are high.
2580
01:56:32.141 --> 01:56:33.160
We could lose a lot here,
2581
01:56:33.542 --> 01:56:34.488
but we are headed west.
2582
01:56:34.589 --> 01:56:35.454
This is the frontier.
2583
01:56:35.615 --> 01:56:36.420
It's not for everyone,
2584
01:56:36.580 --> 01:56:38.554
but we're glad you're with us on the bankless journey.
2585
01:56:38.937 --> 01:56:39.400
Thanks a lot.