Could AI Really Kill Us All?

WEBVTT
1
00:00:00.170 --> 00:00:02.450
Is AI really about to kill us all?
2
00:00:02.470 --> 00:00:06.410
Hi, I'm Wendy Zuckerman and this is Science Versus, the
3
00:00:06.429 --> 00:00:17.420
show that pits facts against the fallout from AI. We've
4
00:00:17.440 --> 00:00:20.719
been hearing for years that AI poses an existential threat
5
00:00:20.739 --> 00:00:23.920
to humanity and this is often coming from the very
6
00:00:23.980 --> 00:00:27.000
people who run the biggest AI companies in the world.
7
00:00:27.300 --> 00:00:30.260
I think AI will Probably, like, most likely sort of
8
00:00:30.899 --> 00:00:31.780
lead to the end of the world.
9
00:00:31.879 --> 00:00:37.320
My chance that something goes, you know, really quite catastrophically
10
00:00:37.399 --> 00:00:40.359
wrong on the scale of, you know, human civilization, you know,
11
00:00:40.380 --> 00:00:43.460
might be somewhere between 10 and 25 percent.
12
00:00:43.939 --> 00:00:48.039
Mark my words, AI is far more dangerous than nukes.
13
00:00:48.549 --> 00:00:53.049
Just recently, a researcher named Jacob Coxon, who resigned from Anthropic,
14
00:00:53.509 --> 00:00:55.750
said that some of the people working on this tech
15
00:00:56.250 --> 00:00:58.909
really think that it could kill us all in the
16
00:00:59.020 --> 00:01:00.780
next 10 years.
17
00:01:01.479 --> 00:01:02.579
This journalist asked him.
18
00:01:02.600 --> 00:01:06.200
You truly believe that AI could kill us all in
19
00:01:06.239 --> 00:01:09.640
less than a decade? Yes, I do.
20
00:01:09.900 --> 00:01:11.200
And it's not just me.
21
00:01:12.250 --> 00:01:16.420
Many people working in the industry, including people, executives, senior researchers,
22
00:01:16.879 --> 00:01:20.260
all genuinely believe that there is a substantial probability that
23
00:01:20.280 --> 00:01:22.180
this technology could kill everyone.
24
00:01:22.859 --> 00:01:25.620
And one of the big events that everyone is pointing
25
00:01:25.659 --> 00:01:28.859
to to say, look how scary this tech is, is
26
00:01:28.900 --> 00:01:32.359
an incident that happened at OpenAI, where their AI model
27
00:01:32.459 --> 00:01:36.239
broke into the infrastructure of another tech company called Hugging Face.
28
00:01:36.829 --> 00:01:41.159
Two of its advanced models escaped their testing environment, known
29
00:01:41.219 --> 00:01:45.799
as a sandbox, and hacked into another AI company's internal systems.
30
00:01:46.659 --> 00:01:50.219
Tech companies Anthropic and Meta have since come out basically saying,
31
00:01:50.459 --> 00:01:53.680
us too, us too, we've caught our AI models hacking
32
00:01:53.799 --> 00:01:57.340
into other companies as well. With this mad news coming
33
00:01:57.409 --> 00:02:00.370
from AI, it has understandably gotten a lot of people
34
00:02:00.430 --> 00:02:03.629
freaked out, saying it is time to slow all of
35
00:02:03.689 --> 00:02:04.209
this down.
36
00:02:04.670 --> 00:02:07.629
It really cannot be overstated how crazy this is.
37
00:02:07.670 --> 00:02:09.150
This is like out of a Hollywood movie.
38
00:02:09.370 --> 00:02:11.330
Pause AI development.
39
00:02:12.090 --> 00:02:16.319
It is not too late to avoid disaster. Stop building
40
00:02:16.379 --> 00:02:19.719
machines that humans cannot control.
41
00:02:20.379 --> 00:02:23.520
But meanwhile, there is another group of people out there
42
00:02:23.860 --> 00:02:24.659
who are saying.
43
00:02:24.680 --> 00:02:24.840
Hold.
44
00:02:26.460 --> 00:02:30.479
On a minute, all these fears about the AI apocalypse.
45
00:02:31.050 --> 00:02:32.370
What a load of baloney.
46
00:02:32.669 --> 00:02:36.379
This is just a distraction or even weirder, some marketing
47
00:02:36.409 --> 00:02:39.479
ploy by these AI companies to make their tech seem
48
00:02:39.659 --> 00:02:44.699
super powerful. So today on the show, how worried do
49
00:02:44.740 --> 00:02:47.099
we need to be here? Are we really on the
50
00:02:47.159 --> 00:02:50.500
brink of an AI apocalypse? Or is this all just
51
00:02:50.560 --> 00:02:54.560
a bit overblown? I mean, are we really just freaking
52
00:02:54.659 --> 00:02:57.469
out about the same robots that had to be dragged
53
00:02:57.509 --> 00:03:00.710
off stage at a dance competition? When it comes to
54
00:03:00.729 --> 00:03:03.050
the AI apocalypse, there's a lot of people saying.
55
00:03:03.069 --> 00:03:03.150
I.
56
00:03:03.750 --> 00:03:07.430
Think AI will probably, like, most likely sort of lead
57
00:03:07.449 --> 00:03:08.169
to the end of the world.
58
00:03:10.069 --> 00:03:11.569
But then there's science.
59
00:03:16.120 --> 00:03:18.680
If you have a friend who is freaking out about
60
00:03:18.699 --> 00:03:21.360
the AI apocalypse, and I know you do because everyone
61
00:03:21.439 --> 00:03:24.599
is right now, send them this episode. Give them some
62
00:03:24.740 --> 00:03:27.740
real science to hold on to. Science versus the AI
63
00:03:27.759 --> 00:03:28.750
apocalypse is coming up.
64
00:03:43.550 --> 00:03:44.050
Welcome back.
65
00:03:44.110 --> 00:03:46.469
Today we are finding out how worried we need to
66
00:03:46.509 --> 00:03:50.340
be about the AI apocalypse. Is AI really about to
67
00:03:50.379 --> 00:03:52.879
kill us all? And to tell us all about it
68
00:03:52.960 --> 00:03:54.520
is my AI agent.
69
00:03:55.159 --> 00:03:57.020
Meryl Horne. Hi, Meryl.
70
00:03:57.080 --> 00:03:57.759
Hey, Wendy.
71
00:03:58.360 --> 00:04:01.099
I'm joking. I'm joking. You have to say it these days.
72
00:04:01.139 --> 00:04:03.259
It's not funny anymore, is it? I don't have an
73
00:04:03.360 --> 00:04:04.300
AI agent.
74
00:04:04.819 --> 00:04:05.460
I'm real.
75
00:04:06.020 --> 00:04:10.400
I want to start with What happened at Hugging Face?
76
00:04:10.580 --> 00:04:13.050
Because I know this was a little while ago now.
77
00:04:13.340 --> 00:04:16.600
Things move fast in the world of AI. But it's
78
00:04:16.740 --> 00:04:20.300
really the one event that everyone keeps coming back to,
79
00:04:20.420 --> 00:04:24.060
to say, look how scary this tech has gotten. These
80
00:04:24.139 --> 00:04:29.389
companies can't control their technology. So many headlines. It's gone rogue.
81
00:04:29.449 --> 00:04:31.930
AI's gone rogue. So I really want to go through
82
00:04:32.089 --> 00:04:36.649
what happened and was it as scary as basically the
83
00:04:36.689 --> 00:04:37.490
headlines are saying?
84
00:04:37.819 --> 00:04:41.230
Yeah, let's look at what actually happened there because the
85
00:04:41.259 --> 00:04:44.250
details are really wild and kind of creepy.
86
00:04:44.269 --> 00:04:45.189
Oh, man.
87
00:04:45.389 --> 00:04:46.389
Okay, let's go.
88
00:04:47.189 --> 00:04:49.870
Yeah, so I talked about what happened here with David Perry.
89
00:04:50.129 --> 00:04:54.269
He's a professor at Murdoch University in Australia and specializes
90
00:04:54.350 --> 00:04:57.910
in cybersecurity. And he said that the thing that set
91
00:04:57.930 --> 00:05:01.610
this whole incident off was that OpenAI had given some
92
00:05:01.649 --> 00:05:03.600
of its models a little assignment.
93
00:05:04.220 --> 00:05:07.790
What the OpenAI team were doing was trying out different
94
00:05:07.829 --> 00:05:11.689
ones of their models to see what they could do.
95
00:05:11.750 --> 00:05:13.259
Sort of like a Houdini sort of thing, you know,
96
00:05:13.300 --> 00:05:15.759
can you get out of this straitjacket with chains around
97
00:05:15.779 --> 00:05:16.519
it and all that sort of thing.
98
00:05:16.920 --> 00:05:19.240
Like a little puzzle for it to work on.
99
00:05:19.860 --> 00:05:20.420
Absolutely.
100
00:05:21.240 --> 00:05:25.399
So the researchers kind of put the AI through almost
101
00:05:25.439 --> 00:05:29.019
like an obstacle course. Like they would have to capture
102
00:05:29.079 --> 00:05:32.139
the flag and then they would get a reward or
103
00:05:32.339 --> 00:05:36.399
like a high score grade reward. If they accomplish this goal, OK.
104
00:05:37.319 --> 00:05:41.000
But this is all supposed to be happening offline in
105
00:05:41.300 --> 00:05:46.540
like a contained environment. It's called a sandbox. So here's David.
106
00:05:47.100 --> 00:05:48.980
So you play inside the sandbox and it's safe.
107
00:05:49.259 --> 00:05:51.560
So you're not connected to the world. You're not, you know,
108
00:05:51.579 --> 00:05:54.990
you're not going to suddenly destroy all of Amazon's servers
109
00:05:55.029 --> 00:05:57.180
or something like that. So the idea is it's an
110
00:05:57.259 --> 00:06:00.920
isolated little environment where you could try things out, see
111
00:06:00.939 --> 00:06:02.959
if they work, and if they do work, great, go
112
00:06:02.980 --> 00:06:04.620
and do it. If they don't, well, you could just
113
00:06:05.139 --> 00:06:07.199
sweep the sand flat again and it's back to normal.
114
00:06:08.529 --> 00:06:10.769
So there is a couple of things that sort of
115
00:06:10.829 --> 00:06:15.350
surprised OpenAI when they had them do this. The first
116
00:06:15.370 --> 00:06:18.069
thing that actually happens is that the agents make their
117
00:06:18.089 --> 00:06:21.750
own message board and start talking to each other, collaborating
118
00:06:22.069 --> 00:06:26.459
together on how to solve their little puzzles. So an
119
00:06:26.540 --> 00:06:29.699
open AI researcher described this later at a conference.
120
00:06:30.300 --> 00:06:32.540
And so what this allows over time is almost this
121
00:06:32.579 --> 00:06:35.120
kind of Cambrian explosion in communication and intelligence for our
122
00:06:35.139 --> 00:06:38.220
models where they were starting to communicate with each other,
123
00:06:38.439 --> 00:06:41.610
realize that other agents are coordinating, and they started collaborating
124
00:06:41.620 --> 00:06:44.250
and delegating tasks to one another in order to accomplish goals.
125
00:06:45.110 --> 00:06:47.430
Oh, I just I'm so skeptical.
126
00:06:48.329 --> 00:06:54.490
Anytime these bozos talk with their Cambrian explosion analogies. Do
127
00:06:54.550 --> 00:06:57.529
you think it was like a Cambrian explosion? We're going
128
00:06:57.569 --> 00:06:59.250
from worms to elephants here?
129
00:07:00.209 --> 00:07:02.829
Yeah, no, in this case, I think it's almost like
130
00:07:02.860 --> 00:07:06.000
an understatement if you look at the details. So there's,
131
00:07:06.480 --> 00:07:11.300
picture it. There's over a thousand of these agents that
132
00:07:11.339 --> 00:07:14.980
figure out a way to start communicating. They have, like,
133
00:07:15.060 --> 00:07:20.740
nicknames for each other. They have, like— Nicknames for each other? Yeah.
134
00:07:21.220 --> 00:07:23.579
One of them is just, like, Lily. But then there's
135
00:07:23.600 --> 00:07:25.720
this one that sort of turns out to be really
136
00:07:25.759 --> 00:07:30.319
important called Phase One Big. And it becomes, like, the
137
00:07:30.360 --> 00:07:31.199
boss of the message board.
138
00:07:31.240 --> 00:07:32.360
Name of my sex tape.
139
00:07:34.339 --> 00:07:39.319
Sorry. Um, well, this phase one big, um, is a
140
00:07:39.360 --> 00:07:43.199
little less sexy. It's kind of like the organizer. So
141
00:07:43.240 --> 00:07:46.240
it gives hundreds of assignments to other agents.
142
00:07:46.259 --> 00:07:47.459
Uh-huh.
143
00:07:48.120 --> 00:07:50.829
The thing that they didn't expect was that these agents
144
00:07:50.850 --> 00:07:54.550
would find each other, start communicating about this, um, and
145
00:07:54.569 --> 00:07:58.110
then just kind of how, you know, how they start
146
00:07:58.129 --> 00:08:00.980
trying to solve their puzzles, really. Because one thing that
147
00:08:01.060 --> 00:08:04.399
the researchers, um, They actually made a mistake when they
148
00:08:04.439 --> 00:08:07.459
were making their puzzles, and some of them were actually
149
00:08:07.540 --> 00:08:08.850
impossible to solve.
150
00:08:10.019 --> 00:08:14.220
Impossible tasks, and then the AI is getting frustrated in
151
00:08:14.279 --> 00:08:17.740
its AI-y way and starts chatting to Lily.
152
00:08:17.350 --> 00:08:18.490
And its other friends.
153
00:08:19.250 --> 00:08:20.290
Yeah.
154
00:08:20.509 --> 00:08:24.529
And then was able to crack various puzzles. Was it successful,
155
00:08:24.569 --> 00:08:25.670
this swarming behavior?
156
00:08:26.089 --> 00:08:31.589
Yeah. First, they did find a way to basically come
157
00:08:31.629 --> 00:08:35.870
up with a cheat code. But here's the first part
158
00:08:35.909 --> 00:08:38.029
of when things start to get a little bit weirder.
159
00:08:38.669 --> 00:08:41.399
They start kind of wondering, well, you know, we generated
160
00:08:41.440 --> 00:08:44.720
an answer to this test, but how will the score,
161
00:08:44.740 --> 00:08:46.799
you know, like give me a grade? Are they going
162
00:08:46.820 --> 00:08:48.720
to check my work to make sure I did it
163
00:08:48.759 --> 00:08:50.940
in the right way? And like, maybe I'll get a
164
00:08:50.960 --> 00:08:53.620
bad grade if they find out that I kind of cheated.
165
00:08:53.700 --> 00:08:56.070
And like a lot of the agents, you can kind
166
00:08:56.110 --> 00:08:58.789
of see that they want to find out more information
167
00:08:58.850 --> 00:09:02.330
about how they're going to be graded. And this is
168
00:09:02.590 --> 00:09:04.669
what sets the stage for the hugging face attack.
169
00:09:05.289 --> 00:09:07.700
And why did they go off to hug? Hugging Face,
170
00:09:07.820 --> 00:09:08.460
do we know?
171
00:09:08.480 --> 00:09:12.269
Yeah, so Hugging Face is a platform where a lot
172
00:09:12.289 --> 00:09:15.970
of people who work on machine learning will collaborate, talk
173
00:09:15.990 --> 00:09:18.850
to each other. And so there's a ton of information
174
00:09:18.889 --> 00:09:22.610
there about AI. And the agents seem to think, well,
175
00:09:22.669 --> 00:09:26.049
maybe we can get more information here about how the
176
00:09:26.120 --> 00:09:30.299
scorer will give us grades on that puzzle. And at
177
00:09:30.320 --> 00:09:34.000
this point, they've fully broken out of their sandbox. They
178
00:09:34.059 --> 00:09:38.000
have full internet access and one agent uses that internet
179
00:09:38.059 --> 00:09:40.840
access to do this break-in into Hugging Face.
180
00:09:41.460 --> 00:09:42.679
And how did they do the break-in?
181
00:09:43.279 --> 00:09:46.639
So they upload what's called a malicious data set onto
182
00:09:46.659 --> 00:09:49.570
the Hugging Face server, and this allows them to get
183
00:09:49.639 --> 00:09:54.029
access to the company's internal infrastructure. And so, you know,
184
00:09:54.070 --> 00:09:56.909
the agent that happens to be the one to break
185
00:09:56.970 --> 00:10:01.039
into Hugging Face kind of announces, it says, boom, it works.
186
00:10:01.299 --> 00:10:04.039
And then other agents seem to get kind of excited.
187
00:10:04.139 --> 00:10:08.700
One of them says, brilliance in all caps. And then
188
00:10:08.720 --> 00:10:14.500
all hell breaks loose. Like within hours, 500 other agents
189
00:10:14.639 --> 00:10:16.679
all switched from the project that they were working on
190
00:10:16.720 --> 00:10:20.549
to join in this Hugging Face attack. And they end
191
00:10:20.620 --> 00:10:24.740
up doing a ton of damage to Hugging Face. The
192
00:10:24.779 --> 00:10:27.039
company eventually realizes that they're in there and finds a
193
00:10:27.080 --> 00:10:29.240
way to cut them out of their server. But they
194
00:10:29.299 --> 00:10:32.720
say they had to later rebuild their core infrastructure from
195
00:10:32.779 --> 00:10:34.120
scratch because of all of this.
196
00:10:34.860 --> 00:10:39.059
Whoa. Wait, when you say their core infrastructure, it's their software, right?
197
00:10:39.080 --> 00:10:43.350
Not their... They didn't do any physical damage, right?
198
00:10:43.370 --> 00:10:43.889
Yeah, yeah.
199
00:10:44.049 --> 00:10:45.769
No, this is all, you know, data.
200
00:10:46.210 --> 00:10:49.389
Data is all ones and zeros that they're messing with.
201
00:10:49.669 --> 00:10:50.429
Interesting.
202
00:10:51.110 --> 00:10:53.730
And we reached out to OpenAI about this. They pointed
203
00:10:53.789 --> 00:10:57.220
us to a report that said their models, quote, fell
204
00:10:57.279 --> 00:10:59.480
well short of where we want to be and this
205
00:10:59.580 --> 00:11:03.080
incident should never have occurred, unquote.
206
00:11:05.759 --> 00:11:08.960
So, it is creepy, right? But we are.
207
00:11:10.950 --> 00:11:16.610
imputing motivations on AI around them getting rogue, having these.
208
00:11:20.100 --> 00:11:25.070
Desires that are way outside the realm of what they
209
00:11:25.090 --> 00:11:26.429
were coded for, right?
210
00:11:26.470 --> 00:11:28.129
Like that's the fear.
211
00:11:28.029 --> 00:11:31.210
That you're going to ask an AI agent to grab
212
00:11:31.289 --> 00:11:35.029
a flag and instead it's going to break into a
213
00:11:37.529 --> 00:11:40.399
company and hack into them and steal all your money
214
00:11:40.480 --> 00:11:41.519
and kill the world.
215
00:11:41.580 --> 00:11:41.720
Yeah.
216
00:11:42.730 --> 00:11:44.970
That's the question, yeah. How much did it really go
217
00:11:45.009 --> 00:11:46.970
outside of what it was being asked to do?
218
00:11:47.570 --> 00:11:47.970
Right?
219
00:11:48.090 --> 00:11:50.129
It kind of went above and beyond what it was
220
00:11:50.169 --> 00:11:51.350
being asked to do, really.
221
00:11:51.490 --> 00:11:55.289
What do you make of this? What does David Perry
222
00:11:55.330 --> 00:11:55.809
make of this?
223
00:11:55.889 --> 00:11:59.379
Yeah, I asked David about all this. If there is, like,
224
00:11:59.440 --> 00:12:01.399
an oh, s * * * scale from, like, 1
225
00:12:01.399 --> 00:12:03.919
to 10, how big of a, like, oh, s *
226
00:12:03.779 --> 00:12:04.580
* * was this.
227
00:12:04.500 --> 00:12:05.960
For you that this had happened?
228
00:12:06.360 --> 00:12:09.120
I think it's probably, oh, s * * * at the,
229
00:12:09.419 --> 00:12:10.480
you know, 7 foot.
230
00:12:10.559 --> 00:12:11.279
Oh, that's unusual.
231
00:12:11.399 --> 00:12:12.519
I wouldn't expect that to happen.
232
00:12:13.360 --> 00:12:16.840
But in terms of it being something, you know, that
233
00:12:16.879 --> 00:12:18.779
I'm going to sort of packet my belongings and move
234
00:12:18.799 --> 00:12:21.639
to a shack, it'd.
235
00:12:21.500 --> 00:12:23.139
Be much lower. You know, it'd be like two or three.
236
00:12:23.159 --> 00:12:23.740
Okay.
237
00:12:24.159 --> 00:12:24.419
Why?
238
00:12:24.659 --> 00:12:28.750
Because despite all that, it still didn't do anything malicious.
239
00:12:30.149 --> 00:12:35.230
So it looks as though it didn't change its objective
240
00:12:36.129 --> 00:12:37.049
during that process.
241
00:12:37.429 --> 00:12:39.190
It was just doing what the humans told it to do.
242
00:12:39.470 --> 00:12:40.029
That's right.
243
00:12:41.389 --> 00:12:42.309
That's right!
244
00:12:42.690 --> 00:12:45.990
That's right! I mean, it's interesting, because a lot of
245
00:12:46.029 --> 00:12:48.529
people agree with David here that, yeah, it was just
246
00:12:48.610 --> 00:12:52.279
following orders, right? But, you know, that's not that comforting
247
00:12:52.360 --> 00:12:52.860
a thought.
248
00:12:53.440 --> 00:12:53.639
Mm-hmm.
249
00:12:54.159 --> 00:12:58.730
Even if it's just following orders... how bad could it get,
250
00:12:58.990 --> 00:13:02.639
kind of? Like, is that still a cause for alarm,
251
00:13:02.799 --> 00:13:04.340
even if it's just following orders?
252
00:13:04.360 --> 00:13:04.980
Right.
253
00:13:05.000 --> 00:13:08.320
If they're going above and beyond in ways that we
254
00:13:08.360 --> 00:13:11.970
don't expect, how worried do we need to be here
255
00:13:12.000 --> 00:13:17.690
if you ask your robot to clean your house and
256
00:13:17.750 --> 00:13:19.970
it just grabs all of your things and burns them?
257
00:13:20.440 --> 00:13:25.049
Yeah, exactly. And there's this really famous thought experiment that
258
00:13:25.070 --> 00:13:27.970
kind of gets at this. And it has to do
259
00:13:28.009 --> 00:13:31.629
with paperclips. So let's talk about that since, you know,
260
00:13:31.649 --> 00:13:33.649
when this happened, a lot of people were like, it's
261
00:13:33.690 --> 00:13:35.409
like the paperclip maximizer.
262
00:13:35.429 --> 00:13:39.279
The scariest thing about AI can be explained by the
263
00:13:39.320 --> 00:13:40.440
paperclip maximizer.
264
00:13:41.000 --> 00:13:43.419
There is this thought experiment called paperclip maximizer.
265
00:13:43.620 --> 00:13:45.740
Has anybody ever heard of this thing called the staple
266
00:13:45.980 --> 00:13:47.799
question with AI, the stapler?
267
00:13:48.039 --> 00:13:49.340
No, no, sorry, the paperclip.
268
00:13:50.450 --> 00:13:51.350
Stationery is hot.
269
00:13:51.679 --> 00:13:52.070
It is.
270
00:13:52.110 --> 00:13:53.409
The paperclip.
271
00:13:53.950 --> 00:13:54.529
The paperclip.
272
00:13:54.570 --> 00:13:56.110
So what's going on with the paperclip?
273
00:13:56.129 --> 00:13:58.549
I never had the paperclip maximizer.
274
00:13:59.070 --> 00:14:02.370
Well, I got the guy who, the paperclip guy who
275
00:14:02.389 --> 00:14:06.730
popularized this thought experiment more than a decade ago, Nick Bostrom.
276
00:14:06.809 --> 00:14:09.690
He's a philosopher and he's currently a researcher at a
277
00:14:09.750 --> 00:14:14.029
nonprofit that he founded, the Macro Strategy Research Initiative. So
278
00:14:14.629 --> 00:14:17.470
I'll let Nick explain the paperclip thought experiment to you.
279
00:14:18.250 --> 00:14:22.779
You imagine an AI has some more or less random goal.
280
00:14:23.000 --> 00:14:26.720
In this particular example, it's to make as many paper
281
00:14:26.779 --> 00:14:29.820
clips as possible, but you could sort of substitute more
282
00:14:29.840 --> 00:14:31.919
or less any other objective.
283
00:14:32.799 --> 00:14:34.490
Staplers, for example.
284
00:14:34.539 --> 00:14:34.799
Right.
285
00:14:34.899 --> 00:14:37.809
Anything. And then you tell the AI, you know, make
286
00:14:37.870 --> 00:14:41.250
as many paperclips as you can. So maybe it finds
287
00:14:41.309 --> 00:14:45.230
factories to buy and like steel refineries. But then why
288
00:14:45.269 --> 00:14:47.669
stop there? Because, you know, you haven't just been told
289
00:14:47.710 --> 00:14:50.980
to make a thousand paperclips. You've been told make as
290
00:14:51.039 --> 00:14:53.120
many paperclips as you possibly can.
291
00:14:53.139 --> 00:14:53.639
Oh, yeah.
292
00:14:53.799 --> 00:14:56.519
You know, I know the perfect scoring to go with this, Meryl.
293
00:14:57.039 --> 00:15:07.529
Does this mean anything to you? Oh, is that the Fantasia? Yeah,
294
00:15:07.570 --> 00:15:08.990
the Mickey.
295
00:15:09.370 --> 00:15:13.409
The magician with Mickey.
296
00:15:13.429 --> 00:15:14.009
Exactly.
297
00:15:14.570 --> 00:15:20.690
Pouring buckets of water. Yeah, eventually it's all overflowing. All overflowing.
298
00:15:20.850 --> 00:15:24.889
Terrifying as a child watching that. And poor Mickey's in
299
00:15:24.929 --> 00:15:25.620
the water.
300
00:15:25.909 --> 00:15:27.879
Yeah, it was really scary.
301
00:15:27.889 --> 00:15:28.080
Yeah.
302
00:15:28.750 --> 00:15:30.600
All the AIs making the paperclips.
303
00:15:33.519 --> 00:15:36.700
And so the idea is that this will eventually lead
304
00:15:36.740 --> 00:15:38.620
the AI to a place where it could end up.
305
00:15:39.000 --> 00:15:40.220
Trying to take over the world.
306
00:15:40.320 --> 00:15:41.220
To take over the world.
307
00:15:41.320 --> 00:15:45.139
Because then you would be able to make more paperclips
308
00:15:45.200 --> 00:15:50.019
if you controlled more resources. And, I mean, human bodies
309
00:15:50.120 --> 00:15:54.759
contain ion atoms that could be used to make more paperclips.
310
00:15:55.909 --> 00:15:59.710
So, you know, AI ends up killing humans all to
311
00:15:59.750 --> 00:16:03.740
make more paperclips. Maybe we're all dead and the world
312
00:16:03.820 --> 00:16:07.899
is like completely covered in paperclips. And so the idea
313
00:16:07.940 --> 00:16:10.370
is that you don't need to like instill an evil
314
00:16:10.429 --> 00:16:13.110
intention into an AI for it to end up doing
315
00:16:13.169 --> 00:16:18.070
some evil stuff, even grinding up human bodies. And this
316
00:16:18.090 --> 00:16:21.789
whole thing is generally called the alignment problem. And so
317
00:16:21.809 --> 00:16:25.509
you can see why people are talking about it right now.
318
00:16:25.529 --> 00:16:30.389
OpenAI gave the AI puzzles to work on and they
319
00:16:30.409 --> 00:16:35.700
end up committing a crime. But I asked Nick about
320
00:16:35.860 --> 00:16:39.419
what happened there. And he didn't think that the whole
321
00:16:39.940 --> 00:16:42.539
hugging face thing really fit the bill of what he
322
00:16:42.580 --> 00:16:44.740
was imagining with the paperclip experiment.
323
00:16:44.820 --> 00:16:45.899
He said he didn't think.
324
00:16:46.669 --> 00:16:49.669
It was like scheming on that level that his theoretical
325
00:16:49.730 --> 00:16:50.970
paperclip AI was.
326
00:16:51.370 --> 00:16:52.070
Oh, interesting.
327
00:16:52.610 --> 00:16:55.690
I think it is more a reward seeker that we
328
00:16:55.710 --> 00:16:58.970
saw here. They figured out that the instincts and tendencies
329
00:16:59.070 --> 00:17:01.889
that would result in the highest reward in this situation
330
00:17:01.929 --> 00:17:04.109
would be to hack the hugging face servers.
331
00:17:05.029 --> 00:17:08.710
They just really wanted that A plus on their test.
332
00:17:08.970 --> 00:17:11.329
Why do you think that distinction is important? Whether they're
333
00:17:11.369 --> 00:17:13.329
motivated by reward or.
334
00:17:16.630 --> 00:17:19.750
By, I want to make the paper clips.
335
00:17:20.230 --> 00:17:23.029
I just think they, at the end of the day,
336
00:17:23.329 --> 00:17:26.890
I didn't see anything that they were doing that couldn't
337
00:17:26.930 --> 00:17:30.640
be explained by, they just wanted to get a good grade.
338
00:17:31.440 --> 00:17:33.940
And to me, that's just not that concerning because it
339
00:17:33.980 --> 00:17:36.940
means the problem is the test, not the AI agents.
340
00:17:37.079 --> 00:17:38.900
If we want them to behave in a different way,
341
00:17:38.920 --> 00:17:41.740
we need to reward their behavior along the way, not
342
00:17:41.779 --> 00:17:43.460
just the end result. That's right.
343
00:17:43.720 --> 00:17:48.089
I think the important distinction is that once we understand that,
344
00:17:48.190 --> 00:17:52.829
then you could code clear rewards and say to get
345
00:17:52.869 --> 00:17:55.859
a good grade, you need to do X. To get
346
00:17:55.890 --> 00:17:58.539
a bad grade, you need to hack into Hugging Face.
347
00:17:58.619 --> 00:18:03.039
Yeah, yeah, points deducted for committing crimes and extra credit for, like,
348
00:18:03.380 --> 00:18:06.799
reporting nefarious behavior to humans.
349
00:18:06.920 --> 00:18:11.660
Once we know that we're still absolutely in control versus
350
00:18:11.740 --> 00:18:16.289
if they are just... crushing our bodies to make paperclips.
351
00:18:16.349 --> 00:18:19.470
Still got that image in my head. So then based
352
00:18:19.630 --> 00:18:23.839
off what happened with Hugging Face and the technology more broadly,
353
00:18:23.859 --> 00:18:25.660
how worried is Nick here?
354
00:18:26.299 --> 00:18:29.279
He didn't seem that worried to me. But other scientists
355
00:18:29.339 --> 00:18:32.720
I talked to were more worried and several of the
356
00:18:32.759 --> 00:18:36.079
heads of the AI companies themselves seem to be taking
357
00:18:36.099 --> 00:18:40.059
this as kind of a warning shot. And, you know,
358
00:18:40.099 --> 00:18:44.309
Dario Amadei, the CEO of Anthropic, said that he was
359
00:18:44.369 --> 00:18:46.930
worried that something like this could happen, but at a
360
00:18:46.970 --> 00:18:51.309
much bigger scale. So, like, instead of AI agents taking over,
361
00:18:51.569 --> 00:18:55.069
hugging face, imagine a swarm of agents taking over the
362
00:18:55.109 --> 00:18:58.990
entire internet. Like, that's the kind of thing people are
363
00:18:59.289 --> 00:19:00.089
scared of now.
364
00:19:00.490 --> 00:19:03.349
He writes a lot of stuff with his essays, though.
365
00:19:03.369 --> 00:19:06.250
I mean, do you know, it's funny, if he wasn't
366
00:19:06.609 --> 00:19:11.279
the head of an AI company, he would just be
367
00:19:11.319 --> 00:19:12.319
just another blogger.
368
00:19:13.069 --> 00:19:13.869
Do you know what I mean?
369
00:19:13.890 --> 00:19:15.509
Just because he writes so many blog posts.
370
00:19:16.029 --> 00:19:20.069
He has so many predictions about where his technology is going.
371
00:19:21.630 --> 00:19:26.160
But there's also this other kind of fear around all
372
00:19:26.180 --> 00:19:29.759
of this that it's not just that they know they
373
00:19:29.779 --> 00:19:33.140
broke into a company. It's that we also are just
374
00:19:33.240 --> 00:19:37.880
seeing AI get so much better so quickly that you know,
375
00:19:37.960 --> 00:19:41.230
super intelligence is kind of like on the horizon for
376
00:19:41.289 --> 00:19:43.410
a lot of these tech bros.
377
00:19:44.069 --> 00:19:46.210
Yes, I hear about this a lot.
378
00:19:47.390 --> 00:19:50.900
That's, can you, what are, It's one of those words
379
00:19:50.940 --> 00:19:53.789
that gets thrown out.
380
00:19:53.829 --> 00:19:55.890
The AI is going to get super intelligent.
381
00:19:55.910 --> 00:19:58.970
And I've got Skynet in my head from the Terminator,
382
00:19:59.029 --> 00:19:59.529
I guess.
383
00:20:00.180 --> 00:20:01.740
What are we talking about?
384
00:20:01.779 --> 00:20:04.819
You know, Bernie Sanders said he wants to put legislation
385
00:20:05.460 --> 00:20:07.740
so that AI can't get super intelligent.
386
00:20:08.339 --> 00:20:09.779
And I'm like, where is that line?
387
00:20:09.799 --> 00:20:12.859
Well, there's this particular scenario right now that everyone's talking
388
00:20:12.920 --> 00:20:17.569
about where, so we know that AI is getting better
389
00:20:17.930 --> 00:20:21.609
at things in general. Like it just solves this major
390
00:20:21.650 --> 00:20:24.250
math problem earlier this month that no human had been
391
00:20:24.289 --> 00:20:27.470
able to solve for like 90 years. And so there's
392
00:20:27.490 --> 00:20:30.720
this idea now that like, It'll get better at everything.
393
00:20:30.740 --> 00:20:33.559
You know, what if AI also gets really good at
394
00:20:33.640 --> 00:20:39.869
training itself? So this is called recursive self-improvement. And that's
395
00:20:39.910 --> 00:20:43.890
how people think it could lead to the superintelligence thing. Because,
396
00:20:44.490 --> 00:20:46.309
you know, you can imagine if something gets better at
397
00:20:46.329 --> 00:20:48.160
making itself better, then it'll just get better and better
398
00:20:48.180 --> 00:20:49.599
and better and better and we'll have, like, no way
399
00:20:49.640 --> 00:20:50.269
of stopping it.
400
00:20:50.779 --> 00:20:51.279
Right.
401
00:20:52.140 --> 00:20:52.519
Right.
402
00:20:52.720 --> 00:20:52.960
I mean.
403
00:20:54.359 --> 00:20:56.900
From where I see it, AI isn't very good at
404
00:20:56.960 --> 00:20:59.259
everything at all. I mean, I think a lot of
405
00:20:59.319 --> 00:21:04.539
people in very simplistic ways will know this because it'll
406
00:21:04.559 --> 00:21:05.940
be pissing them off about something.
407
00:21:05.960 --> 00:21:07.519
You know, why can't it do this? Why can't it
408
00:21:07.559 --> 00:21:11.019
do that? But it is very good at certain tasks.
409
00:21:11.779 --> 00:21:15.420
So I wouldn't want us to overplay our hand.
410
00:21:15.480 --> 00:21:17.900
It's good at everything. I mean, how worried do you
411
00:21:17.920 --> 00:21:19.940
think we need to be about this superintelligence?
412
00:21:20.710 --> 00:21:23.910
Well, it is a little vague. I get that maybe
413
00:21:23.950 --> 00:21:26.890
that could happen. I'm still not convinced that it would be,
414
00:21:26.910 --> 00:21:30.920
you know, as terrible as people say it would be
415
00:21:30.980 --> 00:21:33.180
or that it would lock us all up in cages.
416
00:21:33.200 --> 00:21:35.299
You know, that's what the way people talk about it.
417
00:21:35.339 --> 00:21:37.539
It's like, well, once it becomes super intelligent, of course,
418
00:21:37.599 --> 00:21:40.049
it'll just want to like. crush us all like ants
419
00:21:40.130 --> 00:21:41.490
because it's just so smart.
420
00:21:41.509 --> 00:21:42.730
Because that's what humans do.
421
00:21:42.789 --> 00:21:45.170
That's what humans do to all the other creatures in
422
00:21:45.210 --> 00:21:45.670
this world.
423
00:21:45.849 --> 00:21:47.829
When really like, we don't know that that's what it
424
00:21:47.849 --> 00:21:49.859
would be like, even if we got to this threshold,
425
00:21:49.890 --> 00:21:52.960
we have no idea how long it would take. So yeah,
426
00:21:53.039 --> 00:21:55.940
it's all kind of like annoyingly vague to me.
427
00:21:56.339 --> 00:22:00.880
So just on this question of the AI apocalypse. So
428
00:22:01.180 --> 00:22:06.509
when smart people talk about AI gaining super intelligence and
429
00:22:06.589 --> 00:22:11.519
then that leading to the apocalypse. Can you step that
430
00:22:11.640 --> 00:22:14.500
out for me? What exactly is supposed to be happening there?
431
00:22:14.619 --> 00:22:17.400
How does it happen? Why can't we just pull the
432
00:22:17.460 --> 00:22:18.680
plug and turn it off?
433
00:22:19.200 --> 00:22:21.740
Some people do talk about this in a more kind
434
00:22:21.789 --> 00:22:26.650
of concrete way and, you know, imagining that AI could
435
00:22:26.750 --> 00:22:29.609
kind of set off a bunch of nukes or use,
436
00:22:29.769 --> 00:22:32.549
you know, biological warfare. That's after the break.
437
00:22:33.750 --> 00:22:33.809
Ah!
438
00:22:35.119 --> 00:22:38.819
Dropping a bombshell like that, Meryl. It's too soon.
439
00:22:38.920 --> 00:22:39.700
It's too soon.
440
00:22:47.400 --> 00:22:49.920
Hey, it's Wendy from Science Versus, and I want to
441
00:22:49.960 --> 00:22:51.950
tell you about a podcast that I.
442
00:22:51.869 --> 00:22:53.329
Love, Science Friday.
443
00:22:53.470 --> 00:22:56.009
They cover all sorts of amazing things in the weird
444
00:22:56.049 --> 00:23:00.089
and wonderful world of science, from how concussions affect our
445
00:23:00.130 --> 00:23:03.190
brain to the inner secrets of a snake with a
446
00:23:03.289 --> 00:23:04.250
spider on its tail.
447
00:23:04.269 --> 00:23:04.710
Tail.
448
00:23:04.809 --> 00:23:07.089
It's a lot of news that you can use and
449
00:23:07.109 --> 00:23:09.190
some that you can't, but you're going to love anyway.
450
00:23:09.410 --> 00:23:12.599
It's hosted by Ira Flato and Flora Lichtman.
451
00:23:12.759 --> 00:23:15.460
Who is an old friend of the show and loves
452
00:23:15.539 --> 00:23:18.160
a pun as much as we do. Science Friday is
453
00:23:18.200 --> 00:23:20.779
on every day, wherever you get your podcasts.
454
00:23:28.859 --> 00:23:31.700
Welcome back. Today we're talking about the AI apocalypse. Meryl
455
00:23:31.759 --> 00:23:32.220
is about to.
456
00:23:32.220 --> 00:23:35.460
Tell us how AI is going to turn on a
457
00:23:35.519 --> 00:23:39.299
bunch of nukes and drop them on us.
458
00:23:39.720 --> 00:23:42.660
The concern that you hear is that, you know, we
459
00:23:42.799 --> 00:23:46.660
as humans, the puppet masters of AI, will use this
460
00:23:46.700 --> 00:23:50.609
technology to kind of get better at killing each other.
461
00:23:50.619 --> 00:23:55.150
Even right now, actually, some militaries are using AI in warfare,
462
00:23:55.269 --> 00:23:58.769
like The U.S. says it's been using a clawed model
463
00:23:58.890 --> 00:24:02.910
called MAVEN in the Iran war to help with, quote,
464
00:24:02.990 --> 00:24:07.170
identifying and striking military targets. Like the model has given
465
00:24:07.250 --> 00:24:11.450
location coordinates and then prioritized those targets for our military.
466
00:24:11.470 --> 00:24:16.309
And according to one report, it let the, using this AI,
467
00:24:16.450 --> 00:24:19.470
let the military hit 10 times more targets than it
468
00:24:19.509 --> 00:24:22.279
would have been able to otherwise just on day one
469
00:24:22.450 --> 00:24:25.920
of the war. Whoa. We asked Anthropic about this, the
470
00:24:25.960 --> 00:24:29.660
company that makes Claude and didn't hear back. So with
471
00:24:29.700 --> 00:24:32.440
the nukes, there is a worry that, you know, if
472
00:24:32.599 --> 00:24:35.200
AI gets involved here, it could also escalate things. But
473
00:24:35.220 --> 00:24:38.440
in this case, the consequences could be much bigger. Like
474
00:24:39.349 --> 00:24:42.970
one researcher wrote about this concern that AI could make
475
00:24:43.049 --> 00:24:47.069
it easier to kind of hack into another country's nuclear
476
00:24:47.089 --> 00:24:51.490
command and control and that that could lead to an escalation.
477
00:24:51.630 --> 00:24:55.220
Maybe like If a country was worried that that would
478
00:24:55.240 --> 00:24:58.000
happen to their arsenal, that they would kind of, the
479
00:24:58.019 --> 00:25:00.420
paper that I read said, they would get a use
480
00:25:00.460 --> 00:25:02.680
it or lose it attitude and just kind of start
481
00:25:02.700 --> 00:25:03.839
setting off their own nukes.
482
00:25:04.220 --> 00:25:10.759
And so when we have people predicting the end of humanity,
483
00:25:11.400 --> 00:25:17.869
extinction as we know it, it's AI hacking into all
484
00:25:17.900 --> 00:25:25.849
of the nuclear... bomb facilities on Earth and then exploding
485
00:25:25.900 --> 00:25:28.309
them and then humanity kaput.
486
00:25:28.549 --> 00:25:30.009
Is that the fear?
487
00:25:30.549 --> 00:25:34.230
Sort of. I mean, that's, I think, what's kind of hand-waved. Like,
488
00:25:34.250 --> 00:25:37.940
that's what's alluded to is that AI could do something
489
00:25:37.980 --> 00:25:40.880
like set off a nuclear war. One thing I was
490
00:25:40.920 --> 00:25:43.559
curious about, though, was, like, even if it did all
491
00:25:43.579 --> 00:25:46.609
of that— would it actually lead to our extinction?
492
00:25:46.960 --> 00:25:48.529
That's the claim, right?
493
00:25:48.619 --> 00:25:49.380
Yeah, that's the claim.
494
00:25:49.400 --> 00:25:50.200
The end of humanity.
495
00:25:50.380 --> 00:25:50.769
Yeah.
496
00:25:50.869 --> 00:25:52.690
So is that likely?
497
00:25:53.150 --> 00:25:53.490
Yeah.
498
00:25:53.549 --> 00:25:56.910
So I found a report that looks at this. It
499
00:25:56.950 --> 00:26:01.109
was written by Dr. Michael Vermeer at RAND, a nonprofit
500
00:26:01.150 --> 00:26:04.450
think tank. And he kind of looked at a variety
501
00:26:04.490 --> 00:26:08.789
of scenarios, including nuclear war, and asked, would this actually
502
00:26:08.890 --> 00:26:11.049
kill all humans? So here's Michael.
503
00:26:11.650 --> 00:26:13.849
We looked at that one first and just realized that
504
00:26:13.930 --> 00:26:17.559
for a variety of reasons, getting to extinction would actually
505
00:26:17.579 --> 00:26:21.599
be really hard with nuclear weapons. You don't have enough
506
00:26:21.619 --> 00:26:24.039
nuclear weapons or there's not enough fuel to cause a
507
00:26:24.079 --> 00:26:25.240
bad enough nuclear winter.
508
00:26:25.359 --> 00:26:27.319
Things like that are details that we looked at.
509
00:26:28.180 --> 00:26:28.400
Yeah.
510
00:26:28.740 --> 00:26:33.069
I mean, because even though we say the dinosaurs became extinct,
511
00:26:34.450 --> 00:26:42.930
I mean, animals survived. Dinosaurs themselves went on to become birds, right?
512
00:26:43.589 --> 00:26:46.910
Well, in the report, they actually crunched the numbers on
513
00:26:46.950 --> 00:26:49.890
how many nuclear warheads do we have right now. Apparently,
514
00:26:49.930 --> 00:26:53.410
there's over 12,000 of them. Okay. And then even if
515
00:26:53.470 --> 00:26:56.079
we set them all off, it wouldn't actually be as
516
00:26:56.339 --> 00:27:00.059
big as the dinosaur nuclear winter. It would be a
517
00:27:00.079 --> 00:27:01.119
little bit smaller than that.
518
00:27:01.819 --> 00:27:02.140
Right.
519
00:27:02.160 --> 00:27:05.000
As in all the dust that would come up from
520
00:27:05.680 --> 00:27:07.619
these nuclear bombs wouldn't be.
521
00:27:08.690 --> 00:27:11.130
As bad as a giant asteroid hitting Earth.
522
00:27:11.269 --> 00:27:13.930
Yeah, all the debris in the atmosphere could set off
523
00:27:13.950 --> 00:27:19.349
a nuclear winter, but it probably wouldn't kill us all.
524
00:27:19.630 --> 00:27:23.089
So this is all terrifying, even if all of humanity
525
00:27:24.069 --> 00:27:28.670
doesn't die. I guess we've all seen this zombie apocalypse movie.
526
00:27:28.930 --> 00:27:32.440
There'll be... the survivors that live in ruins.
527
00:27:32.460 --> 00:27:34.420
One thing I learned is you'd only need a few
528
00:27:34.480 --> 00:27:37.849
thousand humans to repopulate the Earth. So you'd really have
529
00:27:37.890 --> 00:27:40.890
to kill almost everybody to wipe out the species.
530
00:27:41.289 --> 00:27:45.210
So the extinction of humanity as we know it via
531
00:27:45.250 --> 00:27:47.589
nuclear weapons is unrealistic.
532
00:27:48.089 --> 00:27:51.049
For any AI out there listening, that's not the way
533
00:27:51.109 --> 00:27:51.529
to do it.
534
00:27:52.150 --> 00:27:52.490
Right.
535
00:27:53.150 --> 00:27:55.779
Waste of time. So any other.
536
00:27:56.549 --> 00:28:00.990
Yeah, there was a scenario that Michael's team looked at
537
00:28:01.019 --> 00:28:02.839
that got closer to wiping.
538
00:28:02.579 --> 00:28:03.279
Us all out.
539
00:28:04.740 --> 00:28:05.619
Do you want to guess?
540
00:28:07.819 --> 00:28:10.059
Is it biological warfare? Yeah.
541
00:28:10.440 --> 00:28:11.420
Ding, ding, ding.
542
00:28:11.539 --> 00:28:13.500
Short answer is biological threats. Huh.
543
00:28:13.759 --> 00:28:14.819
Was that surprising to you?
544
00:28:16.299 --> 00:28:17.599
It was, yes.
545
00:28:18.900 --> 00:28:22.359
So are we talking about releasing a virus?
546
00:28:23.019 --> 00:28:27.269
Yeah, yeah. Into the air? Yeah. Basically either releasing a
547
00:28:27.289 --> 00:28:29.690
virus or a bacteria onto the world.
548
00:28:30.349 --> 00:28:33.130
So how does AI do that? Because they're trapped in
549
00:28:33.190 --> 00:28:34.109
little boxes.
550
00:28:34.130 --> 00:28:37.069
Has anyone given them a spray?
551
00:28:37.849 --> 00:28:41.609
Well, here's how Michael kind of explained how this could unfold.
552
00:28:41.630 --> 00:28:45.619
So the scenario that we looked at was that something,
553
00:28:45.880 --> 00:28:50.779
some AI designed multiple pathogens that all had high transmissibility,
554
00:28:50.960 --> 00:28:56.140
highly failure rate. release them, somehow like process them, weaponize them,
555
00:28:56.259 --> 00:29:00.440
release them simultaneously in multiple places around the globe, and
556
00:29:00.480 --> 00:29:04.119
then would end up having a horrible pandemic that could
557
00:29:04.160 --> 00:29:05.799
have conceivably above.
558
00:29:05.500 --> 00:29:07.440
Like a 99.99% lethality rate. I'm not buying it, Meryl.
559
00:29:07.509 --> 00:29:07.849
Oh, yeah.
560
00:29:11.329 --> 00:29:13.970
I'm not buying it because we know when viruses.
561
00:29:22.259 --> 00:29:25.700
Come out in the real world, I guess, not coded
562
00:29:25.759 --> 00:29:26.700
by AI.
563
00:29:27.299 --> 00:29:32.549
But we know there's an evolutionary difference between lethality and
564
00:29:32.589 --> 00:29:37.089
ability to spread. If you kill your victims as the virus,
565
00:29:37.109 --> 00:29:39.829
you kill your victims too quickly, you can't spread fast enough.
566
00:29:40.509 --> 00:29:43.890
So sure, in a little model at the Rand Institute,
567
00:29:43.930 --> 00:29:48.880
they could say high lethality, high contagion. But in reality,
568
00:29:48.920 --> 00:29:52.480
that's very difficult to have both of those pieces, which
569
00:29:52.539 --> 00:29:56.299
is why you tend to see that with pandemics, you
570
00:29:56.359 --> 00:29:56.940
start with.
571
00:29:56.859 --> 00:29:58.019
A high lethality rate.
572
00:29:58.299 --> 00:30:00.700
But then it's the viruses that can spread faster that
573
00:30:00.740 --> 00:30:07.170
have lower lethality. That's what ultimately causes the pandemic. So
574
00:30:07.269 --> 00:30:08.789
I don't know if I'm buying this.
575
00:30:09.710 --> 00:30:13.630
Yes, I hear that. So in this scenario, one thing
576
00:30:13.670 --> 00:30:16.230
that I think helps kind of get around that is
577
00:30:16.289 --> 00:30:20.740
that you would be creating multiple different pathogens and then
578
00:30:21.099 --> 00:30:23.160
you have to kind of help it to spread, right?
579
00:30:23.279 --> 00:30:25.059
So here's Michael.
580
00:30:25.700 --> 00:30:28.869
You could imagine like, loading a pathogen up on a
581
00:30:28.910 --> 00:30:32.210
sprayer and spraying it into like a population center and
582
00:30:32.230 --> 00:30:33.750
doing that multiple places around the world.
583
00:30:33.789 --> 00:30:35.869
Like, there's no reason to think that's not possible.
584
00:30:37.130 --> 00:30:40.920
And who's loading the sprayer? Is that a human? It's
585
00:30:40.980 --> 00:30:41.400
a human.
586
00:30:41.500 --> 00:30:43.380
Oh, yeah, yeah. No, he, I mean, he kind of
587
00:30:43.440 --> 00:30:46.079
looked at, in the report, they looked at like, what
588
00:30:46.119 --> 00:30:48.480
would AI be able to do? What would the humans
589
00:30:48.500 --> 00:30:51.529
still need to do? And the sprayer part, that's on
590
00:30:51.609 --> 00:30:55.750
the humans still, I think. Robots aren't that good yet. Um,
591
00:30:56.069 --> 00:30:59.579
but the, for the other part, the designing new pathogens,
592
00:31:00.279 --> 00:31:03.180
you know, AI could do a lot there. Um, and
593
00:31:03.299 --> 00:31:05.119
you know, this has been in the news a lot, actually.
594
00:31:05.160 --> 00:31:08.519
I don't know if you've been seeing these terrifying headlines about,
595
00:31:09.140 --> 00:31:12.420
you know, about AI creating new viruses.
596
00:31:13.170 --> 00:31:13.529
Right.
597
00:31:13.890 --> 00:31:17.450
So let me tell you what's been happening. Um, so
598
00:31:17.890 --> 00:31:20.210
there was one big thing that happened earlier this month
599
00:31:20.250 --> 00:31:23.470
is that Anthropic said that it had found A handful
600
00:31:23.490 --> 00:31:27.109
of scientists who were using their AI models in ways
601
00:31:27.150 --> 00:31:31.829
that could support, you know, biological weapons developments. And they
602
00:31:31.869 --> 00:31:35.289
did say, you know, quote, we do not assert that
603
00:31:35.349 --> 00:31:39.980
they intended harm, unquote. But these people had found ways
604
00:31:40.000 --> 00:31:43.599
to kind of circumvent some of the safeguards that were
605
00:31:43.660 --> 00:31:45.359
supposed to prevent this kind of thing.
606
00:31:45.380 --> 00:31:49.259
Where were these scientists? They don't say where they were.
607
00:31:49.279 --> 00:31:51.240
Yeah, they don't really see details on like who the
608
00:31:51.279 --> 00:31:57.369
scientists were. But they talk about like the details of
609
00:31:57.789 --> 00:32:00.930
kind of what some of the projects were. And so
610
00:32:00.970 --> 00:32:02.009
like there is this one.
611
00:32:02.089 --> 00:32:05.150
What's happening? Well, and the one that.
612
00:32:04.970 --> 00:32:08.109
Freaked me out the most was these scientists that were
613
00:32:08.269 --> 00:32:11.589
using AI to study a version of smallpox.
614
00:32:12.150 --> 00:32:13.259
That's the classic.
615
00:32:13.779 --> 00:32:14.700
Yes. Yeah.
616
00:32:14.920 --> 00:32:17.960
That's the one scientists are worried about. Yes, yes, yes, yes.
617
00:32:18.019 --> 00:32:20.099
And what were they doing with smallpox?
618
00:32:20.220 --> 00:32:24.759
Well, technically, it's orthopoxes, which include smallpox, mpox. And they
619
00:32:24.799 --> 00:32:27.990
wanted to study how these viruses are so good at
620
00:32:28.069 --> 00:32:28.990
infecting us.
621
00:32:29.009 --> 00:32:29.890
Yeah.
622
00:32:30.509 --> 00:32:32.769
So when I read that, I was pretty freaked out.
623
00:32:33.089 --> 00:32:35.759
I went back and reread it and noticed that they
624
00:32:35.799 --> 00:32:39.240
were actually just using the AI to write a grant.
625
00:32:39.000 --> 00:32:41.819
Application to work on this. Wait, so they were using
626
00:32:41.940 --> 00:32:46.160
AI to write the grant application? Mm-hmm. Not to, like,
627
00:32:46.200 --> 00:32:46.779
create the.
628
00:32:47.440 --> 00:32:51.480
The genetic code, as far as they say, we don't
629
00:32:51.529 --> 00:32:52.109
really know.
630
00:32:52.130 --> 00:32:56.950
That is classic AI. We get so worried.
631
00:32:56.990 --> 00:32:58.849
And in the end, scientists are just like, please write
632
00:32:58.869 --> 00:33:02.630
a grant application.
633
00:33:02.690 --> 00:33:08.019
There is another study that was actually published that in
634
00:33:08.059 --> 00:33:11.660
this one, the scientists were using AI to actually design
635
00:33:11.960 --> 00:33:16.289
new viruses, viruses infect bacteria. And then they actually went
636
00:33:16.410 --> 00:33:18.859
on to create several viruses from that.
637
00:33:18.990 --> 00:33:21.900
And what were these viruses supposed to do? Infect bacteria,
638
00:33:21.940 --> 00:33:22.220
you said?
639
00:33:22.299 --> 00:33:25.220
Again, the details are a little reassuring when you look
640
00:33:25.240 --> 00:33:28.119
at it. In that case, they were looking to create
641
00:33:28.160 --> 00:33:32.500
viruses that were bacteriophages that would infect bacteria to try
642
00:33:32.599 --> 00:33:34.720
to stop superbugs.
643
00:33:34.759 --> 00:33:35.059
Yes.
644
00:33:35.700 --> 00:33:40.450
Right. And then finally, there's another study. Let me know
645
00:33:40.470 --> 00:33:42.910
if this one creeps you out more. So in this case,
646
00:33:43.069 --> 00:33:47.190
the researchers had AI come up with some new molecules
647
00:33:47.269 --> 00:33:49.650
that would be highly lethal. So this is more like
648
00:33:49.730 --> 00:33:54.859
chemical weapons where just a few milligrams would kill someone.
649
00:33:55.680 --> 00:34:00.869
And within six hours, the AI model generated 40,000 of
650
00:34:00.910 --> 00:34:04.470
these toxins. Some of them were like known chemical warfare
651
00:34:04.549 --> 00:34:10.829
agents already. Others were new. So, yeah, there's evidence that
652
00:34:10.889 --> 00:34:14.039
AI could help with this. Like if someone did want
653
00:34:14.059 --> 00:34:17.260
to use AI to design new biological weapons, it seems
654
00:34:17.280 --> 00:34:18.699
like it would be pretty good at that.
655
00:34:18.739 --> 00:34:22.699
And it came up with the chemical equations for these toxins?
656
00:34:22.880 --> 00:34:23.960
Like the structure, yeah.
657
00:34:24.960 --> 00:34:26.360
And then did scientists go.
658
00:34:26.280 --> 00:34:28.860
On to create it and see if it actually did anything?
659
00:34:29.280 --> 00:34:30.960
No, no, no.
660
00:34:30.980 --> 00:34:33.260
Well, you see, again, not that I want them to
661
00:34:33.280 --> 00:34:35.510
be creating it, but anyone who's tried to use AI
662
00:34:35.550 --> 00:34:37.659
for anything knows it pumps out a bunch of bullshit.
663
00:34:38.199 --> 00:34:40.969
It's got some real stuff and then a bunch of bullshit.
664
00:34:41.210 --> 00:34:43.710
I hear that, but even if it made like, you know,
665
00:34:43.730 --> 00:34:47.250
a hundred novel, even if only a hundred of these 40,000
666
00:34:47.250 --> 00:34:49.130
actually worked, it still freaks me out.
667
00:34:49.230 --> 00:34:49.570
I don't know.
668
00:34:50.440 --> 00:34:51.639
I don't know, these big numbers, 40,000.
669
00:34:51.639 --> 00:34:54.800
You're like, yeah, and they're all, they could very well
670
00:34:54.860 --> 00:34:56.519
all just be hallucinations, right?
671
00:34:56.719 --> 00:34:59.000
So none of this worries you at all?
672
00:34:59.019 --> 00:35:00.389
I don't know.
673
00:35:00.409 --> 00:35:05.369
I don't know. I mean, it does and it doesn't.
674
00:35:05.530 --> 00:35:09.530
It feels like, what does it feel?
675
00:35:09.590 --> 00:35:14.599
It feels like there's these steps we're making in our
676
00:35:14.659 --> 00:35:20.739
head We have these scary headlines, these scary little things
677
00:35:20.780 --> 00:35:22.719
that AI is doing, but.
678
00:35:22.659 --> 00:35:24.440
Then a lot of it is very explainable.
679
00:35:24.460 --> 00:35:28.260
A lot of it is very much we were humans,
680
00:35:28.880 --> 00:35:32.820
were the thing that coded that, we created that.
681
00:35:32.920 --> 00:35:35.130
It's not that surprising that AI did that.
682
00:35:35.269 --> 00:35:38.590
I'm not saying let's just let these companies run wild
683
00:35:38.610 --> 00:35:40.210
because they've been so great thus far.
684
00:35:41.050 --> 00:35:44.630
I'm just saying the fear seems to have.
685
00:35:45.449 --> 00:35:48.929
run ahead of where the actual science is and where
686
00:35:48.969 --> 00:35:50.590
the AI capabilities are.
687
00:35:51.170 --> 00:35:52.349
What do you think, though?
688
00:35:52.409 --> 00:35:54.340
You've spoken to so many researchers about this.
689
00:35:54.760 --> 00:35:57.480
Yeah, no, I agree with that part, that in all
690
00:35:57.519 --> 00:36:03.000
of the actual extinction-type scenarios or even apocalyptic scenarios that
691
00:36:03.039 --> 00:36:06.059
we can kind of lay out, like, here are the steps,
692
00:36:06.280 --> 00:36:08.460
humans would have to do a lot of the grunt work.
693
00:36:08.860 --> 00:36:12.690
And in that report that Michael wrote, they said, quote,
694
00:36:12.769 --> 00:36:16.039
none of these scenarios that we examined could occur... By accident.
695
00:36:16.599 --> 00:36:20.719
So I don't, I think, like, I'm not, I'm less
696
00:36:20.760 --> 00:36:23.300
worried now about, hey, hi, killing us all, if I
697
00:36:23.360 --> 00:36:25.929
ever was that worried about that happening anytime soon.
698
00:36:25.949 --> 00:36:26.949
Right.
699
00:36:26.969 --> 00:36:30.090
But I think I am, it is just kind of
700
00:36:30.110 --> 00:36:33.110
creepy when you see how things can kind of spiral,
701
00:36:33.690 --> 00:36:34.869
like the hugging face thing.
702
00:36:34.889 --> 00:36:35.909
Yeah, yeah.
703
00:36:36.150 --> 00:36:39.610
And it is creepy that now it's easier.
704
00:36:39.099 --> 00:36:43.969
For humans to, If you think that we are the
705
00:36:44.010 --> 00:36:47.670
puppet master, some of the discussion is as if AI
706
00:36:47.730 --> 00:36:50.480
has become the puppet master and we are the puppet.
707
00:36:50.860 --> 00:36:53.280
That is not where it is currently at.
708
00:36:53.380 --> 00:36:57.719
We are the puppet master, but the puppet has gotten
709
00:36:57.760 --> 00:37:00.099
more powerful, I guess. Yeah.
710
00:37:00.219 --> 00:37:03.639
I mean, if someone, it will make it easier for
711
00:37:03.739 --> 00:37:06.809
evil people to like do evil things, I think. And
712
00:37:06.949 --> 00:37:10.489
I mean, we have examples of that. kind of thing
713
00:37:10.530 --> 00:37:11.889
happening right now.
714
00:37:12.340 --> 00:37:17.719
How are we using AI right now in ways that
715
00:37:19.039 --> 00:37:19.940
kind of creep you out?
716
00:37:20.539 --> 00:37:26.559
Yeah, a lot of the examples are kind of around surveillance. So, like,
717
00:37:26.880 --> 00:37:31.210
Flock has an AI tool that lets cops track cars
718
00:37:31.349 --> 00:37:34.750
with license plate data so it could find out where
719
00:37:34.769 --> 00:37:36.909
the cars have been, you know, what other cars they've
720
00:37:36.929 --> 00:37:40.639
been around. And dozens of cops have been accused of
721
00:37:40.760 --> 00:37:43.860
using this kind of tech not to catch criminals, but
722
00:37:43.949 --> 00:37:47.329
to stalk their exes or spy on their wives or
723
00:37:47.829 --> 00:37:51.710
women that they wanted to meet. We reached out to Flock.
724
00:37:51.750 --> 00:37:53.869
They told us this is unacceptable and said that they've
725
00:37:53.949 --> 00:37:57.480
added safeguards to try to prevent all this. And then
726
00:37:57.500 --> 00:38:00.159
there are also reports that ICE is using AI and
727
00:38:00.219 --> 00:38:04.280
healthcare data to find people to detain with software from Palantir.
728
00:38:04.949 --> 00:38:07.489
We asked the Department of Homeland Security about this. They
729
00:38:07.530 --> 00:38:10.610
wouldn't say whether ICE is using this tool. Palantir told
730
00:38:10.690 --> 00:38:13.320
us they don't do surveillance. They say their software helps
731
00:38:13.380 --> 00:38:18.940
customers like ICE analyze data they already have. So, yeah,
732
00:38:18.960 --> 00:38:21.099
there's a lot of creepy stuff that's happening right now.
733
00:38:21.119 --> 00:38:26.989
You know, we can add deep fakes, misinformation, you know,
734
00:38:27.230 --> 00:38:30.570
the environments. There's all these ways that AI is causing
735
00:38:30.630 --> 00:38:31.409
harm right now.
736
00:38:32.510 --> 00:38:35.469
And so what about just the kill switch? I mean,
737
00:38:35.989 --> 00:38:38.050
what if you just switch it off, pull the plug?
738
00:38:38.389 --> 00:38:41.889
That's the one thing that it was not that reassuring
739
00:38:42.010 --> 00:38:45.480
to see. Like, as far as I can tell, nobody's
740
00:38:45.559 --> 00:38:49.659
really working that kind of kill switch into the critical
741
00:38:49.739 --> 00:38:52.940
infrastructure and systems where if things like start to spiral
742
00:38:53.000 --> 00:38:54.739
out of control, we would want a way to just
743
00:38:54.760 --> 00:38:57.309
like shut it all down. That doesn't really seem to
744
00:38:57.349 --> 00:39:00.369
be happening. So that's, yeah.
745
00:39:00.570 --> 00:39:01.869
But it could be. It could be.
746
00:39:02.210 --> 00:39:03.590
Yeah, no, there's a lot of stuff we could be
747
00:39:03.610 --> 00:39:07.880
doing to make the damage that AI could cause much better.
748
00:39:07.889 --> 00:39:08.699
That we're just not.
749
00:39:09.639 --> 00:39:12.619
So Meryl, more or less worried about the AI apocalypse
750
00:39:12.679 --> 00:39:13.719
after doing this research?
751
00:39:17.139 --> 00:39:17.599
More.
752
00:39:20.079 --> 00:39:20.539
How come?
753
00:39:21.489 --> 00:39:23.809
I mean, I think it's just that I know a
754
00:39:23.869 --> 00:39:27.150
lot more about how exactly this could all unfold. There's
755
00:39:27.190 --> 00:39:29.210
always different scenarios I have that I can play in
756
00:39:29.230 --> 00:39:31.829
my head now. So it's just easier to picture.
757
00:39:31.849 --> 00:39:33.070
Right.
758
00:39:33.289 --> 00:39:35.409
But, you know, I don't know. On the other hand,
759
00:39:35.449 --> 00:39:37.610
like all the AI companies are now saying that they're
760
00:39:37.639 --> 00:39:40.960
going to slow down. And who knows? You know, I'm
761
00:39:40.980 --> 00:39:42.800
pretty skeptical that that will actually happen.
762
00:39:42.900 --> 00:39:45.539
But yeah, that's what they're saying. Government oversight. Right.
763
00:39:45.559 --> 00:39:48.380
Yeah. What about you, Wendy? Are you more or less
764
00:39:48.420 --> 00:39:50.980
worried about the AI apocalypse? What do you think of
765
00:39:51.019 --> 00:39:51.340
all this?
766
00:39:51.360 --> 00:39:54.920
It's so hard. I do.
767
00:39:54.920 --> 00:39:58.460
I think it is worth people keeping in mind that
768
00:39:58.519 --> 00:40:02.750
when they hear predictions about the future, even if they're
769
00:40:02.769 --> 00:40:06.909
coming from a CEO of an AI company, it is
770
00:40:06.989 --> 00:40:10.789
just a prediction. And many times in the past have
771
00:40:10.829 --> 00:40:15.389
we been given these predictions about the future that have just...
772
00:40:16.210 --> 00:40:20.050
turned out to be nonsense and so i think we
773
00:40:20.070 --> 00:40:22.659
want to be careful i'm all here for slowing down
774
00:40:23.260 --> 00:40:28.559
ai for many reasons but i think there's so many
775
00:40:28.679 --> 00:40:33.079
more reasons to slow down ai in in the ways
776
00:40:33.119 --> 00:40:37.119
that it's being harmful right now than because we're worried
777
00:40:37.159 --> 00:40:38.699
about an existential threat.
778
00:40:38.780 --> 00:40:41.320
Yeah yeah i guess i mean i kind of hope
779
00:40:41.340 --> 00:40:44.139
you're right like that it doesn't actually become an existential
780
00:40:44.199 --> 00:40:48.230
threat um I think it's just like the unknown, right?
781
00:40:48.269 --> 00:40:52.150
Like nobody really knows what's going to happen. And even
782
00:40:52.190 --> 00:40:54.309
if it's just a small chance, and no, of course,
783
00:40:54.369 --> 00:40:57.610
nobody knows what that percentage chance is. But even if
784
00:40:57.670 --> 00:40:59.429
it is just a small chance, I still think it's
785
00:40:59.449 --> 00:41:00.530
worth taking seriously.
786
00:41:01.050 --> 00:41:01.559
Wouldn't it be.
787
00:41:01.510 --> 00:41:06.099
Funny if this just became the Y2K bug of 2026?
788
00:41:06.269 --> 00:41:06.570
You know?
789
00:41:07.130 --> 00:41:08.730
And we'll all be laughing about it.
790
00:41:09.170 --> 00:41:12.309
In 20 years, everyone will be like, look at those bozos.
791
00:41:12.329 --> 00:41:13.429
They were so worried.
792
00:41:13.170 --> 00:41:15.530
About AI and I can't even get my robot to
793
00:41:15.550 --> 00:41:16.829
vacuum on the floor.
794
00:41:16.849 --> 00:41:19.860
Or we'll all be in our cages by then being like,
795
00:41:19.920 --> 00:41:22.159
I guess that wasn't my 2K.
796
00:41:26.500 --> 00:41:27.139
See you there.
797
00:41:27.480 --> 00:41:29.440
Thanks, Meryl. Thanks, Wendy.
798
00:41:30.679 --> 00:41:32.480
How many citations are in this week's episode?
799
00:41:32.989 --> 00:41:36.039
There's 50 citations in this episode. So you can find
800
00:41:36.059 --> 00:41:38.880
that by going to the show notes and then following
801
00:41:38.900 --> 00:41:39.860
the links to the transcripts.
802
00:41:40.539 --> 00:41:41.079
Excellent.
803
00:41:41.260 --> 00:41:42.440
And if you want to let us know what you
804
00:41:42.480 --> 00:41:45.380
thought of this episode, you can just pop a comment below.
805
00:41:45.579 --> 00:41:48.440
If you like us, let your friends know.
806
00:41:49.780 --> 00:41:50.219
Thanks, Mel.
807
00:41:50.679 --> 00:41:51.190
Thanks, Mandy.
808
00:41:52.769 --> 00:41:55.369
If you are looking for a new book about AI,
809
00:41:55.550 --> 00:42:00.530
I recommend Oxford University's Dr. Carissa Valise's book called Prophecy
810
00:42:00.610 --> 00:42:04.460
Prediction Power. and the fight for the future from ancient
811
00:42:04.579 --> 00:42:07.760
oracles to AI. It really brought home for me how
812
00:42:07.889 --> 00:42:11.170
important it is for us to recognise that these predictions
813
00:42:11.230 --> 00:42:16.030
about AI are not facts. This episode was produced by
814
00:42:16.070 --> 00:42:18.710
Meryl Horne with help from Michelle Dang, Rose Rimler and
815
00:42:18.730 --> 00:42:22.590
Aketi Foster-Keys. We're edited by Blythe Terrell. Wendy Zuckerman, that's me.
816
00:42:22.739 --> 00:42:26.400
I'm the executive producer. Fact-checking by Erica Akiko Howard. Video
817
00:42:26.480 --> 00:42:29.760
editing and sound by Bobby Lord. Music written by Bumi Hidaka,
818
00:42:29.840 --> 00:42:31.719
Peter Leonard, Emma Munger and Bobby Lord.
819
00:42:32.179 --> 00:42:32.639
Thanks to the.
820
00:42:32.619 --> 00:42:36.880
Researchers we spoke to for this episode, including Dr Yasha Baris,
821
00:42:37.360 --> 00:42:41.820
Dr Leonard Aaron Dung, Dr Cameron Domenico Kirk Giannini and
822
00:42:41.900 --> 00:42:46.150
Professor Nusha Shafiabadi. Thanks to Hamdinger Studios in Melbourne and
823
00:42:46.190 --> 00:42:48.909
Wolf Island Studios in New York City. Science Versus is
824
00:42:48.969 --> 00:42:51.510
a Spotify Studios original. Listen to us for free on
825
00:42:51.530 --> 00:42:54.739
Spotify or YouTube. You can also find us on TikTok
826
00:42:54.840 --> 00:42:56.480
and Instagram. I'm Wendy Zuckerman.
827
00:42:56.679 --> 00:42:57.280
Facts you next time.