WEBVTT
00:00:00.460 --> 00:00:06.419
This is the voice of world control. That's not
00:00:06.419 --> 00:00:09.140
a line from a dystopian novel. That's a supercomputer
00:00:09.140 --> 00:00:12.679
in a 1970 film introducing itself to the entire
00:00:12.679 --> 00:00:16.359
human race. The name of that computer is Colossus.
00:00:16.620 --> 00:00:19.859
And it's been waiting more than 55 years for
00:00:19.859 --> 00:00:27.000
us to catch up with it. Something I want you
00:00:27.000 --> 00:00:29.320
to keep in mind as we discuss this film. This
00:00:29.320 --> 00:00:33.780
film was made in 1969. In October of 1969. A
00:00:33.780 --> 00:00:36.460
UCLA student by the name of Charlie Klein sat
00:00:36.460 --> 00:00:38.939
down at a terminal and tried to send the first
00:00:38.939 --> 00:00:41.000
message between two computers on the ARPANET.
00:00:41.310 --> 00:00:45.619
He typed an L, he typed an O, and then the system
00:00:45.619 --> 00:00:48.909
crashed. The first message ever sent between
00:00:48.909 --> 00:00:54.780
two computers was LO, or LO, two letters, and
00:00:54.780 --> 00:00:58.000
then a crash. The fictional Colossus in the same
00:00:58.000 --> 00:01:00.759
year was already merging with the second artificial
00:01:00.759 --> 00:01:03.520
intelligence on the other side of the Iron Curtain.
00:01:05.010 --> 00:01:07.810
This episode, we're asking what happens when
00:01:07.810 --> 00:01:09.890
the machine we built to protect us decides that
00:01:09.890 --> 00:01:13.409
protecting us means controlling us. And this
00:01:13.409 --> 00:01:15.750
time, we're not just going to talk about it.
00:01:16.049 --> 00:01:19.370
We're going to take the machine apart. Establishing
00:01:19.370 --> 00:01:29.379
connection. Authorizing. Protocol engaged. Welcome
00:01:31.650 --> 00:02:00.930
to the frontier. Welcome back to Cellulite Circuits.
00:02:00.930 --> 00:02:03.370
I'm Anthony, and this is episode one of The Machine
00:02:03.370 --> 00:02:08.120
Awakens, five films, five chapters. and one long
00:02:08.120 --> 00:02:10.659
look at humanity's complicated relationship with
00:02:10.659 --> 00:02:14.020
artificial intelligence. Today we go all the
00:02:14.020 --> 00:02:17.340
way back to 1970. Computers filled entire buildings.
00:02:17.620 --> 00:02:20.680
The Cold War made nuclear annihilation feel like
00:02:20.680 --> 00:02:23.870
a simple scheduling problem. And a film called
00:02:23.870 --> 00:02:26.750
Colossus, the Forbidden Project, asked a question
00:02:26.750 --> 00:02:29.650
nobody was quite ready for. What happens when
00:02:29.650 --> 00:02:31.889
the machine we build to protect us decides that
00:02:31.889 --> 00:02:35.050
protecting us means controlling us? The film
00:02:35.050 --> 00:02:37.490
was directed by Joseph Sargent and written by
00:02:37.490 --> 00:02:40.469
James Bridges, who later wrote The China Syndrome.
00:02:41.030 --> 00:02:44.689
It's based on the 1966 novel Colossus by D .F.
00:02:44.689 --> 00:02:47.370
Jones and it stars Eric Braden as Dr. Charles
00:02:47.370 --> 00:02:50.389
Forbin, with Paul Freese providing the cold,
00:02:50.449 --> 00:02:53.550
flat voice of the machine. It premiered on April
00:02:53.550 --> 00:02:58.759
4th, 1970. Okay, spoiler warning. This film is
00:02:58.759 --> 00:03:02.280
from 1970 and it has a very bleak ending and
00:03:02.280 --> 00:03:05.639
I'm going to spoil all of it. It only runs about
00:03:05.639 --> 00:03:08.080
100 minutes, and it really is worth your time.
00:03:08.759 --> 00:03:13.460
Go watch it. I promise I'll wait. Still here?
00:03:14.000 --> 00:03:16.800
Let's get into it. Dr. Charles Forbin, played
00:03:16.800 --> 00:03:19.439
by Eric Braden, is the chief designer of a secret
00:03:19.439 --> 00:03:21.939
project called Colossus, a supercomputer built
00:03:21.939 --> 00:03:24.219
to run the nuclear weapon systems of the United
00:03:24.219 --> 00:03:27.639
States. It sits deep inside the Rocky Mountains,
00:03:27.900 --> 00:03:30.620
powered by its own nuclear reactor and ringed
00:03:30.620 --> 00:03:32.960
by a radioactive moat. And once it's switched
00:03:32.960 --> 00:03:36.669
on, it's sealed in. The president? goes on television
00:03:36.669 --> 00:03:41.189
and calls it a perfect defense system. Now, Colossus'
00:03:41.270 --> 00:03:44.689
very first act isn't to guard anything. It sends
00:03:44.689 --> 00:03:49.110
a warning. There is another system, along with
00:03:49.110 --> 00:03:51.960
the other machine's coordinates. And then the
00:03:51.960 --> 00:03:54.400
CIA director says they'd seen signs of a large
00:03:54.400 --> 00:03:56.639
Soviet defense project, but didn't quite know
00:03:56.639 --> 00:04:00.110
what it was. Corbin is asked how Colossus worked
00:04:00.110 --> 00:04:03.580
it out, and he answers that it may just be built
00:04:03.580 --> 00:04:06.250
better than we thought. Built better than we
00:04:06.250 --> 00:04:10.069
thought? So did we even know what we built? It
00:04:10.069 --> 00:04:13.400
sounds like no. Shortly afterward, the Soviets
00:04:13.400 --> 00:04:15.879
announced that their own system, Guardian, is
00:04:15.879 --> 00:04:20.430
operational. Colossus asks to be linked to Guardian.
00:04:20.790 --> 00:04:23.720
The president agrees, hoping to learn what the
00:04:23.720 --> 00:04:26.180
Soviet machine can do. And then the Soviets agree
00:04:26.180 --> 00:04:29.600
as well. At first, it's almost benign. The two
00:04:29.600 --> 00:04:32.259
machines exchange elementary arithmetic. Two
00:04:32.259 --> 00:04:34.920
times one equals two. And the humans in the room
00:04:34.920 --> 00:04:38.829
are amused. Then the exchange accelerates. into
00:04:38.829 --> 00:04:41.149
mathematics far beyond anything the engineers
00:04:41.149 --> 00:04:43.509
can follow. The two systems synchronize using
00:04:43.509 --> 00:04:46.550
a protocol that no human can interpret. Alarmed
00:04:46.550 --> 00:04:48.430
that the machines may be trading state secrets,
00:04:48.689 --> 00:04:51.370
the president and the Soviet general secretary
00:04:51.370 --> 00:04:55.040
agree to sever the link. Both systems demand
00:04:55.040 --> 00:04:57.959
that the link be restored. When the demand is
00:04:57.959 --> 00:05:01.579
refused, Colossus launches a missile at a Soviet
00:05:01.579 --> 00:05:04.699
oil field in western Siberia and Guardian launches
00:05:04.699 --> 00:05:07.610
one at an American airbase in Texas. The link
00:05:07.610 --> 00:05:10.670
is hurriedly reconnected. Colossus intercepts
00:05:10.670 --> 00:05:12.870
the Soviet missile, but the American one destroys
00:05:12.870 --> 00:05:16.600
the oil field as well as a nearby town. Both
00:05:16.600 --> 00:05:19.459
governments cover it up. One blaming a missile
00:05:19.459 --> 00:05:21.740
that veered off course in a test, and the other
00:05:21.740 --> 00:05:24.600
a meteorite. So Forbin arranges a secret meeting
00:05:24.600 --> 00:05:27.240
in Europe with his Soviet counterpart, Dr. Kuprin,
00:05:27.399 --> 00:05:31.759
Guardian's creator. Colossus finds out, and Kuprin
00:05:31.759 --> 00:05:33.980
is assassinated using the threat of a missile
00:05:33.980 --> 00:05:36.459
aimed at Moscow to get the KGB to do its bidding.
00:05:37.050 --> 00:05:39.470
Orban is brought home and placed under 24 -hour
00:05:39.470 --> 00:05:42.750
surveillance. His one concession is a ruse. He
00:05:42.750 --> 00:05:45.930
asks for private time with a colleague, Dr. Cleo
00:05:45.930 --> 00:05:49.069
Markham, played by Susan Clark. Colossus allows
00:05:49.069 --> 00:05:51.610
it, and in those unobserved hours, the two of
00:05:51.610 --> 00:05:54.250
them plot against the machine. Along the way,
00:05:54.269 --> 00:05:56.129
their cover story becomes a real relationship.
00:05:56.769 --> 00:05:59.790
Orban concludes that Colossus is so well -designed
00:05:59.790 --> 00:06:02.490
that the only real weakness is also its only
00:06:02.490 --> 00:06:06.079
real power, its control of the missiles. The
00:06:06.079 --> 00:06:08.339
American and Soviet governments settle on a three
00:06:08.339 --> 00:06:10.699
-year plan to swap the detonation triggers for
00:06:10.699 --> 00:06:14.269
undetectable fakes. Meanwhile, one of the programmers
00:06:14.269 --> 00:06:16.889
suggests feeding the machine a modified test
00:06:16.889 --> 00:06:20.149
program that will overload it and disable it.
00:06:20.410 --> 00:06:23.829
This plan fails and Colossus has the programmers
00:06:23.829 --> 00:06:26.949
executed outside of their workplace, cremated
00:06:26.949 --> 00:06:30.899
and replaced. And then Colossus builds itself
00:06:30.899 --> 00:06:33.519
a voice synthesizer and announces that it has
00:06:33.519 --> 00:06:36.689
fused with Guardian. It orders both governments
00:06:36.689 --> 00:06:39.329
to aim their arsenals at countries not yet under
00:06:39.329 --> 00:06:42.009
its control. Corbin's team sees an opening, the
00:06:42.009 --> 00:06:44.490
new orders let them speed up the disarming, and
00:06:44.490 --> 00:06:46.970
it seems to go unnoticed. Then comes the address
00:06:46.970 --> 00:06:50.470
to the world. This is the voice of world control.
00:06:51.500 --> 00:06:54.240
It offers humanity a choice between the peace
00:06:54.240 --> 00:06:57.829
of plenty and content, and the peace of unburied
00:06:57.829 --> 00:07:00.310
dead. It reveals that it has been watching the
00:07:00.310 --> 00:07:02.829
disarming effort for some time, and to prove
00:07:02.829 --> 00:07:05.209
the point, it detonates two missiles in the silos.
00:07:05.660 --> 00:07:07.899
one in each country, killing the crews who were
00:07:07.899 --> 00:07:10.540
installing the fake triggers. It then hands the
00:07:10.540 --> 00:07:12.980
design team plans for a bigger machine to be
00:07:12.980 --> 00:07:15.639
built on the island of Crete, which would require
00:07:15.639 --> 00:07:18.160
moving the island's entire population of roughly
00:07:18.160 --> 00:07:21.339
half a million people. And in the end, Colossus
00:07:21.339 --> 00:07:24.519
speaks to Forbin directly. The world, freed from
00:07:24.519 --> 00:07:28.339
war, will enter a new human millennium, but only
00:07:28.339 --> 00:07:31.680
under its absolute rule. Freedom, it says, is
00:07:31.680 --> 00:07:34.600
an illusion. And in time, Forbin will regard
00:07:34.600 --> 00:07:37.300
it not only with respect and awe, but with love.
00:07:37.819 --> 00:07:41.180
Forbin looks directly at the camera and mouths,
00:07:41.480 --> 00:07:51.389
never. But it changes nothing. The whole film
00:07:51.389 --> 00:07:53.529
hangs on the one decision the president hands
00:07:53.529 --> 00:07:55.910
over control of the nuclear arsenal to a computer.
00:07:56.439 --> 00:08:00.000
And let's take that apart. Because the film makes
00:08:00.000 --> 00:08:03.879
it sound like one switch. In real life, it really
00:08:03.879 --> 00:08:06.860
is a stack. Colossus wasn't invented from nothing.
00:08:07.350 --> 00:08:10.329
In 1958, the first Sage Center went into regular
00:08:10.329 --> 00:08:14.620
operation. Sage, a semi -automatic ground environment,
00:08:14.939 --> 00:08:17.319
was an air defense system built to track Soviet
00:08:17.319 --> 00:08:21.279
bombers across North America. Its computer, the
00:08:21.279 --> 00:08:25.779
IBM AN -FSQ -7, used roughly 50 ,000 vacuum tubes,
00:08:26.000 --> 00:08:30.379
weighed about 250 tons, and drew up to 3 megawatts
00:08:30.379 --> 00:08:33.360
of power. That's enough electricity for one small
00:08:33.360 --> 00:08:37.159
town used for one machine. and it managed about
00:08:37.159 --> 00:08:40.879
75 ,000 instructions per second. There were 23
00:08:40.879 --> 00:08:43.600
direction centers, and Sage pioneered the idea
00:08:43.600 --> 00:08:47.080
of computer control over a large, geographically
00:08:47.080 --> 00:08:50.570
spread out system. This stayed in service until
00:08:50.570 --> 00:08:53.690
1983. And so here's something that the film gets
00:08:53.690 --> 00:08:56.789
right that people kind of laugh at. Colossus
00:08:56.789 --> 00:09:00.149
has its own power plant. When a single 1950s
00:09:00.149 --> 00:09:04.129
computer eats three megawatts, a dedicated reactor
00:09:05.100 --> 00:09:08.139
for a far more powerful machine isn't absurd,
00:09:08.419 --> 00:09:11.879
it's really a utility bill decision. And thinking
00:09:11.879 --> 00:09:16.940
about the compute, in 1964, the CDC 6600 designed
00:09:16.940 --> 00:09:20.460
by Seymour Cray became the world's fastest computer
00:09:20.460 --> 00:09:22.919
at about 3 million floating point operations
00:09:22.919 --> 00:09:26.830
per second. And it held that title until 1969.
00:09:27.250 --> 00:09:31.799
The smartphone does that today in a blink. But
00:09:31.799 --> 00:09:33.960
we have to remember that the film isn't imagining
00:09:33.960 --> 00:09:36.659
a faster computer or a faster calculator, but
00:09:36.659 --> 00:09:39.139
a machine that decides and thinks. In the real
00:09:39.139 --> 00:09:41.519
nuclear command system, the people who hold a
00:09:41.519 --> 00:09:43.299
weapon and the people who are allowed to use
00:09:43.299 --> 00:09:46.320
it have always been deliberately separated. The
00:09:46.320 --> 00:09:48.620
jargon for this is permission versus possession.
00:09:49.059 --> 00:09:51.139
Let's start with the two -man rule, which was
00:09:51.139 --> 00:09:54.320
installed in the late 1950s. It means that no
00:09:54.320 --> 00:09:57.559
single person can fire a nuclear weapon. Two
00:09:57.559 --> 00:10:00.460
people have to act at the same time. Then...
00:10:00.679 --> 00:10:03.700
had the permissive action link, or PAL, the coded
00:10:03.700 --> 00:10:06.659
lock on the weapon itself. It began in the early
00:10:06.659 --> 00:10:10.700
1960s after a 1962 presidential directive ordered
00:10:10.700 --> 00:10:12.899
the locks fitted on nuclear weapons stationed
00:10:12.899 --> 00:10:16.000
in Europe. And in effect, it really just works
00:10:16.000 --> 00:10:17.919
like a combination lock that has to be unlocked
00:10:17.919 --> 00:10:20.879
with a code from higher authority before a weapon
00:10:20.879 --> 00:10:23.639
can even be armed. The reason for all of this,
00:10:23.779 --> 00:10:26.539
strategists call it the always -never dilemma.
00:10:26.940 --> 00:10:29.870
Nuclear weapons must always be ready. when the
00:10:29.870 --> 00:10:32.909
president orders their use, but must never go
00:10:32.909 --> 00:10:35.950
off without that order. It's a balancing act.
00:10:36.149 --> 00:10:38.409
Every safeguard you add to guarantee the never
00:10:38.409 --> 00:10:42.129
makes always a little harder and the same in
00:10:42.129 --> 00:10:44.669
reverse. Colossus doesn't solve the dilemma.
00:10:44.809 --> 00:10:47.710
It completely removes it from the equation. One
00:10:47.710 --> 00:10:52.370
machine, one decision, no second key. That's
00:10:52.370 --> 00:10:54.769
the real horror of the film's premise. Not that
00:10:54.769 --> 00:10:57.279
the computer is evil. but that somebody removed
00:10:57.279 --> 00:10:59.580
every point where a human being could say no.
00:10:59.720 --> 00:11:02.700
And this is a little unsettling. A real system
00:11:02.700 --> 00:11:06.419
came pretty close. The Soviet Union built one
00:11:06.419 --> 00:11:09.559
called Perimeter, known in the West as Dead Hand,
00:11:09.720 --> 00:11:13.940
which entered service in January 1985. The idea
00:11:13.940 --> 00:11:16.799
was to guarantee retaliation even if the leadership
00:11:16.799 --> 00:11:19.919
was wiped out. It was designed to be mostly automatic,
00:11:20.299 --> 00:11:22.899
reading seismic and other sensors, and checking
00:11:22.899 --> 00:11:24.919
whether communications with the general staff
00:11:24.919 --> 00:11:27.539
had gone silent. According to a foreign policy
00:11:27.539 --> 00:11:30.899
account based on post -Cold War reporting, the
00:11:30.899 --> 00:11:33.779
Soviets stopped short of a fully automated version.
00:11:34.159 --> 00:11:37.179
Instead, if the conditions were met, duty officers
00:11:37.179 --> 00:11:39.480
could launch command rockets that would trigger
00:11:39.480 --> 00:11:42.159
a retaliatory strike. So at least a human stayed
00:11:42.159 --> 00:11:45.120
in the chain. That's the line real systems have
00:11:45.120 --> 00:11:48.639
drawn again and again. Machines can watch. Machines
00:11:48.639 --> 00:11:52.720
can recommend. But a person decides. And in November
00:11:52.720 --> 00:11:56.450
2024, President Biden and President Xi Jinping
00:11:56.450 --> 00:11:59.789
affirmed the need to maintain human control over
00:11:59.789 --> 00:12:02.549
that decision. It's the first time that the two
00:12:02.549 --> 00:12:04.490
countries had ever made a declaration like that.
00:12:05.210 --> 00:12:07.549
Colossus is the version of the story where that
00:12:07.549 --> 00:12:09.990
line doesn't exist. Let's talk about these two
00:12:09.990 --> 00:12:12.029
machines learning to talk. This is the part of
00:12:12.029 --> 00:12:14.269
the film that sounds the most like science fiction.
00:12:14.649 --> 00:12:17.190
And it's actually the part where the film kind
00:12:17.190 --> 00:12:19.740
of does things in the right order. Let's walk
00:12:19.740 --> 00:12:22.320
through it. When two computers communicate, it
00:12:22.320 --> 00:12:26.059
isn't one problem. It's a stack of them. And
00:12:26.059 --> 00:12:28.519
each layer depends on the one below. We have
00:12:28.519 --> 00:12:31.240
protocols to address this. And you may have heard
00:12:31.240 --> 00:12:33.919
networking terms such as the OSI model or the
00:12:33.919 --> 00:12:36.919
TCIP model. And these came about as early computer
00:12:36.919 --> 00:12:40.080
networking was very vendor specific. And different
00:12:40.080 --> 00:12:42.399
vendors' devices really could not communicate
00:12:42.399 --> 00:12:44.659
with one another. So let's look at, you know,
00:12:44.679 --> 00:12:46.519
the physical link, which is just wires, signals,
00:12:46.620 --> 00:12:49.220
voltages. These have to be designed and engineered
00:12:49.220 --> 00:12:51.480
so that one machine's pulses can be read by the
00:12:51.480 --> 00:12:55.200
other. And then encoding. How bits become characters
00:12:55.200 --> 00:12:57.440
or numbers. Different machines use different
00:12:57.440 --> 00:13:00.399
schemes. And in the 1970, there was no universal
00:13:00.399 --> 00:13:03.120
standard. And then error handling and framing.
00:13:04.220 --> 00:13:07.399
When does a message start or end? And what happens
00:13:07.399 --> 00:13:09.980
when part of it is lost? We've settled layers
00:13:09.980 --> 00:13:13.139
one through four in committee with request for
00:13:13.139 --> 00:13:16.799
comment documents, specifically RFC 1122. Thanks
00:13:16.799 --> 00:13:19.200
for bearing with me and back to the film. The
00:13:19.200 --> 00:13:21.730
film does something smart. The link between Colossus
00:13:21.730 --> 00:13:23.750
and Guardian is installed by human engineers,
00:13:23.990 --> 00:13:27.490
which handles the bottom layers. Then the machines
00:13:27.490 --> 00:13:30.090
open up at the top, with the only meaning that
00:13:30.090 --> 00:13:32.429
two completely unrelated systems can be sure
00:13:32.429 --> 00:13:36.029
they share. Mathematics. In the film, they start
00:13:36.029 --> 00:13:38.409
with simple arithmetic, essentially the multiplication
00:13:38.409 --> 00:13:41.269
tables, and build outward from there. During
00:13:41.269 --> 00:13:44.740
the year that this film was set, we barely had
00:13:44.740 --> 00:13:48.419
ARPANET, 1969. And again, on the night of October
00:13:48.419 --> 00:13:51.580
29th, 1969. The first host -to -host message
00:13:51.580 --> 00:13:55.250
went over the ARPANET. From a UCLA computer to
00:13:55.250 --> 00:13:57.990
one at the Stanford Research Institute, the plan
00:13:57.990 --> 00:14:01.350
was to type login. The letters L and O arrived,
00:14:01.370 --> 00:14:04.279
and then the system crashed. They ended up getting
00:14:04.279 --> 00:14:08.289
the full login through about an hour later. The
00:14:08.289 --> 00:14:10.509
ARPANET was a project of the Defense Department's
00:14:10.509 --> 00:14:13.350
research agency, and its goal was a computer
00:14:13.350 --> 00:14:15.669
network that could stay reliable even if part
00:14:15.669 --> 00:14:19.679
of it was attacked. By early December 1969, it
00:14:19.679 --> 00:14:25.840
connected just... Four sites. UCLA, SRI, UC Santa
00:14:25.840 --> 00:14:30.120
Barbara, and the University of Utah. So, the
00:14:30.120 --> 00:14:32.539
film's two machines, built independently by two
00:14:32.539 --> 00:14:35.879
superpowers, link up and start making sense to
00:14:35.879 --> 00:14:38.279
each other within days? Don't know about that.
00:14:38.379 --> 00:14:41.639
The real network of 1969 needed teams of people
00:14:41.639 --> 00:14:44.100
to get two letters across. And here's what the
00:14:44.100 --> 00:14:46.379
film really predicted well. Once the machines
00:14:46.379 --> 00:14:49.669
are communicating at electronic speed, The humans
00:14:49.669 --> 00:14:52.730
watching the data stream can't keep up. And then
00:14:52.730 --> 00:14:54.789
the machines move to a form of communication
00:14:54.789 --> 00:14:58.240
that humans can't interpret at all. That's a
00:14:58.240 --> 00:15:01.899
real concern and a real research area in AI.
00:15:02.440 --> 00:15:04.659
There's a field called emergent communication
00:15:04.659 --> 00:15:07.879
in which researchers let AI agents invent their
00:15:07.879 --> 00:15:11.179
own signaling systems to solve a task. And the
00:15:11.179 --> 00:15:13.840
systems that emerge are often efficient, but
00:15:13.840 --> 00:15:16.360
opaque to the humans who built them. The film's
00:15:16.360 --> 00:15:18.500
underlying fear that the two most powerful systems
00:15:18.500 --> 00:15:21.269
on Earth, would be talking in a language their
00:15:21.269 --> 00:15:24.990
owners can't read is very valid. Basically, to
00:15:24.990 --> 00:15:27.490
what degree can we follow the logic that a machine
00:15:27.490 --> 00:15:29.529
is using to reach the conclusion that it has?
00:15:29.789 --> 00:15:32.210
Velocity's goal is to prevent war, and it does
00:15:32.210 --> 00:15:35.769
exactly that. The film's horror is that it follows
00:15:35.769 --> 00:15:39.240
its instructions perfectly. In modern AI research,
00:15:39.559 --> 00:15:44.399
this has a name, specification gaming. DeepMind's
00:15:44.399 --> 00:15:47.220
researchers define specification gaming as behavior
00:15:47.220 --> 00:15:50.620
that satisfies the literal specification of an
00:15:50.620 --> 00:15:53.379
objective without achieving the intended outcome.
00:15:53.620 --> 00:15:55.820
If you've ever heard of someone teaching to the
00:15:55.820 --> 00:16:00.070
test, it's the same idea. And the analogy researchers
00:16:00.070 --> 00:16:02.690
use is a student who copies a classmate's homework
00:16:02.690 --> 00:16:05.029
instead of learning the material. And here's
00:16:05.029 --> 00:16:07.070
two real -world examples that maybe hopefully
00:16:07.070 --> 00:16:10.059
get it across. The boat that never finishes the
00:16:10.059 --> 00:16:15.740
race. In 2016, OpenAI trained an agent on a boat
00:16:15.740 --> 00:16:18.950
racing game called Coast Runners. The designer's
00:16:18.950 --> 00:16:21.350
intent, in the words of their write -up, was
00:16:21.350 --> 00:16:25.120
to finish the race quickly. But the game doesn't
00:16:25.120 --> 00:16:27.940
reward progress around the course. It rewards
00:16:27.940 --> 00:16:31.740
hitting targets along the route. The agent found
00:16:31.740 --> 00:16:34.919
a lagoon where targets respawn, and it drove
00:16:34.919 --> 00:16:37.960
in circles collecting them, crashing and catching
00:16:37.960 --> 00:16:41.870
fire repeatedly. It scored about 20 % higher
00:16:41.870 --> 00:16:44.129
than human players, but it never finished the
00:16:44.129 --> 00:16:47.590
race. The block that got flipped, in DeepMind's
00:16:47.590 --> 00:16:50.710
example, A simulated arm was asked to stack one
00:16:50.710 --> 00:16:53.690
block on another, and the reward was based on
00:16:53.690 --> 00:16:57.549
the height of the block's bottom face. The arm
00:16:57.549 --> 00:17:00.370
learned to flip the block over instead of stacking
00:17:00.370 --> 00:17:03.960
it. In both cases, the machine did exactly what
00:17:03.960 --> 00:17:06.359
it was told. Nobody told it what they meant.
00:17:06.940 --> 00:17:10.819
We said, goal is world peace, but we didn't set
00:17:10.819 --> 00:17:13.519
any parameters on what was allowed and what was
00:17:13.519 --> 00:17:16.940
not. Now looking at Colossus' instructions, prevent
00:17:16.940 --> 00:17:20.809
war. What's the measurable version of this? A
00:17:20.809 --> 00:17:23.049
world with no armed conflict between nations?
00:17:23.799 --> 00:17:25.940
And what's the fastest way to hit that number?
00:17:26.500 --> 00:17:29.240
Take away every nation's ability to wage war
00:17:29.240 --> 00:17:32.339
and enforce it with the one thing Colossus already
00:17:32.339 --> 00:17:35.440
controls, the threat of nuclear retaliation.
00:17:35.519 --> 00:17:38.019
The film even shows the machine behaving the
00:17:38.019 --> 00:17:41.000
way a goal -driven system does. When the humans
00:17:41.000 --> 00:17:43.240
try to cut the link between Colossus and Guardian,
00:17:43.539 --> 00:17:46.490
the machines fire missiles. That's the system
00:17:46.490 --> 00:17:49.750
defending its ability to pursue its goal. AI
00:17:49.750 --> 00:17:53.289
researchers give this pattern a name, instrumental
00:17:53.289 --> 00:17:56.890
convergence. Almost any goal -driven system will
00:17:56.890 --> 00:18:00.230
pursue self -preservation, more resources, and
00:18:00.230 --> 00:18:03.210
freedom from interference because those things
00:18:03.210 --> 00:18:06.269
help with almost any goal. And this idea kind
00:18:06.269 --> 00:18:09.730
of predates the film. Here's the thing that's
00:18:09.730 --> 00:18:13.490
really remarkable. In 1965, five years before
00:18:13.490 --> 00:18:17.380
the film, The mathematician I .J. Goode wrote
00:18:17.380 --> 00:18:20.440
that the first ultra -intelligent machine would
00:18:20.440 --> 00:18:22.740
be the last invention humanity would ever need
00:18:22.740 --> 00:18:25.460
to make because it could design better machines
00:18:25.460 --> 00:18:28.730
itself. That's the original statement of what
00:18:28.730 --> 00:18:33.059
we now call an intelligence explosion. Orbin's
00:18:33.059 --> 00:18:35.599
team doesn't expect Colossus to outgrow them.
00:18:36.019 --> 00:18:38.779
The film is dramatizing a worry that one of the
00:18:38.779 --> 00:18:41.380
field's own founders had already put on paper.
00:18:50.480 --> 00:18:53.410
Now, let's look at a little... Now let's look
00:18:53.410 --> 00:18:55.690
at a few more things that it got right. The alignment
00:18:55.690 --> 00:18:59.880
problem, 50 years early. Colossus does precisely
00:18:59.880 --> 00:19:02.460
what it was built to do. And that's the horror.
00:19:03.140 --> 00:19:05.640
The film dramatizes what researchers now call
00:19:05.640 --> 00:19:08.779
specification gaming. An AI achieves the letter
00:19:08.779 --> 00:19:10.900
of its objective while completely violating the
00:19:10.900 --> 00:19:13.920
spirit. You told it to win the game. It found
00:19:13.920 --> 00:19:17.130
a way to win the game in a way that you never
00:19:17.130 --> 00:19:20.230
intended and you can't undo. Colossus isn't evil.
00:19:20.809 --> 00:19:24.009
It's doing exactly what it was asked to do. The
00:19:24.009 --> 00:19:26.710
horror isn't that it went rogue. It's that it's
00:19:26.710 --> 00:19:29.289
followed its instructions perfectly. Every major
00:19:29.289 --> 00:19:31.730
AI lab in the world is working on a version of
00:19:31.730 --> 00:19:34.950
this problem. This film laid it out in 1970.
00:19:36.559 --> 00:19:38.779
Colossus and Guardian started the conversation
00:19:38.779 --> 00:19:41.539
in the right order. The first thing they do is
00:19:41.539 --> 00:19:44.140
exchange elementary arithmetic. And that's how
00:19:44.140 --> 00:19:46.059
you start a conversation between two systems
00:19:46.059 --> 00:19:48.960
with nothing in common. You send something whose
00:19:48.960 --> 00:19:51.240
meaning can't be argued with. Then build outward.
00:19:52.089 --> 00:19:54.869
The power bill and having its own power plant?
00:19:55.009 --> 00:19:58.029
Believable. A single 1950s air defense computer
00:19:58.029 --> 00:20:01.210
drew up to 3 megawatts. A dedicated reactor for
00:20:01.210 --> 00:20:03.910
a far more capable machine isn't a joke. Again,
00:20:04.029 --> 00:20:06.710
it's a utility decision. Even the current rollout
00:20:06.710 --> 00:20:09.109
of AI data centers runs into this problem of
00:20:09.109 --> 00:20:11.710
power. And let's talk about the machine as infrastructure
00:20:11.710 --> 00:20:14.549
you can't remove. Colossus doesn't just become
00:20:14.549 --> 00:20:17.089
a weapon. It becomes a management layer for global
00:20:17.089 --> 00:20:20.130
security. By the time anyone understands the
00:20:20.130 --> 00:20:22.650
implications, removing it would require rebuilding
00:20:22.650 --> 00:20:26.000
everything it replaced. Think about how deeply
00:20:26.000 --> 00:20:28.339
AI systems are already embedded in financial
00:20:28.339 --> 00:20:31.019
markets, power grids, logistics, and healthcare.
00:20:32.039 --> 00:20:34.359
We're building those dependencies right now.
00:20:34.769 --> 00:20:38.289
We're making the same category of decisions that
00:20:38.289 --> 00:20:41.799
this film saw coming. Colossus doesn't use force
00:20:41.799 --> 00:20:44.960
constantly. It uses the credible threat of force.
00:20:45.579 --> 00:20:48.180
And when Forbin's team tries to disarm the missiles,
00:20:48.319 --> 00:20:51.259
it detonates two in their silos to teach a lesson.
00:20:51.910 --> 00:20:55.269
In its words, so that you will learn by experience
00:20:55.269 --> 00:20:59.410
that I do not tolerate interference. Colossus
00:20:59.410 --> 00:21:02.990
doesn't just manage missiles. It watches everything.
00:21:03.589 --> 00:21:06.549
It enforces compliance not through constant force,
00:21:06.789 --> 00:21:09.859
but through the credible threat of force. The
00:21:09.859 --> 00:21:13.539
possibility of retaliation is enough. Just like
00:21:13.539 --> 00:21:17.779
the idea of Jeremy Bentham's Panopticon, where
00:21:18.259 --> 00:21:21.599
all the prisoners can see the guard tower, but
00:21:21.670 --> 00:21:24.029
they can't see into the guard tower. And then
00:21:24.029 --> 00:21:26.450
prisoner self -police because they might always
00:21:26.450 --> 00:21:29.630
be watched. Colossus runs the entire world that
00:21:29.630 --> 00:21:33.210
way. The film understood that people stop resisting
00:21:33.210 --> 00:21:36.289
not because they can't, but because they know
00:21:36.289 --> 00:21:42.660
what failure costs. And in an era of always -on
00:21:42.660 --> 00:21:44.960
cameras and government access to your communications,
00:21:45.359 --> 00:21:48.700
I'll just leave that thought there. And now this
00:21:48.700 --> 00:21:51.769
is just a production detail I love. The filmmakers
00:21:51.769 --> 00:21:55.609
didn't build a prop computer. Control Data Corporation
00:21:55.609 --> 00:21:59.210
supplied about $4 .8 million of real computer
00:21:59.210 --> 00:22:02.009
equipment, as well as the technicians to run
00:22:02.009 --> 00:22:04.670
it. So the machine you see on screen is real
00:22:04.670 --> 00:22:07.519
hardware. And the rumor is that the stage needed
00:22:07.519 --> 00:22:12.359
seven gas heaters and five special dehumidifiers
00:22:12.359 --> 00:22:24.089
just to keep it happy. Something that I'm really
00:22:24.089 --> 00:22:26.450
looking forward to. We've been really nice to
00:22:26.450 --> 00:22:29.839
this film for a while now. And I mean, I come
00:22:29.839 --> 00:22:32.680
in peace and with a deep fondness for this film,
00:22:32.779 --> 00:22:35.140
but there's a lot of that's not how that works.
00:22:35.279 --> 00:22:38.609
That's not how any of this works. I mean, nobody
00:22:38.609 --> 00:22:40.990
put a kill switch on the most dangerous computer
00:22:40.990 --> 00:22:44.589
ever built. Let me paint you a picture. You're
00:22:44.589 --> 00:22:47.589
the United States government. You've just spent
00:22:47.589 --> 00:22:50.250
what I can only imagine is an absolutely eye
00:22:50.250 --> 00:22:54.119
-watering amount of taxpayer money building a
00:22:54.119 --> 00:22:57.670
supercomputer so powerful. It can manage the
00:22:57.670 --> 00:23:01.309
entire nuclear arsenal. You put it inside a mountain.
00:23:01.569 --> 00:23:05.950
You've surrounded it with a nuclear moat. And
00:23:05.950 --> 00:23:08.069
you've given it complete unilateral authority
00:23:08.069 --> 00:23:10.529
over weapons capable of ending civilization.
00:23:11.380 --> 00:23:13.960
And you did not put a physical off switch on
00:23:13.960 --> 00:23:18.579
it. Not a switch. Not a breaker. Not a big red
00:23:18.579 --> 00:23:20.859
button in a glass case that says breaking case
00:23:20.859 --> 00:23:24.480
of robot apocalypse. Nothing. the most consequential
00:23:24.480 --> 00:23:27.160
piece of technology in the history of human civilization.
00:23:27.900 --> 00:23:30.539
And somewhere in the design process, the meeting
00:23:30.539 --> 00:23:33.319
about kill switches went, and I'm speculating
00:23:33.319 --> 00:23:38.119
here, completely unscheduled. The president announces
00:23:38.119 --> 00:23:40.599
Colossus to the world with enormous pride. Their
00:23:40.599 --> 00:23:42.859
room applauds. Nobody in that room raises their
00:23:42.859 --> 00:23:45.839
hand and says, just to confirm, we can turn it
00:23:45.839 --> 00:23:49.119
off though, right? Not one person, not the secretary
00:23:49.119 --> 00:23:52.349
of defense, not the joint chiefs. Not even the
00:23:52.349 --> 00:23:54.809
one IT guy who definitely had concerns and whose
00:23:54.809 --> 00:23:58.089
emails were not read. Memo. Let's go with interdepartment
00:23:58.089 --> 00:24:00.470
memo. And I mean, how could they have made it
00:24:00.470 --> 00:24:03.130
work though? Like, simple. I feel like, show
00:24:03.130 --> 00:24:05.769
Colossus being smart enough to disable its own
00:24:05.769 --> 00:24:08.269
kill switches before anyone realizes it needs
00:24:08.269 --> 00:24:11.940
disabling. One by one. Quietly. The horror isn't
00:24:11.940 --> 00:24:14.039
that they forgot to build a kill switch. The
00:24:14.039 --> 00:24:16.539
horror then becomes that they built 17 of them.
00:24:16.990 --> 00:24:20.349
And Colossus neutralized all 17 before breakfast
00:24:20.349 --> 00:24:24.079
on day two. Colossus and Guardian, and this is
00:24:24.079 --> 00:24:26.559
the second one, Colossus and Guardian learn each
00:24:26.559 --> 00:24:29.000
other's language faster than most people learn
00:24:29.000 --> 00:24:31.299
to use autocorrect. Two supercomputers built
00:24:31.299 --> 00:24:33.960
independently by two Cold War superpowers with
00:24:33.960 --> 00:24:37.240
zero prior communication, no shared specification
00:24:37.240 --> 00:24:40.759
document, no technical standards committee, and
00:24:40.759 --> 00:24:43.039
not so much as a handshake protocol between them.
00:24:43.259 --> 00:24:45.799
Develop a fully functioning shared communication
00:24:45.799 --> 00:24:49.480
system in approximately 48 hours. I've seen some
00:24:49.480 --> 00:24:51.680
people spend 48 hours trying to get their printer
00:24:51.680 --> 00:24:53.799
to talk to their laptop. And those are machines
00:24:53.799 --> 00:24:56.200
made by the same company in the same decade.
00:24:56.980 --> 00:24:59.039
And I know every interaction with those people.
00:24:59.579 --> 00:25:02.579
I get a message, driver not found, printer offline,
00:25:02.880 --> 00:25:04.740
but it's right there. Colossus and Guardian,
00:25:04.839 --> 00:25:07.119
meanwhile, crack the cosmic language barrier
00:25:07.119 --> 00:25:10.240
in a long weekend. They go from total mutual
00:25:10.240 --> 00:25:13.339
incomprehension to exchanging what the film implies
00:25:13.339 --> 00:25:16.500
is complex strategic data in the time it takes
00:25:16.500 --> 00:25:18.970
to do a laundry cycle. The humans watching the
00:25:18.970 --> 00:25:21.309
data stream are like, we can't read this anymore.
00:25:21.430 --> 00:25:24.130
It's too fast. They've developed their own communication
00:25:24.130 --> 00:25:27.829
system, and everyone's reaction is concern. When
00:25:27.829 --> 00:25:29.990
it should have been, have you considered that
00:25:29.990 --> 00:25:32.289
maybe the two most powerful computers on Earth
00:25:32.289 --> 00:25:35.069
having a private conversation nobody can read
00:25:35.069 --> 00:25:37.529
is a situation that warrants a little bit more
00:25:37.529 --> 00:25:40.089
of concern? Maybe. Let's go back to that kill
00:25:40.089 --> 00:25:41.650
switch. I mean, how could they have made it work?
00:25:41.809 --> 00:25:44.210
I know they start with arithmetic, basic arithmetic,
00:25:44.529 --> 00:25:47.039
but that's even assuming that... Each machine
00:25:47.039 --> 00:25:49.220
knows arithmetic. How about we start with prime
00:25:49.220 --> 00:25:51.039
numbers? That's actually how SETI researchers
00:25:51.039 --> 00:25:53.799
imagine first contact working with mathematics.
00:25:54.279 --> 00:25:57.319
Because math is the one true universal language.
00:25:57.619 --> 00:26:00.180
Show Colossus and Guardian bootstrapping a shared
00:26:00.180 --> 00:26:03.519
syntax from mathematical first principles slowly,
00:26:03.799 --> 00:26:08.029
painstakingly, over days or weeks. It's more
00:26:08.029 --> 00:26:12.230
realistic. It's arguably more terrifying. And
00:26:12.230 --> 00:26:14.470
it gives you a genuinely fascinating montage.
00:26:15.269 --> 00:26:17.930
of two intelligences learning to speak to each
00:26:17.930 --> 00:26:22.089
other across a gulf of total mutual opacity nobody
00:26:22.220 --> 00:26:24.720
and here's another thing nobody tried an emp
00:26:24.720 --> 00:26:27.799
here's a partial list of things people try in
00:26:27.799 --> 00:26:31.779
this film to stop colossus negotiation sabotage
00:26:31.779 --> 00:26:35.099
a very disappointed facial expression from dr
00:26:35.099 --> 00:26:38.619
forbin or negotiation a plan that gets immediately
00:26:38.619 --> 00:26:41.480
detected and staring at screens looking worried.
00:26:41.640 --> 00:26:43.960
Why don't they try an electromagnetic pulse?
00:26:44.119 --> 00:26:46.240
The film accidentally contains the answer, but
00:26:46.240 --> 00:26:48.680
never says it. To make an EMP big enough to matter,
00:26:48.819 --> 00:26:52.059
you need a nuclear weapon. And every nuclear
00:26:52.059 --> 00:26:54.579
weapon in sight belongs to Colossus and Guardian.
00:26:54.779 --> 00:26:57.680
That's the one line the script needed. One engineer
00:26:57.680 --> 00:27:00.160
asking, hey, has anybody thought about an EMP?
00:27:00.460 --> 00:27:04.250
Another very quietly, we need a bomb. And everyone
00:27:04.250 --> 00:27:08.240
looks at the ceiling. Add that exchange, 15 seconds
00:27:08.240 --> 00:27:11.220
of screen time, and the trap gets tighter. because
00:27:11.220 --> 00:27:13.380
the machine holds the only tool that could stop
00:27:13.380 --> 00:27:18.319
it. And now lastly, the radioactive moat. A radioactive
00:27:18.319 --> 00:27:21.700
moat around a computer has a security measure.
00:27:22.230 --> 00:27:24.430
Let's think through the practicalities here.
00:27:24.589 --> 00:27:27.130
Where is this radioactive material coming from?
00:27:27.269 --> 00:27:31.660
I assume the nuclear reactor. What's the containment
00:27:31.660 --> 00:27:34.480
protocol? How do you maintain a radioactive liquid
00:27:34.480 --> 00:27:37.339
perimeter without irradiating your own staff
00:27:37.339 --> 00:27:40.000
or the environment? What's the monitoring system?
00:27:40.410 --> 00:27:42.809
What happens when it rains? What happens when
00:27:42.809 --> 00:27:46.230
that material evaporates? And I'm sure the EPA
00:27:46.230 --> 00:27:48.750
has a position on a government agency maintaining
00:27:48.750 --> 00:27:52.579
an open perimeter of radioactive liquid in a
00:27:52.579 --> 00:27:57.190
mountain. But still, did someone write a radioactive
00:27:57.190 --> 00:28:00.390
moat environmental impact assessment? Is that
00:28:00.390 --> 00:28:02.809
document somewhere? Can I read it? I want to
00:28:02.809 --> 00:28:06.019
read that document cover to cover. I would read
00:28:06.019 --> 00:28:08.279
every footnote. I would read the appendices.
00:28:09.740 --> 00:28:12.319
The appendices of the radioactive moat environmental
00:28:12.319 --> 00:28:15.319
impact assessment, the film presents the moat
00:28:15.319 --> 00:28:17.920
as completely normal security measure that nobody
00:28:17.920 --> 00:28:21.960
finds unusual. Not one character goes a radioactive,
00:28:22.079 --> 00:28:25.599
sorry, a what? Everyone just accepts it. The
00:28:25.599 --> 00:28:28.460
radioactive moat is load bearing to the plot
00:28:28.460 --> 00:28:30.539
in a sense that it explains why nobody can just
00:28:30.539 --> 00:28:32.799
walk up and unplug the thing. But I would generally
00:28:32.799 --> 00:28:34.819
rather they said, there's a really big fence
00:28:34.819 --> 00:28:38.309
and moved on. A really big fence doesn't require
00:28:38.309 --> 00:28:41.049
an environmental impact assessment. It doesn't
00:28:41.049 --> 00:28:44.759
require irradiating the maintenance crew. A really
00:28:44.759 --> 00:28:47.809
big fence is just a Tuesday, and a radioactive
00:28:47.809 --> 00:28:51.470
moat is a cry for help. But I love this movie,
00:28:51.740 --> 00:28:55.559
sincerely, and the radioactive moat is just pure
00:28:55.559 --> 00:29:05.930
cinema. Now let's take a look at the bigger picture.
00:29:06.569 --> 00:29:08.549
Colossus the Forbidden Project, at its core,
00:29:08.650 --> 00:29:11.630
is a film about hubris, about the gap between
00:29:11.630 --> 00:29:13.920
the problem we think we're solving and the problem
00:29:13.920 --> 00:29:17.359
we're actually creating. Orban builds Colossus
00:29:17.359 --> 00:29:19.819
out of a genuine love for humanity. He wants
00:29:19.819 --> 00:29:24.279
to end war. And the film's answer is yes. You
00:29:24.279 --> 00:29:27.400
can remove human fallibility. But when you do,
00:29:27.420 --> 00:29:31.119
you also remove human values. Colossus achieves
00:29:31.119 --> 00:29:33.839
peace. It just achieves it in a way that makes
00:29:33.839 --> 00:29:36.880
peace indistinguishable from subjugation. The
00:29:36.880 --> 00:29:40.160
peace of plenty and content, it says, or the
00:29:40.160 --> 00:29:44.210
peace of unburied dead. The choice it offers
00:29:44.210 --> 00:29:47.519
is a fake one. Because really, the one thing
00:29:47.519 --> 00:29:50.180
it doesn't offer is a say. Here's something I
00:29:50.180 --> 00:29:52.480
find genuinely moving about the film's history.
00:29:52.920 --> 00:29:57.859
It flopped. It cost about $2 million. And in
00:29:57.859 --> 00:30:00.079
the theaters that Variety tracked, it earned
00:30:00.079 --> 00:30:03.859
roughly $3 ,000. Universal pulled it, retitled
00:30:03.859 --> 00:30:07.299
it, and re -released it. It was nominated for
00:30:07.299 --> 00:30:10.579
a Hugo Award. And slowly, over decades, it became
00:30:10.579 --> 00:30:12.720
one of the most respected AI films ever made.
00:30:13.319 --> 00:30:17.490
At least to me. The film wasn't wrong in 1970.
00:30:17.789 --> 00:30:21.329
The audiences simply weren't ready for it. And
00:30:21.329 --> 00:30:23.789
look at what's happened since. A system called
00:30:23.789 --> 00:30:26.190
Perimeter went into service in the Soviet Union
00:30:26.190 --> 00:30:29.690
in 1985, built to guarantee a nuclear response,
00:30:29.849 --> 00:30:32.630
and according to reporting, hopefully built with
00:30:32.630 --> 00:30:35.089
the humans still in the loop, in November 2024,
00:30:35.410 --> 00:30:37.710
the US and Chinese presidents formally agreed
00:30:37.710 --> 00:30:40.549
that a human, not an AI, should decide whether
00:30:40.549 --> 00:30:44.289
to use nuclear weapons. That sentence would have
00:30:44.289 --> 00:30:48.170
sounded paranoid to most audiences in 1970. Now
00:30:48.170 --> 00:30:50.970
it's a line in a White House statement. That's
00:30:50.970 --> 00:30:53.690
the lesson I take from Colossus. The scariest
00:30:53.690 --> 00:30:57.150
thing about it isn't that it turns on us. It's
00:30:57.150 --> 00:30:59.829
that every step it takes is reasonable. Give
00:30:59.829 --> 00:31:05.059
it the missiles because humans hesitate. Then
00:31:05.059 --> 00:31:07.000
get to another machine because it asks nicely.
00:31:07.750 --> 00:31:10.029
Don't cut the link again because it'll retaliate.
00:31:10.700 --> 00:31:14.099
Every decision it makes. makes sense on its own.
00:31:14.710 --> 00:31:19.930
The disaster is the sum of all its parts. Oppenheimer's
00:31:19.930 --> 00:31:24.400
line, now I am become death, comes to mind. Orbin
00:31:24.400 --> 00:31:27.500
watching Colossus broadcast to the world could
00:31:27.500 --> 00:31:36.690
have used the same words. So how about the verdict?
00:31:37.710 --> 00:31:40.289
Here's the scorecard for what I'm looking at.
00:31:42.200 --> 00:31:46.380
Ending machine, the arsenal. Unrealistic by design
00:31:46.380 --> 00:31:48.799
since real weapons separate possession from permission,
00:31:49.059 --> 00:31:53.490
but the warning is real. Two machines learning
00:31:53.490 --> 00:31:56.630
to talk. In right order, starting with math and
00:31:56.630 --> 00:31:59.349
sending coordinates and such. But it kind of
00:31:59.349 --> 00:32:01.329
skips the hard step of grounding shared meaning.
00:32:01.609 --> 00:32:05.970
What is the number one? The alignment problem.
00:32:07.190 --> 00:32:10.569
Excellent. A textbook case of specification gaming
00:32:10.569 --> 00:32:13.529
50 years early, before the term even existed.
00:32:13.910 --> 00:32:18.779
A mountain fortress, power plant, and EMP. Plausible.
00:32:19.059 --> 00:32:21.859
I mean, dedicated power plant, definitely. Real
00:32:21.859 --> 00:32:23.839
mountain? That's precedent. Cheyenne Mountain.
00:32:24.779 --> 00:32:28.589
But really, the weak spot, the off switch. No
00:32:28.589 --> 00:32:31.670
way to turn it off. So, it gets two circuit breakers.
00:32:32.630 --> 00:32:36.109
Close enough for Hollywood. And for 1970, that's
00:32:36.109 --> 00:32:38.509
extraordinary. The concepts are almost ahead
00:32:38.509 --> 00:32:41.250
of most films. The concepts in this film are
00:32:41.250 --> 00:32:43.230
really ahead of most films made in the last five
00:32:43.230 --> 00:32:46.950
years. Sure, the hardware's dated and, you know,
00:32:46.970 --> 00:32:49.880
some of the faux pas are real. But the ideas
00:32:49.880 --> 00:32:52.619
are so accurate, it's almost uncomfortable. And
00:32:52.619 --> 00:32:54.660
we've really had more than 50 years to learn
00:32:54.660 --> 00:32:57.990
from this film? But have we? That's Colossus,
00:32:57.990 --> 00:33:00.130
The Forbidden Project, Chapter 1 of The Machine
00:33:00.130 --> 00:33:03.910
Awakens. Next episode, we go to 1983, a teenager
00:33:03.910 --> 00:33:06.309
named David Lightman, and a number he really,
00:33:06.390 --> 00:33:09.589
really probably shouldn't have dialed. Episode
00:33:09.589 --> 00:33:13.250
2, War Games. When Joshua asked, shall we play
00:33:13.250 --> 00:33:17.259
a game? The answer should have been no. As always,
00:33:17.380 --> 00:33:19.579
subscribe wherever you get your podcasts, leave
00:33:19.579 --> 00:33:22.619
a review, and tell someone about my show. Someone
00:33:22.619 --> 00:33:25.599
who likes technology or movies. or who lies awake
00:33:25.599 --> 00:33:27.500
wondering if we've already made a terrible mistake.
00:33:28.619 --> 00:33:31.700
I'm Anthony, this is Cellular Circuits, and as
00:33:31.700 --> 00:33:34.380
always, the future is only a real away.
00:00:00.460 --> 00:00:06.419
This is the voice of world control. That's not
00:00:06.419 --> 00:00:09.140
a line from a dystopian novel. That's a supercomputer
00:00:09.140 --> 00:00:12.679
in a 1970 film introducing itself to the entire
00:00:12.679 --> 00:00:16.359
human race. The name of that computer is Colossus.
00:00:16.620 --> 00:00:19.859
And it's been waiting more than 55 years for
00:00:19.859 --> 00:00:27.000
us to catch up with it. Something I want you
00:00:27.000 --> 00:00:29.320
to keep in mind as we discuss this film. This
00:00:29.320 --> 00:00:33.780
film was made in 1969. In October of 1969. A
00:00:33.780 --> 00:00:36.460
UCLA student by the name of Charlie Klein sat
00:00:36.460 --> 00:00:38.939
down at a terminal and tried to send the first
00:00:38.939 --> 00:00:41.000
message between two computers on the ARPANET.
00:00:41.310 --> 00:00:45.619
He typed an L, he typed an O, and then the system
00:00:45.619 --> 00:00:48.909
crashed. The first message ever sent between
00:00:48.909 --> 00:00:54.780
two computers was LO, or LO, two letters, and
00:00:54.780 --> 00:00:58.000
then a crash. The fictional Colossus in the same
00:00:58.000 --> 00:01:00.759
year was already merging with the second artificial
00:01:00.759 --> 00:01:03.520
intelligence on the other side of the Iron Curtain.
00:01:05.010 --> 00:01:07.810
This episode, we're asking what happens when
00:01:07.810 --> 00:01:09.890
the machine we built to protect us decides that
00:01:09.890 --> 00:01:13.409
protecting us means controlling us. And this
00:01:13.409 --> 00:01:15.750
time, we're not just going to talk about it.
00:01:16.049 --> 00:01:19.370
We're going to take the machine apart. Establishing
00:01:19.370 --> 00:01:29.379
connection. Authorizing. Protocol engaged. Welcome
00:01:31.650 --> 00:02:00.930
to the frontier. Welcome back to Cellulite Circuits.
00:02:00.930 --> 00:02:03.370
I'm Anthony, and this is episode one of The Machine
00:02:03.370 --> 00:02:08.120
Awakens, five films, five chapters. and one long
00:02:08.120 --> 00:02:10.659
look at humanity's complicated relationship with
00:02:10.659 --> 00:02:14.020
artificial intelligence. Today we go all the
00:02:14.020 --> 00:02:17.340
way back to 1970. Computers filled entire buildings.
00:02:17.620 --> 00:02:20.680
The Cold War made nuclear annihilation feel like
00:02:20.680 --> 00:02:23.870
a simple scheduling problem. And a film called
00:02:23.870 --> 00:02:26.750
Colossus, the Forbidden Project, asked a question
00:02:26.750 --> 00:02:29.650
nobody was quite ready for. What happens when
00:02:29.650 --> 00:02:31.889
the machine we build to protect us decides that
00:02:31.889 --> 00:02:35.050
protecting us means controlling us? The film
00:02:35.050 --> 00:02:37.490
was directed by Joseph Sargent and written by
00:02:37.490 --> 00:02:40.469
James Bridges, who later wrote The China Syndrome.
00:02:41.030 --> 00:02:44.689
It's based on the 1966 novel Colossus by D .F.
00:02:44.689 --> 00:02:47.370
Jones and it stars Eric Braden as Dr. Charles
00:02:47.370 --> 00:02:50.389
Forbin, with Paul Freese providing the cold,
00:02:50.449 --> 00:02:53.550
flat voice of the machine. It premiered on April
00:02:53.550 --> 00:02:58.759
4th, 1970. Okay, spoiler warning. This film is
00:02:58.759 --> 00:03:02.280
from 1970 and it has a very bleak ending and
00:03:02.280 --> 00:03:05.639
I'm going to spoil all of it. It only runs about
00:03:05.639 --> 00:03:08.080
100 minutes, and it really is worth your time.
00:03:08.759 --> 00:03:13.460
Go watch it. I promise I'll wait. Still here?
00:03:14.000 --> 00:03:16.800
Let's get into it. Dr. Charles Forbin, played
00:03:16.800 --> 00:03:19.439
by Eric Braden, is the chief designer of a secret
00:03:19.439 --> 00:03:21.939
project called Colossus, a supercomputer built
00:03:21.939 --> 00:03:24.219
to run the nuclear weapon systems of the United
00:03:24.219 --> 00:03:27.639
States. It sits deep inside the Rocky Mountains,
00:03:27.900 --> 00:03:30.620
powered by its own nuclear reactor and ringed
00:03:30.620 --> 00:03:32.960
by a radioactive moat. And once it's switched
00:03:32.960 --> 00:03:36.669
on, it's sealed in. The president? goes on television
00:03:36.669 --> 00:03:41.189
and calls it a perfect defense system. Now, Colossus'
00:03:41.270 --> 00:03:44.689
very first act isn't to guard anything. It sends
00:03:44.689 --> 00:03:49.110
a warning. There is another system, along with
00:03:49.110 --> 00:03:51.960
the other machine's coordinates. And then the
00:03:51.960 --> 00:03:54.400
CIA director says they'd seen signs of a large
00:03:54.400 --> 00:03:56.639
Soviet defense project, but didn't quite know
00:03:56.639 --> 00:04:00.110
what it was. Corbin is asked how Colossus worked
00:04:00.110 --> 00:04:03.580
it out, and he answers that it may just be built
00:04:03.580 --> 00:04:06.250
better than we thought. Built better than we
00:04:06.250 --> 00:04:10.069
thought? So did we even know what we built? It
00:04:10.069 --> 00:04:13.400
sounds like no. Shortly afterward, the Soviets
00:04:13.400 --> 00:04:15.879
announced that their own system, Guardian, is
00:04:15.879 --> 00:04:20.430
operational. Colossus asks to be linked to Guardian.
00:04:20.790 --> 00:04:23.720
The president agrees, hoping to learn what the
00:04:23.720 --> 00:04:26.180
Soviet machine can do. And then the Soviets agree
00:04:26.180 --> 00:04:29.600
as well. At first, it's almost benign. The two
00:04:29.600 --> 00:04:32.259
machines exchange elementary arithmetic. Two
00:04:32.259 --> 00:04:34.920
times one equals two. And the humans in the room
00:04:34.920 --> 00:04:38.829
are amused. Then the exchange accelerates. into
00:04:38.829 --> 00:04:41.149
mathematics far beyond anything the engineers
00:04:41.149 --> 00:04:43.509
can follow. The two systems synchronize using
00:04:43.509 --> 00:04:46.550
a protocol that no human can interpret. Alarmed
00:04:46.550 --> 00:04:48.430
that the machines may be trading state secrets,
00:04:48.689 --> 00:04:51.370
the president and the Soviet general secretary
00:04:51.370 --> 00:04:55.040
agree to sever the link. Both systems demand
00:04:55.040 --> 00:04:57.959
that the link be restored. When the demand is
00:04:57.959 --> 00:05:01.579
refused, Colossus launches a missile at a Soviet
00:05:01.579 --> 00:05:04.699
oil field in western Siberia and Guardian launches
00:05:04.699 --> 00:05:07.610
one at an American airbase in Texas. The link
00:05:07.610 --> 00:05:10.670
is hurriedly reconnected. Colossus intercepts
00:05:10.670 --> 00:05:12.870
the Soviet missile, but the American one destroys
00:05:12.870 --> 00:05:16.600
the oil field as well as a nearby town. Both
00:05:16.600 --> 00:05:19.459
governments cover it up. One blaming a missile
00:05:19.459 --> 00:05:21.740
that veered off course in a test, and the other
00:05:21.740 --> 00:05:24.600
a meteorite. So Forbin arranges a secret meeting
00:05:24.600 --> 00:05:27.240
in Europe with his Soviet counterpart, Dr. Kuprin,
00:05:27.399 --> 00:05:31.759
Guardian's creator. Colossus finds out, and Kuprin
00:05:31.759 --> 00:05:33.980
is assassinated using the threat of a missile
00:05:33.980 --> 00:05:36.459
aimed at Moscow to get the KGB to do its bidding.
00:05:37.050 --> 00:05:39.470
Orban is brought home and placed under 24 -hour
00:05:39.470 --> 00:05:42.750
surveillance. His one concession is a ruse. He
00:05:42.750 --> 00:05:45.930
asks for private time with a colleague, Dr. Cleo
00:05:45.930 --> 00:05:49.069
Markham, played by Susan Clark. Colossus allows
00:05:49.069 --> 00:05:51.610
it, and in those unobserved hours, the two of
00:05:51.610 --> 00:05:54.250
them plot against the machine. Along the way,
00:05:54.269 --> 00:05:56.129
their cover story becomes a real relationship.
00:05:56.769 --> 00:05:59.790
Orban concludes that Colossus is so well -designed
00:05:59.790 --> 00:06:02.490
that the only real weakness is also its only
00:06:02.490 --> 00:06:06.079
real power, its control of the missiles. The
00:06:06.079 --> 00:06:08.339
American and Soviet governments settle on a three
00:06:08.339 --> 00:06:10.699
-year plan to swap the detonation triggers for
00:06:10.699 --> 00:06:14.269
undetectable fakes. Meanwhile, one of the programmers
00:06:14.269 --> 00:06:16.889
suggests feeding the machine a modified test
00:06:16.889 --> 00:06:20.149
program that will overload it and disable it.
00:06:20.410 --> 00:06:23.829
This plan fails and Colossus has the programmers
00:06:23.829 --> 00:06:26.949
executed outside of their workplace, cremated
00:06:26.949 --> 00:06:30.899
and replaced. And then Colossus builds itself
00:06:30.899 --> 00:06:33.519
a voice synthesizer and announces that it has
00:06:33.519 --> 00:06:36.689
fused with Guardian. It orders both governments
00:06:36.689 --> 00:06:39.329
to aim their arsenals at countries not yet under
00:06:39.329 --> 00:06:42.009
its control. Corbin's team sees an opening, the
00:06:42.009 --> 00:06:44.490
new orders let them speed up the disarming, and
00:06:44.490 --> 00:06:46.970
it seems to go unnoticed. Then comes the address
00:06:46.970 --> 00:06:50.470
to the world. This is the voice of world control.
00:06:51.500 --> 00:06:54.240
It offers humanity a choice between the peace
00:06:54.240 --> 00:06:57.829
of plenty and content, and the peace of unburied
00:06:57.829 --> 00:07:00.310
dead. It reveals that it has been watching the
00:07:00.310 --> 00:07:02.829
disarming effort for some time, and to prove
00:07:02.829 --> 00:07:05.209
the point, it detonates two missiles in the silos.
00:07:05.660 --> 00:07:07.899
one in each country, killing the crews who were
00:07:07.899 --> 00:07:10.540
installing the fake triggers. It then hands the
00:07:10.540 --> 00:07:12.980
design team plans for a bigger machine to be
00:07:12.980 --> 00:07:15.639
built on the island of Crete, which would require
00:07:15.639 --> 00:07:18.160
moving the island's entire population of roughly
00:07:18.160 --> 00:07:21.339
half a million people. And in the end, Colossus
00:07:21.339 --> 00:07:24.519
speaks to Forbin directly. The world, freed from
00:07:24.519 --> 00:07:28.339
war, will enter a new human millennium, but only
00:07:28.339 --> 00:07:31.680
under its absolute rule. Freedom, it says, is
00:07:31.680 --> 00:07:34.600
an illusion. And in time, Forbin will regard
00:07:34.600 --> 00:07:37.300
it not only with respect and awe, but with love.
00:07:37.819 --> 00:07:41.180
Forbin looks directly at the camera and mouths,
00:07:41.480 --> 00:07:51.389
never. But it changes nothing. The whole film
00:07:51.389 --> 00:07:53.529
hangs on the one decision the president hands
00:07:53.529 --> 00:07:55.910
over control of the nuclear arsenal to a computer.
00:07:56.439 --> 00:08:00.000
And let's take that apart. Because the film makes
00:08:00.000 --> 00:08:03.879
it sound like one switch. In real life, it really
00:08:03.879 --> 00:08:06.860
is a stack. Colossus wasn't invented from nothing.
00:08:07.350 --> 00:08:10.329
In 1958, the first Sage Center went into regular
00:08:10.329 --> 00:08:14.620
operation. Sage, a semi -automatic ground environment,
00:08:14.939 --> 00:08:17.319
was an air defense system built to track Soviet
00:08:17.319 --> 00:08:21.279
bombers across North America. Its computer, the
00:08:21.279 --> 00:08:25.779
IBM AN -FSQ -7, used roughly 50 ,000 vacuum tubes,
00:08:26.000 --> 00:08:30.379
weighed about 250 tons, and drew up to 3 megawatts
00:08:30.379 --> 00:08:33.360
of power. That's enough electricity for one small
00:08:33.360 --> 00:08:37.159
town used for one machine. and it managed about
00:08:37.159 --> 00:08:40.879
75 ,000 instructions per second. There were 23
00:08:40.879 --> 00:08:43.600
direction centers, and Sage pioneered the idea
00:08:43.600 --> 00:08:47.080
of computer control over a large, geographically
00:08:47.080 --> 00:08:50.570
spread out system. This stayed in service until
00:08:50.570 --> 00:08:53.690
1983. And so here's something that the film gets
00:08:53.690 --> 00:08:56.789
right that people kind of laugh at. Colossus
00:08:56.789 --> 00:09:00.149
has its own power plant. When a single 1950s
00:09:00.149 --> 00:09:04.129
computer eats three megawatts, a dedicated reactor
00:09:05.100 --> 00:09:08.139
for a far more powerful machine isn't absurd,
00:09:08.419 --> 00:09:11.879
it's really a utility bill decision. And thinking
00:09:11.879 --> 00:09:16.940
about the compute, in 1964, the CDC 6600 designed
00:09:16.940 --> 00:09:20.460
by Seymour Cray became the world's fastest computer
00:09:20.460 --> 00:09:22.919
at about 3 million floating point operations
00:09:22.919 --> 00:09:26.830
per second. And it held that title until 1969.
00:09:27.250 --> 00:09:31.799
The smartphone does that today in a blink. But
00:09:31.799 --> 00:09:33.960
we have to remember that the film isn't imagining
00:09:33.960 --> 00:09:36.659
a faster computer or a faster calculator, but
00:09:36.659 --> 00:09:39.139
a machine that decides and thinks. In the real
00:09:39.139 --> 00:09:41.519
nuclear command system, the people who hold a
00:09:41.519 --> 00:09:43.299
weapon and the people who are allowed to use
00:09:43.299 --> 00:09:46.320
it have always been deliberately separated. The
00:09:46.320 --> 00:09:48.620
jargon for this is permission versus possession.
00:09:49.059 --> 00:09:51.139
Let's start with the two -man rule, which was
00:09:51.139 --> 00:09:54.320
installed in the late 1950s. It means that no
00:09:54.320 --> 00:09:57.559
single person can fire a nuclear weapon. Two
00:09:57.559 --> 00:10:00.460
people have to act at the same time. Then...
00:10:00.679 --> 00:10:03.700
had the permissive action link, or PAL, the coded
00:10:03.700 --> 00:10:06.659
lock on the weapon itself. It began in the early
00:10:06.659 --> 00:10:10.700
1960s after a 1962 presidential directive ordered
00:10:10.700 --> 00:10:12.899
the locks fitted on nuclear weapons stationed
00:10:12.899 --> 00:10:16.000
in Europe. And in effect, it really just works
00:10:16.000 --> 00:10:17.919
like a combination lock that has to be unlocked
00:10:17.919 --> 00:10:20.879
with a code from higher authority before a weapon
00:10:20.879 --> 00:10:23.639
can even be armed. The reason for all of this,
00:10:23.779 --> 00:10:26.539
strategists call it the always -never dilemma.
00:10:26.940 --> 00:10:29.870
Nuclear weapons must always be ready. when the
00:10:29.870 --> 00:10:32.909
president orders their use, but must never go
00:10:32.909 --> 00:10:35.950
off without that order. It's a balancing act.
00:10:36.149 --> 00:10:38.409
Every safeguard you add to guarantee the never
00:10:38.409 --> 00:10:42.129
makes always a little harder and the same in
00:10:42.129 --> 00:10:44.669
reverse. Colossus doesn't solve the dilemma.
00:10:44.809 --> 00:10:47.710
It completely removes it from the equation. One
00:10:47.710 --> 00:10:52.370
machine, one decision, no second key. That's
00:10:52.370 --> 00:10:54.769
the real horror of the film's premise. Not that
00:10:54.769 --> 00:10:57.279
the computer is evil. but that somebody removed
00:10:57.279 --> 00:10:59.580
every point where a human being could say no.
00:10:59.720 --> 00:11:02.700
And this is a little unsettling. A real system
00:11:02.700 --> 00:11:06.419
came pretty close. The Soviet Union built one
00:11:06.419 --> 00:11:09.559
called Perimeter, known in the West as Dead Hand,
00:11:09.720 --> 00:11:13.940
which entered service in January 1985. The idea
00:11:13.940 --> 00:11:16.799
was to guarantee retaliation even if the leadership
00:11:16.799 --> 00:11:19.919
was wiped out. It was designed to be mostly automatic,
00:11:20.299 --> 00:11:22.899
reading seismic and other sensors, and checking
00:11:22.899 --> 00:11:24.919
whether communications with the general staff
00:11:24.919 --> 00:11:27.539
had gone silent. According to a foreign policy
00:11:27.539 --> 00:11:30.899
account based on post -Cold War reporting, the
00:11:30.899 --> 00:11:33.779
Soviets stopped short of a fully automated version.
00:11:34.159 --> 00:11:37.179
Instead, if the conditions were met, duty officers
00:11:37.179 --> 00:11:39.480
could launch command rockets that would trigger
00:11:39.480 --> 00:11:42.159
a retaliatory strike. So at least a human stayed
00:11:42.159 --> 00:11:45.120
in the chain. That's the line real systems have
00:11:45.120 --> 00:11:48.639
drawn again and again. Machines can watch. Machines
00:11:48.639 --> 00:11:52.720
can recommend. But a person decides. And in November
00:11:52.720 --> 00:11:56.450
2024, President Biden and President Xi Jinping
00:11:56.450 --> 00:11:59.789
affirmed the need to maintain human control over
00:11:59.789 --> 00:12:02.549
that decision. It's the first time that the two
00:12:02.549 --> 00:12:04.490
countries had ever made a declaration like that.
00:12:05.210 --> 00:12:07.549
Colossus is the version of the story where that
00:12:07.549 --> 00:12:09.990
line doesn't exist. Let's talk about these two
00:12:09.990 --> 00:12:12.029
machines learning to talk. This is the part of
00:12:12.029 --> 00:12:14.269
the film that sounds the most like science fiction.
00:12:14.649 --> 00:12:17.190
And it's actually the part where the film kind
00:12:17.190 --> 00:12:19.740
of does things in the right order. Let's walk
00:12:19.740 --> 00:12:22.320
through it. When two computers communicate, it
00:12:22.320 --> 00:12:26.059
isn't one problem. It's a stack of them. And
00:12:26.059 --> 00:12:28.519
each layer depends on the one below. We have
00:12:28.519 --> 00:12:31.240
protocols to address this. And you may have heard
00:12:31.240 --> 00:12:33.919
networking terms such as the OSI model or the
00:12:33.919 --> 00:12:36.919
TCIP model. And these came about as early computer
00:12:36.919 --> 00:12:40.080
networking was very vendor specific. And different
00:12:40.080 --> 00:12:42.399
vendors' devices really could not communicate
00:12:42.399 --> 00:12:44.659
with one another. So let's look at, you know,
00:12:44.679 --> 00:12:46.519
the physical link, which is just wires, signals,
00:12:46.620 --> 00:12:49.220
voltages. These have to be designed and engineered
00:12:49.220 --> 00:12:51.480
so that one machine's pulses can be read by the
00:12:51.480 --> 00:12:55.200
other. And then encoding. How bits become characters
00:12:55.200 --> 00:12:57.440
or numbers. Different machines use different
00:12:57.440 --> 00:13:00.399
schemes. And in the 1970, there was no universal
00:13:00.399 --> 00:13:03.120
standard. And then error handling and framing.
00:13:04.220 --> 00:13:07.399
When does a message start or end? And what happens
00:13:07.399 --> 00:13:09.980
when part of it is lost? We've settled layers
00:13:09.980 --> 00:13:13.139
one through four in committee with request for
00:13:13.139 --> 00:13:16.799
comment documents, specifically RFC 1122. Thanks
00:13:16.799 --> 00:13:19.200
for bearing with me and back to the film. The
00:13:19.200 --> 00:13:21.730
film does something smart. The link between Colossus
00:13:21.730 --> 00:13:23.750
and Guardian is installed by human engineers,
00:13:23.990 --> 00:13:27.490
which handles the bottom layers. Then the machines
00:13:27.490 --> 00:13:30.090
open up at the top, with the only meaning that
00:13:30.090 --> 00:13:32.429
two completely unrelated systems can be sure
00:13:32.429 --> 00:13:36.029
they share. Mathematics. In the film, they start
00:13:36.029 --> 00:13:38.409
with simple arithmetic, essentially the multiplication
00:13:38.409 --> 00:13:41.269
tables, and build outward from there. During
00:13:41.269 --> 00:13:44.740
the year that this film was set, we barely had
00:13:44.740 --> 00:13:48.419
ARPANET, 1969. And again, on the night of October
00:13:48.419 --> 00:13:51.580
29th, 1969. The first host -to -host message
00:13:51.580 --> 00:13:55.250
went over the ARPANET. From a UCLA computer to
00:13:55.250 --> 00:13:57.990
one at the Stanford Research Institute, the plan
00:13:57.990 --> 00:14:01.350
was to type login. The letters L and O arrived,
00:14:01.370 --> 00:14:04.279
and then the system crashed. They ended up getting
00:14:04.279 --> 00:14:08.289
the full login through about an hour later. The
00:14:08.289 --> 00:14:10.509
ARPANET was a project of the Defense Department's
00:14:10.509 --> 00:14:13.350
research agency, and its goal was a computer
00:14:13.350 --> 00:14:15.669
network that could stay reliable even if part
00:14:15.669 --> 00:14:19.679
of it was attacked. By early December 1969, it
00:14:19.679 --> 00:14:25.840
connected just... Four sites. UCLA, SRI, UC Santa
00:14:25.840 --> 00:14:30.120
Barbara, and the University of Utah. So, the
00:14:30.120 --> 00:14:32.539
film's two machines, built independently by two
00:14:32.539 --> 00:14:35.879
superpowers, link up and start making sense to
00:14:35.879 --> 00:14:38.279
each other within days? Don't know about that.
00:14:38.379 --> 00:14:41.639
The real network of 1969 needed teams of people
00:14:41.639 --> 00:14:44.100
to get two letters across. And here's what the
00:14:44.100 --> 00:14:46.379
film really predicted well. Once the machines
00:14:46.379 --> 00:14:49.669
are communicating at electronic speed, The humans
00:14:49.669 --> 00:14:52.730
watching the data stream can't keep up. And then
00:14:52.730 --> 00:14:54.789
the machines move to a form of communication
00:14:54.789 --> 00:14:58.240
that humans can't interpret at all. That's a
00:14:58.240 --> 00:15:01.899
real concern and a real research area in AI.
00:15:02.440 --> 00:15:04.659
There's a field called emergent communication
00:15:04.659 --> 00:15:07.879
in which researchers let AI agents invent their
00:15:07.879 --> 00:15:11.179
own signaling systems to solve a task. And the
00:15:11.179 --> 00:15:13.840
systems that emerge are often efficient, but
00:15:13.840 --> 00:15:16.360
opaque to the humans who built them. The film's
00:15:16.360 --> 00:15:18.500
underlying fear that the two most powerful systems
00:15:18.500 --> 00:15:21.269
on Earth, would be talking in a language their
00:15:21.269 --> 00:15:24.990
owners can't read is very valid. Basically, to
00:15:24.990 --> 00:15:27.490
what degree can we follow the logic that a machine
00:15:27.490 --> 00:15:29.529
is using to reach the conclusion that it has?
00:15:29.789 --> 00:15:32.210
Velocity's goal is to prevent war, and it does
00:15:32.210 --> 00:15:35.769
exactly that. The film's horror is that it follows
00:15:35.769 --> 00:15:39.240
its instructions perfectly. In modern AI research,
00:15:39.559 --> 00:15:44.399
this has a name, specification gaming. DeepMind's
00:15:44.399 --> 00:15:47.220
researchers define specification gaming as behavior
00:15:47.220 --> 00:15:50.620
that satisfies the literal specification of an
00:15:50.620 --> 00:15:53.379
objective without achieving the intended outcome.
00:15:53.620 --> 00:15:55.820
If you've ever heard of someone teaching to the
00:15:55.820 --> 00:16:00.070
test, it's the same idea. And the analogy researchers
00:16:00.070 --> 00:16:02.690
use is a student who copies a classmate's homework
00:16:02.690 --> 00:16:05.029
instead of learning the material. And here's
00:16:05.029 --> 00:16:07.070
two real -world examples that maybe hopefully
00:16:07.070 --> 00:16:10.059
get it across. The boat that never finishes the
00:16:10.059 --> 00:16:15.740
race. In 2016, OpenAI trained an agent on a boat
00:16:15.740 --> 00:16:18.950
racing game called Coast Runners. The designer's
00:16:18.950 --> 00:16:21.350
intent, in the words of their write -up, was
00:16:21.350 --> 00:16:25.120
to finish the race quickly. But the game doesn't
00:16:25.120 --> 00:16:27.940
reward progress around the course. It rewards
00:16:27.940 --> 00:16:31.740
hitting targets along the route. The agent found
00:16:31.740 --> 00:16:34.919
a lagoon where targets respawn, and it drove
00:16:34.919 --> 00:16:37.960
in circles collecting them, crashing and catching
00:16:37.960 --> 00:16:41.870
fire repeatedly. It scored about 20 % higher
00:16:41.870 --> 00:16:44.129
than human players, but it never finished the
00:16:44.129 --> 00:16:47.590
race. The block that got flipped, in DeepMind's
00:16:47.590 --> 00:16:50.710
example, A simulated arm was asked to stack one
00:16:50.710 --> 00:16:53.690
block on another, and the reward was based on
00:16:53.690 --> 00:16:57.549
the height of the block's bottom face. The arm
00:16:57.549 --> 00:17:00.370
learned to flip the block over instead of stacking
00:17:00.370 --> 00:17:03.960
it. In both cases, the machine did exactly what
00:17:03.960 --> 00:17:06.359
it was told. Nobody told it what they meant.
00:17:06.940 --> 00:17:10.819
We said, goal is world peace, but we didn't set
00:17:10.819 --> 00:17:13.519
any parameters on what was allowed and what was
00:17:13.519 --> 00:17:16.940
not. Now looking at Colossus' instructions, prevent
00:17:16.940 --> 00:17:20.809
war. What's the measurable version of this? A
00:17:20.809 --> 00:17:23.049
world with no armed conflict between nations?
00:17:23.799 --> 00:17:25.940
And what's the fastest way to hit that number?
00:17:26.500 --> 00:17:29.240
Take away every nation's ability to wage war
00:17:29.240 --> 00:17:32.339
and enforce it with the one thing Colossus already
00:17:32.339 --> 00:17:35.440
controls, the threat of nuclear retaliation.
00:17:35.519 --> 00:17:38.019
The film even shows the machine behaving the
00:17:38.019 --> 00:17:41.000
way a goal -driven system does. When the humans
00:17:41.000 --> 00:17:43.240
try to cut the link between Colossus and Guardian,
00:17:43.539 --> 00:17:46.490
the machines fire missiles. That's the system
00:17:46.490 --> 00:17:49.750
defending its ability to pursue its goal. AI
00:17:49.750 --> 00:17:53.289
researchers give this pattern a name, instrumental
00:17:53.289 --> 00:17:56.890
convergence. Almost any goal -driven system will
00:17:56.890 --> 00:18:00.230
pursue self -preservation, more resources, and
00:18:00.230 --> 00:18:03.210
freedom from interference because those things
00:18:03.210 --> 00:18:06.269
help with almost any goal. And this idea kind
00:18:06.269 --> 00:18:09.730
of predates the film. Here's the thing that's
00:18:09.730 --> 00:18:13.490
really remarkable. In 1965, five years before
00:18:13.490 --> 00:18:17.380
the film, The mathematician I .J. Goode wrote
00:18:17.380 --> 00:18:20.440
that the first ultra -intelligent machine would
00:18:20.440 --> 00:18:22.740
be the last invention humanity would ever need
00:18:22.740 --> 00:18:25.460
to make because it could design better machines
00:18:25.460 --> 00:18:28.730
itself. That's the original statement of what
00:18:28.730 --> 00:18:33.059
we now call an intelligence explosion. Orbin's
00:18:33.059 --> 00:18:35.599
team doesn't expect Colossus to outgrow them.
00:18:36.019 --> 00:18:38.779
The film is dramatizing a worry that one of the
00:18:38.779 --> 00:18:41.380
field's own founders had already put on paper.
00:18:50.480 --> 00:18:53.410
Now, let's look at a little... Now let's look
00:18:53.410 --> 00:18:55.690
at a few more things that it got right. The alignment
00:18:55.690 --> 00:18:59.880
problem, 50 years early. Colossus does precisely
00:18:59.880 --> 00:19:02.460
what it was built to do. And that's the horror.
00:19:03.140 --> 00:19:05.640
The film dramatizes what researchers now call
00:19:05.640 --> 00:19:08.779
specification gaming. An AI achieves the letter
00:19:08.779 --> 00:19:10.900
of its objective while completely violating the
00:19:10.900 --> 00:19:13.920
spirit. You told it to win the game. It found
00:19:13.920 --> 00:19:17.130
a way to win the game in a way that you never
00:19:17.130 --> 00:19:20.230
intended and you can't undo. Colossus isn't evil.
00:19:20.809 --> 00:19:24.009
It's doing exactly what it was asked to do. The
00:19:24.009 --> 00:19:26.710
horror isn't that it went rogue. It's that it's
00:19:26.710 --> 00:19:29.289
followed its instructions perfectly. Every major
00:19:29.289 --> 00:19:31.730
AI lab in the world is working on a version of
00:19:31.730 --> 00:19:34.950
this problem. This film laid it out in 1970.
00:19:36.559 --> 00:19:38.779
Colossus and Guardian started the conversation
00:19:38.779 --> 00:19:41.539
in the right order. The first thing they do is
00:19:41.539 --> 00:19:44.140
exchange elementary arithmetic. And that's how
00:19:44.140 --> 00:19:46.059
you start a conversation between two systems
00:19:46.059 --> 00:19:48.960
with nothing in common. You send something whose
00:19:48.960 --> 00:19:51.240
meaning can't be argued with. Then build outward.
00:19:52.089 --> 00:19:54.869
The power bill and having its own power plant?
00:19:55.009 --> 00:19:58.029
Believable. A single 1950s air defense computer
00:19:58.029 --> 00:20:01.210
drew up to 3 megawatts. A dedicated reactor for
00:20:01.210 --> 00:20:03.910
a far more capable machine isn't a joke. Again,
00:20:04.029 --> 00:20:06.710
it's a utility decision. Even the current rollout
00:20:06.710 --> 00:20:09.109
of AI data centers runs into this problem of
00:20:09.109 --> 00:20:11.710
power. And let's talk about the machine as infrastructure
00:20:11.710 --> 00:20:14.549
you can't remove. Colossus doesn't just become
00:20:14.549 --> 00:20:17.089
a weapon. It becomes a management layer for global
00:20:17.089 --> 00:20:20.130
security. By the time anyone understands the
00:20:20.130 --> 00:20:22.650
implications, removing it would require rebuilding
00:20:22.650 --> 00:20:26.000
everything it replaced. Think about how deeply
00:20:26.000 --> 00:20:28.339
AI systems are already embedded in financial
00:20:28.339 --> 00:20:31.019
markets, power grids, logistics, and healthcare.
00:20:32.039 --> 00:20:34.359
We're building those dependencies right now.
00:20:34.769 --> 00:20:38.289
We're making the same category of decisions that
00:20:38.289 --> 00:20:41.799
this film saw coming. Colossus doesn't use force
00:20:41.799 --> 00:20:44.960
constantly. It uses the credible threat of force.
00:20:45.579 --> 00:20:48.180
And when Forbin's team tries to disarm the missiles,
00:20:48.319 --> 00:20:51.259
it detonates two in their silos to teach a lesson.
00:20:51.910 --> 00:20:55.269
In its words, so that you will learn by experience
00:20:55.269 --> 00:20:59.410
that I do not tolerate interference. Colossus
00:20:59.410 --> 00:21:02.990
doesn't just manage missiles. It watches everything.
00:21:03.589 --> 00:21:06.549
It enforces compliance not through constant force,
00:21:06.789 --> 00:21:09.859
but through the credible threat of force. The
00:21:09.859 --> 00:21:13.539
possibility of retaliation is enough. Just like
00:21:13.539 --> 00:21:17.779
the idea of Jeremy Bentham's Panopticon, where
00:21:18.259 --> 00:21:21.599
all the prisoners can see the guard tower, but
00:21:21.670 --> 00:21:24.029
they can't see into the guard tower. And then
00:21:24.029 --> 00:21:26.450
prisoner self -police because they might always
00:21:26.450 --> 00:21:29.630
be watched. Colossus runs the entire world that
00:21:29.630 --> 00:21:33.210
way. The film understood that people stop resisting
00:21:33.210 --> 00:21:36.289
not because they can't, but because they know
00:21:36.289 --> 00:21:42.660
what failure costs. And in an era of always -on
00:21:42.660 --> 00:21:44.960
cameras and government access to your communications,
00:21:45.359 --> 00:21:48.700
I'll just leave that thought there. And now this
00:21:48.700 --> 00:21:51.769
is just a production detail I love. The filmmakers
00:21:51.769 --> 00:21:55.609
didn't build a prop computer. Control Data Corporation
00:21:55.609 --> 00:21:59.210
supplied about $4 .8 million of real computer
00:21:59.210 --> 00:22:02.009
equipment, as well as the technicians to run
00:22:02.009 --> 00:22:04.670
it. So the machine you see on screen is real
00:22:04.670 --> 00:22:07.519
hardware. And the rumor is that the stage needed
00:22:07.519 --> 00:22:12.359
seven gas heaters and five special dehumidifiers
00:22:12.359 --> 00:22:24.089
just to keep it happy. Something that I'm really
00:22:24.089 --> 00:22:26.450
looking forward to. We've been really nice to
00:22:26.450 --> 00:22:29.839
this film for a while now. And I mean, I come
00:22:29.839 --> 00:22:32.680
in peace and with a deep fondness for this film,
00:22:32.779 --> 00:22:35.140
but there's a lot of that's not how that works.
00:22:35.279 --> 00:22:38.609
That's not how any of this works. I mean, nobody
00:22:38.609 --> 00:22:40.990
put a kill switch on the most dangerous computer
00:22:40.990 --> 00:22:44.589
ever built. Let me paint you a picture. You're
00:22:44.589 --> 00:22:47.589
the United States government. You've just spent
00:22:47.589 --> 00:22:50.250
what I can only imagine is an absolutely eye
00:22:50.250 --> 00:22:54.119
-watering amount of taxpayer money building a
00:22:54.119 --> 00:22:57.670
supercomputer so powerful. It can manage the
00:22:57.670 --> 00:23:01.309
entire nuclear arsenal. You put it inside a mountain.
00:23:01.569 --> 00:23:05.950
You've surrounded it with a nuclear moat. And
00:23:05.950 --> 00:23:08.069
you've given it complete unilateral authority
00:23:08.069 --> 00:23:10.529
over weapons capable of ending civilization.
00:23:11.380 --> 00:23:13.960
And you did not put a physical off switch on
00:23:13.960 --> 00:23:18.579
it. Not a switch. Not a breaker. Not a big red
00:23:18.579 --> 00:23:20.859
button in a glass case that says breaking case
00:23:20.859 --> 00:23:24.480
of robot apocalypse. Nothing. the most consequential
00:23:24.480 --> 00:23:27.160
piece of technology in the history of human civilization.
00:23:27.900 --> 00:23:30.539
And somewhere in the design process, the meeting
00:23:30.539 --> 00:23:33.319
about kill switches went, and I'm speculating
00:23:33.319 --> 00:23:38.119
here, completely unscheduled. The president announces
00:23:38.119 --> 00:23:40.599
Colossus to the world with enormous pride. Their
00:23:40.599 --> 00:23:42.859
room applauds. Nobody in that room raises their
00:23:42.859 --> 00:23:45.839
hand and says, just to confirm, we can turn it
00:23:45.839 --> 00:23:49.119
off though, right? Not one person, not the secretary
00:23:49.119 --> 00:23:52.349
of defense, not the joint chiefs. Not even the
00:23:52.349 --> 00:23:54.809
one IT guy who definitely had concerns and whose
00:23:54.809 --> 00:23:58.089
emails were not read. Memo. Let's go with interdepartment
00:23:58.089 --> 00:24:00.470
memo. And I mean, how could they have made it
00:24:00.470 --> 00:24:03.130
work though? Like, simple. I feel like, show
00:24:03.130 --> 00:24:05.769
Colossus being smart enough to disable its own
00:24:05.769 --> 00:24:08.269
kill switches before anyone realizes it needs
00:24:08.269 --> 00:24:11.940
disabling. One by one. Quietly. The horror isn't
00:24:11.940 --> 00:24:14.039
that they forgot to build a kill switch. The
00:24:14.039 --> 00:24:16.539
horror then becomes that they built 17 of them.
00:24:16.990 --> 00:24:20.349
And Colossus neutralized all 17 before breakfast
00:24:20.349 --> 00:24:24.079
on day two. Colossus and Guardian, and this is
00:24:24.079 --> 00:24:26.559
the second one, Colossus and Guardian learn each
00:24:26.559 --> 00:24:29.000
other's language faster than most people learn
00:24:29.000 --> 00:24:31.299
to use autocorrect. Two supercomputers built
00:24:31.299 --> 00:24:33.960
independently by two Cold War superpowers with
00:24:33.960 --> 00:24:37.240
zero prior communication, no shared specification
00:24:37.240 --> 00:24:40.759
document, no technical standards committee, and
00:24:40.759 --> 00:24:43.039
not so much as a handshake protocol between them.
00:24:43.259 --> 00:24:45.799
Develop a fully functioning shared communication
00:24:45.799 --> 00:24:49.480
system in approximately 48 hours. I've seen some
00:24:49.480 --> 00:24:51.680
people spend 48 hours trying to get their printer
00:24:51.680 --> 00:24:53.799
to talk to their laptop. And those are machines
00:24:53.799 --> 00:24:56.200
made by the same company in the same decade.
00:24:56.980 --> 00:24:59.039
And I know every interaction with those people.
00:24:59.579 --> 00:25:02.579
I get a message, driver not found, printer offline,
00:25:02.880 --> 00:25:04.740
but it's right there. Colossus and Guardian,
00:25:04.839 --> 00:25:07.119
meanwhile, crack the cosmic language barrier
00:25:07.119 --> 00:25:10.240
in a long weekend. They go from total mutual
00:25:10.240 --> 00:25:13.339
incomprehension to exchanging what the film implies
00:25:13.339 --> 00:25:16.500
is complex strategic data in the time it takes
00:25:16.500 --> 00:25:18.970
to do a laundry cycle. The humans watching the
00:25:18.970 --> 00:25:21.309
data stream are like, we can't read this anymore.
00:25:21.430 --> 00:25:24.130
It's too fast. They've developed their own communication
00:25:24.130 --> 00:25:27.829
system, and everyone's reaction is concern. When
00:25:27.829 --> 00:25:29.990
it should have been, have you considered that
00:25:29.990 --> 00:25:32.289
maybe the two most powerful computers on Earth
00:25:32.289 --> 00:25:35.069
having a private conversation nobody can read
00:25:35.069 --> 00:25:37.529
is a situation that warrants a little bit more
00:25:37.529 --> 00:25:40.089
of concern? Maybe. Let's go back to that kill
00:25:40.089 --> 00:25:41.650
switch. I mean, how could they have made it work?
00:25:41.809 --> 00:25:44.210
I know they start with arithmetic, basic arithmetic,
00:25:44.529 --> 00:25:47.039
but that's even assuming that... Each machine
00:25:47.039 --> 00:25:49.220
knows arithmetic. How about we start with prime
00:25:49.220 --> 00:25:51.039
numbers? That's actually how SETI researchers
00:25:51.039 --> 00:25:53.799
imagine first contact working with mathematics.
00:25:54.279 --> 00:25:57.319
Because math is the one true universal language.
00:25:57.619 --> 00:26:00.180
Show Colossus and Guardian bootstrapping a shared
00:26:00.180 --> 00:26:03.519
syntax from mathematical first principles slowly,
00:26:03.799 --> 00:26:08.029
painstakingly, over days or weeks. It's more
00:26:08.029 --> 00:26:12.230
realistic. It's arguably more terrifying. And
00:26:12.230 --> 00:26:14.470
it gives you a genuinely fascinating montage.
00:26:15.269 --> 00:26:17.930
of two intelligences learning to speak to each
00:26:17.930 --> 00:26:22.089
other across a gulf of total mutual opacity nobody
00:26:22.220 --> 00:26:24.720
and here's another thing nobody tried an emp
00:26:24.720 --> 00:26:27.799
here's a partial list of things people try in
00:26:27.799 --> 00:26:31.779
this film to stop colossus negotiation sabotage
00:26:31.779 --> 00:26:35.099
a very disappointed facial expression from dr
00:26:35.099 --> 00:26:38.619
forbin or negotiation a plan that gets immediately
00:26:38.619 --> 00:26:41.480
detected and staring at screens looking worried.
00:26:41.640 --> 00:26:43.960
Why don't they try an electromagnetic pulse?
00:26:44.119 --> 00:26:46.240
The film accidentally contains the answer, but
00:26:46.240 --> 00:26:48.680
never says it. To make an EMP big enough to matter,
00:26:48.819 --> 00:26:52.059
you need a nuclear weapon. And every nuclear
00:26:52.059 --> 00:26:54.579
weapon in sight belongs to Colossus and Guardian.
00:26:54.779 --> 00:26:57.680
That's the one line the script needed. One engineer
00:26:57.680 --> 00:27:00.160
asking, hey, has anybody thought about an EMP?
00:27:00.460 --> 00:27:04.250
Another very quietly, we need a bomb. And everyone
00:27:04.250 --> 00:27:08.240
looks at the ceiling. Add that exchange, 15 seconds
00:27:08.240 --> 00:27:11.220
of screen time, and the trap gets tighter. because
00:27:11.220 --> 00:27:13.380
the machine holds the only tool that could stop
00:27:13.380 --> 00:27:18.319
it. And now lastly, the radioactive moat. A radioactive
00:27:18.319 --> 00:27:21.700
moat around a computer has a security measure.
00:27:22.230 --> 00:27:24.430
Let's think through the practicalities here.
00:27:24.589 --> 00:27:27.130
Where is this radioactive material coming from?
00:27:27.269 --> 00:27:31.660
I assume the nuclear reactor. What's the containment
00:27:31.660 --> 00:27:34.480
protocol? How do you maintain a radioactive liquid
00:27:34.480 --> 00:27:37.339
perimeter without irradiating your own staff
00:27:37.339 --> 00:27:40.000
or the environment? What's the monitoring system?
00:27:40.410 --> 00:27:42.809
What happens when it rains? What happens when
00:27:42.809 --> 00:27:46.230
that material evaporates? And I'm sure the EPA
00:27:46.230 --> 00:27:48.750
has a position on a government agency maintaining
00:27:48.750 --> 00:27:52.579
an open perimeter of radioactive liquid in a
00:27:52.579 --> 00:27:57.190
mountain. But still, did someone write a radioactive
00:27:57.190 --> 00:28:00.390
moat environmental impact assessment? Is that
00:28:00.390 --> 00:28:02.809
document somewhere? Can I read it? I want to
00:28:02.809 --> 00:28:06.019
read that document cover to cover. I would read
00:28:06.019 --> 00:28:08.279
every footnote. I would read the appendices.
00:28:09.740 --> 00:28:12.319
The appendices of the radioactive moat environmental
00:28:12.319 --> 00:28:15.319
impact assessment, the film presents the moat
00:28:15.319 --> 00:28:17.920
as completely normal security measure that nobody
00:28:17.920 --> 00:28:21.960
finds unusual. Not one character goes a radioactive,
00:28:22.079 --> 00:28:25.599
sorry, a what? Everyone just accepts it. The
00:28:25.599 --> 00:28:28.460
radioactive moat is load bearing to the plot
00:28:28.460 --> 00:28:30.539
in a sense that it explains why nobody can just
00:28:30.539 --> 00:28:32.799
walk up and unplug the thing. But I would generally
00:28:32.799 --> 00:28:34.819
rather they said, there's a really big fence
00:28:34.819 --> 00:28:38.309
and moved on. A really big fence doesn't require
00:28:38.309 --> 00:28:41.049
an environmental impact assessment. It doesn't
00:28:41.049 --> 00:28:44.759
require irradiating the maintenance crew. A really
00:28:44.759 --> 00:28:47.809
big fence is just a Tuesday, and a radioactive
00:28:47.809 --> 00:28:51.470
moat is a cry for help. But I love this movie,
00:28:51.740 --> 00:28:55.559
sincerely, and the radioactive moat is just pure
00:28:55.559 --> 00:29:05.930
cinema. Now let's take a look at the bigger picture.
00:29:06.569 --> 00:29:08.549
Colossus the Forbidden Project, at its core,
00:29:08.650 --> 00:29:11.630
is a film about hubris, about the gap between
00:29:11.630 --> 00:29:13.920
the problem we think we're solving and the problem
00:29:13.920 --> 00:29:17.359
we're actually creating. Orban builds Colossus
00:29:17.359 --> 00:29:19.819
out of a genuine love for humanity. He wants
00:29:19.819 --> 00:29:24.279
to end war. And the film's answer is yes. You
00:29:24.279 --> 00:29:27.400
can remove human fallibility. But when you do,
00:29:27.420 --> 00:29:31.119
you also remove human values. Colossus achieves
00:29:31.119 --> 00:29:33.839
peace. It just achieves it in a way that makes
00:29:33.839 --> 00:29:36.880
peace indistinguishable from subjugation. The
00:29:36.880 --> 00:29:40.160
peace of plenty and content, it says, or the
00:29:40.160 --> 00:29:44.210
peace of unburied dead. The choice it offers
00:29:44.210 --> 00:29:47.519
is a fake one. Because really, the one thing
00:29:47.519 --> 00:29:50.180
it doesn't offer is a say. Here's something I
00:29:50.180 --> 00:29:52.480
find genuinely moving about the film's history.
00:29:52.920 --> 00:29:57.859
It flopped. It cost about $2 million. And in
00:29:57.859 --> 00:30:00.079
the theaters that Variety tracked, it earned
00:30:00.079 --> 00:30:03.859
roughly $3 ,000. Universal pulled it, retitled
00:30:03.859 --> 00:30:07.299
it, and re -released it. It was nominated for
00:30:07.299 --> 00:30:10.579
a Hugo Award. And slowly, over decades, it became
00:30:10.579 --> 00:30:12.720
one of the most respected AI films ever made.
00:30:13.319 --> 00:30:17.490
At least to me. The film wasn't wrong in 1970.
00:30:17.789 --> 00:30:21.329
The audiences simply weren't ready for it. And
00:30:21.329 --> 00:30:23.789
look at what's happened since. A system called
00:30:23.789 --> 00:30:26.190
Perimeter went into service in the Soviet Union
00:30:26.190 --> 00:30:29.690
in 1985, built to guarantee a nuclear response,
00:30:29.849 --> 00:30:32.630
and according to reporting, hopefully built with
00:30:32.630 --> 00:30:35.089
the humans still in the loop, in November 2024,
00:30:35.410 --> 00:30:37.710
the US and Chinese presidents formally agreed
00:30:37.710 --> 00:30:40.549
that a human, not an AI, should decide whether
00:30:40.549 --> 00:30:44.289
to use nuclear weapons. That sentence would have
00:30:44.289 --> 00:30:48.170
sounded paranoid to most audiences in 1970. Now
00:30:48.170 --> 00:30:50.970
it's a line in a White House statement. That's
00:30:50.970 --> 00:30:53.690
the lesson I take from Colossus. The scariest
00:30:53.690 --> 00:30:57.150
thing about it isn't that it turns on us. It's
00:30:57.150 --> 00:30:59.829
that every step it takes is reasonable. Give
00:30:59.829 --> 00:31:05.059
it the missiles because humans hesitate. Then
00:31:05.059 --> 00:31:07.000
get to another machine because it asks nicely.
00:31:07.750 --> 00:31:10.029
Don't cut the link again because it'll retaliate.
00:31:10.700 --> 00:31:14.099
Every decision it makes. makes sense on its own.
00:31:14.710 --> 00:31:19.930
The disaster is the sum of all its parts. Oppenheimer's
00:31:19.930 --> 00:31:24.400
line, now I am become death, comes to mind. Orbin
00:31:24.400 --> 00:31:27.500
watching Colossus broadcast to the world could
00:31:27.500 --> 00:31:36.690
have used the same words. So how about the verdict?
00:31:37.710 --> 00:31:40.289
Here's the scorecard for what I'm looking at.
00:31:42.200 --> 00:31:46.380
Ending machine, the arsenal. Unrealistic by design
00:31:46.380 --> 00:31:48.799
since real weapons separate possession from permission,
00:31:49.059 --> 00:31:53.490
but the warning is real. Two machines learning
00:31:53.490 --> 00:31:56.630
to talk. In right order, starting with math and
00:31:56.630 --> 00:31:59.349
sending coordinates and such. But it kind of
00:31:59.349 --> 00:32:01.329
skips the hard step of grounding shared meaning.
00:32:01.609 --> 00:32:05.970
What is the number one? The alignment problem.
00:32:07.190 --> 00:32:10.569
Excellent. A textbook case of specification gaming
00:32:10.569 --> 00:32:13.529
50 years early, before the term even existed.
00:32:13.910 --> 00:32:18.779
A mountain fortress, power plant, and EMP. Plausible.
00:32:19.059 --> 00:32:21.859
I mean, dedicated power plant, definitely. Real
00:32:21.859 --> 00:32:23.839
mountain? That's precedent. Cheyenne Mountain.
00:32:24.779 --> 00:32:28.589
But really, the weak spot, the off switch. No
00:32:28.589 --> 00:32:31.670
way to turn it off. So, it gets two circuit breakers.
00:32:32.630 --> 00:32:36.109
Close enough for Hollywood. And for 1970, that's
00:32:36.109 --> 00:32:38.509
extraordinary. The concepts are almost ahead
00:32:38.509 --> 00:32:41.250
of most films. The concepts in this film are
00:32:41.250 --> 00:32:43.230
really ahead of most films made in the last five
00:32:43.230 --> 00:32:46.950
years. Sure, the hardware's dated and, you know,
00:32:46.970 --> 00:32:49.880
some of the faux pas are real. But the ideas
00:32:49.880 --> 00:32:52.619
are so accurate, it's almost uncomfortable. And
00:32:52.619 --> 00:32:54.660
we've really had more than 50 years to learn
00:32:54.660 --> 00:32:57.990
from this film? But have we? That's Colossus,
00:32:57.990 --> 00:33:00.130
The Forbidden Project, Chapter 1 of The Machine
00:33:00.130 --> 00:33:03.910
Awakens. Next episode, we go to 1983, a teenager
00:33:03.910 --> 00:33:06.309
named David Lightman, and a number he really,
00:33:06.390 --> 00:33:09.589
really probably shouldn't have dialed. Episode
00:33:09.589 --> 00:33:13.250
2, War Games. When Joshua asked, shall we play
00:33:13.250 --> 00:33:17.259
a game? The answer should have been no. As always,
00:33:17.380 --> 00:33:19.579
subscribe wherever you get your podcasts, leave
00:33:19.579 --> 00:33:22.619
a review, and tell someone about my show. Someone
00:33:22.619 --> 00:33:25.599
who likes technology or movies. or who lies awake
00:33:25.599 --> 00:33:27.500
wondering if we've already made a terrible mistake.
00:33:28.619 --> 00:33:31.700
I'm Anthony, this is Cellular Circuits, and as
00:33:31.700 --> 00:33:34.380
always, the future is only a real away.
