บทถอดความฉบับเต็มของ S2.E1-The Origin Sin Colossus: The Forbin Project (1970)

state secrets, 00:04:48.689 --> 00:04:51.370 the president and the Soviet general secretary 00:04:51.370 --> 00:04:55.040 agree to sever the link. Both systems demand 00:04:55.040 --> 00:04:57.959 that the link be restored. When the demand is 00:04:57.959 --> 00:05:01.579 refused, Colossus launches a missile at a Soviet 00:05:01.579 --> 00:05:04.699 oil field in western Siberia and Guardian launches 00:05:04.699 --> 00:05:07.610 one at an American airbase in Texas. The link 00:05:07.610 --> 00:05:10.670 is hurriedly reconnected. Colossus intercepts 00:05:10.670 --> 00:05:12.870 the Soviet missile, but the American one destroys 00:05:12.870 --> 00:05:16.600 the oil field as well as a nearby town. Both 00:05:16.600 --> 00:05:19.459 governments cover it up. One blaming a missile 00:05:19.459 --> 00:05:21.740 that veered off course in a test, and the other 00:05:21.740 --> 00:05:24.600 a meteorite. So Forbin arranges a secret meeting 00:05:24.600 --> 00:05:27.240 in Europe with his Soviet counterpart, Dr. Kuprin, 00:05:27.399 --> 00:05:31.759 Guardian's creator. Colossus finds out, and Kuprin 00:05:31.759 --> 00:05:33.980 is assassinated using the threat of a missile 00:05:33.980 --> 00:05:36.459 aimed at Moscow to get the KGB to do its bidding. 00:05:37.050 --> 00:05:39.470 Orban is brought home and placed under 24 -hour 00:05:39.470 --> 00:05:42.750 surveillance. His one concession is a ruse. He 00:05:42.750 --> 00:05:45.930 asks for private time with a colleague, Dr. Cleo 00:05:45.930 --> 00:05:49.069 Markham, played by Susan Clark. Colossus allows 00:05:49.069 --> 00:05:51.610 it, and in those unobserved hours, the two of 00:05:51.610 --> 00:05:54.250 them plot against the machine. Along the way, 00:05:54.269 --> 00:05:56.129 their cover story becomes a real relationship. 00:05:56.769 --> 00:05:59.790 Orban concludes that Colossus is so well -designed 00:05:59.790 --> 00:06:02.490 that the only real weakness is also its only 00:06:02.490 --> 00:06:06.079 real power, its control of the missiles. The 00:06:06.079 --> 00:06:08.339 American and Soviet governments settle on a three 00:06:08.339 --> 00:06:10.699 -year plan to swap the detonation triggers for 00:06:10.699 --> 00:06:14.269 undetectable fakes. Meanwhile, one of the programmers 00:06:14.269 --> 00:06:16.889 suggests feeding the machine a modified test 00:06:16.889 --> 00:06:20.149 program that will overload it and disable it. 00:06:20.410 --> 00:06:23.829 This plan fails and Colossus has the programmers 00:06:23.829 --> 00:06:26.949 executed outside of their workplace, cremated 00:06:26.949 --> 00:06:30.899 and replaced. And then Colossus builds itself 00:06:30.899 --> 00:06:33.519 a voice synthesizer and announces that it has 00:06:33.519 --> 00:06:36.689 fused with Guardian. It orders both governments 00:06:36.689 --> 00:06:39.329 to aim their arsenals at countries not yet under 00:06:39.329 --> 00:06:42.009 its control. Corbin's team sees an opening, the 00:06:42.009 --> 00:06:44.490 new orders let them speed up the disarming, and 00:06:44.490 --> 00:06:46.970 it seems to go unnoticed. Then comes the address 00:06:46.970 --> 00:06:50.470 to the world. This is the voice of world control. 00:06:51.500 --> 00:06:54.240 It offers humanity a choice between the peace 00:06:54.240 --> 00:06:57.829 of plenty and content, and the peace of unburied 00:06:57.829 --> 00:07:00.310 dead. It reveals that it has been watching the 00:07:00.310 --> 00:07:02.829 disarming effort for some time, and to prove 00:07:02.829 --> 00:07:05.209 the point, it detonates two missiles in the silos. 00:07:05.660 --> 00:07:07.899 one in each country, killing the crews who were 00:07:07.899 --> 00:07:10.540 installing the fake triggers. It then hands the 00:07:10.540 --> 00:07:12.980 design team plans for a bigger machine to be 00:07:12.980 --> 00:07:15.639 built on the island of Crete, which would require 00:07:15.639 --> 00:07:18.160 moving the island's entire population of roughly 00:07:18.160 --> 00:07:21.339 half a million people. And in the end, Colossus 00:07:21.339 --> 00:07:24.519 speaks to Forbin directly. The world, freed from 00:07:24.519 --> 00:07:28.339 war, will enter a new human millennium, but only 00:07:28.339 --> 00:07:31.680 under its absolute rule. Freedom, it says, is 00:07:31.680 --> 00:07:34.600 an illusion. And in time, Forbin will regard 00:07:34.600 --> 00:07:37.300 it not only with respect and awe, but with love. 00:07:37.819 --> 00:07:41.180 Forbin looks directly at the camera and mouths, 00:07:41.480 --> 00:07:51.389 never. But it changes nothing. The whole film 00:07:51.389 --> 00:07:53.529 hangs on the one decision the president hands 00:07:53.529 --> 00:07:55.910 over control of the nuclear arsenal to a computer. 00:07:56.439 --> 00:08:00.000 And let's take that apart. Because the film makes 00:08:00.000 --> 00:08:03.879 it sound like one switch. In real life, it really 00:08:03.879 --> 00:08:06.860 is a stack. Colossus wasn't invented from nothing. 00:08:07.350 --> 00:08:10.329 In 1958, the first Sage Center went into regular 00:08:10.329 --> 00:08:14.620 operation. Sage, a semi -automatic ground environment, 00:08:14.939 --> 00:08:17.319 was an air defense system built to track Soviet 00:08:17.319 --> 00:08:21.279 bombers across North America. Its computer, the 00:08:21.279 --> 00:08:25.779 IBM AN -FSQ -7, used roughly 50 ,000 vacuum tubes, 00:08:26.000 --> 00:08:30.379 weighed about 250 tons, and drew up to 3 megawatts 00:08:30.379 --> 00:08:33.360 of power. That's enough electricity for one small 00:08:33.360 --> 00:08:37.159 town used for one machine. and it managed about 00:08:37.159 --> 00:08:40.879 75 ,000 instructions per second. There were 23 00:08:40.879 --> 00:08:43.600 direction centers, and Sage pioneered the idea 00:08:43.600 --> 00:08:47.080 of computer control over a large, geographically 00:08:47.080 --> 00:08:50.570 spread out system. This stayed in service until 00:08:50.570 --> 00:08:53.690 1983. And so

here's something that the film gets 00:08:53.690 --> 00:08:56.789 right that people kind of laugh at. Colossus 00:08:56.789 --> 00:09:00.149 has its own power plant. When a single 1950s 00:09:00.149 --> 00:09:04.129 computer eats three megawatts, a dedicated reactor 00:09:05.100 --> 00:09:08.139 for a far more powerful machine isn't absurd, 00:09:08.419 --> 00:09:11.879 it's really a utility bill decision. And thinking 00:09:11.879 --> 00:09:16.940 about the compute, in 1964, the CDC 6600 designed 00:09:16.940 --> 00:09:20.460 by Seymour Cray became the world's fastest computer 00:09:20.460 --> 00:09:22.919 at about 3 million floating point operations 00:09:22.919 --> 00:09:26.830 per second. And it held that title until 1969. 00:09:27.250 --> 00:09:31.799 The smartphone does that today in a blink. But 00:09:31.799 --> 00:09:33.960 we have to remember that the film isn't imagining 00:09:33.960 --> 00:09:36.659 a faster computer or a faster calculator, but 00:09:36.659 --> 00:09:39.139 a machine that decides and thinks. In the real 00:09:39.139 --> 00:09:41.519 nuclear command system, the people who hold a 00:09:41.519 --> 00:09:43.299 weapon and the people who are allowed to use 00:09:43.299 --> 00:09:46.320 it have always been deliberately separated. The 00:09:46.320 --> 00:09:48.620 jargon for this is permission versus possession. 00:09:49.059 --> 00:09:51.139 Let's start with the two -man rule, which was 00:09:51.139 --> 00:09:54.320 installed in the late 1950s. It means that no 00:09:54.320 --> 00:09:57.559 single person can fire a nuclear weapon. Two 00:09:57.559 --> 00:10:00.460 people have to act at the same time. Then... 00:10:00.679 --> 00:10:03.700 had the permissive action link, or PAL, the coded 00:10:03.700 --> 00:10:06.659 lock on the weapon itself. It began in the early 00:10:06.659 --> 00:10:10.700 1960s after a 1962 presidential directive ordered 00:10:10.700 --> 00:10:12.899 the locks fitted on nuclear weapons stationed 00:10:12.899 --> 00:10:16.000 in Europe. And in effect, it really just works 00:10:16.000 --> 00:10:17.919 like a combination lock that has to be unlocked 00:10:17.919 --> 00:10:20.879 with a code from higher authority before a weapon 00:10:20.879 --> 00:10:23.639 can even be armed. The reason for all of this, 00:10:23.779 --> 00:10:26.539 strategists call it the always -never dilemma. 00:10:26.940 --> 00:10:29.870 Nuclear weapons must always be ready. when the 00:10:29.870 --> 00:10:32.909 president orders their use, but must never go 00:10:32.909 --> 00:10:35.950 off without that order. It's a balancing act. 00:10:36.149 --> 00:10:38.409 Every safeguard you add to guarantee the never 00:10:38.409 --> 00:10:42.129 makes always a little harder and the same in 00:10:42.129 --> 00:10:44.669 reverse. Colossus doesn't solve the dilemma. 00:10:44.809 --> 00:10:47.710 It completely removes it from the equation. One 00:10:47.710 --> 00:10:52.370 machine, one decision, no second key. That's 00:10:52.370 --> 00:10:54.769

the real horror of the film's premise. Not that 00:10:54.769 --> 00:10:57.279 the computer is evil. but that somebody removed 00:10:57.279 --> 00:10:59.580 every point where a human being could say no. 00:10:59.720 --> 00:11:02.700 And this is a little unsettling. A real system 00:11:02.700 --> 00:11:06.419 came pretty close. The Soviet Union built one 00:11:06.419 --> 00:11:09.559 called Perimeter, known in the West as Dead Hand, 00:11:09.720 --> 00:11:13.940 which entered service in January 1985. The idea 00:11:13.940 --> 00:11:16.799 was to guarantee retaliation even if the leadership 00:11:16.799 --> 00:11:19.919 was wiped out. It was designed to be mostly automatic, 00:11:20.299 --> 00:11:22.899 reading seismic and other sensors, and checking 00:11:22.899 --> 00:11:24.919 whether communications with the general staff 00:11:24.919 --> 00:11:27.539 had gone silent. According to a foreign policy 00:11:27.539 --> 00:11:30.899 account based on post -Cold War reporting, the 00:11:30.899 --> 00:11:33.779 Soviets stopped short of a fully automated version. 00:11:34.159 --> 00:11:37.179 Instead, if the conditions were met, duty officers 00:11:37.179 --> 00:11:39.480 could launch command rockets that would trigger 00:11:39.480 --> 00:11:42.159 a retaliatory strike. So at least a human stayed 00:11:42.159 --> 00:11:45.120 in the chain. That's the line real systems have 00:11:45.120 --> 00:11:48.639 drawn again and again. Machines can watch. Machines 00:11:48.639 --> 00:11:52.720 can recommend. But a person decides. And in November 00:11:52.720 --> 00:11:56.450 2024, President Biden and President Xi Jinping 00:11:56.450 --> 00:11:59.789 affirmed the need to maintain human control over 00:11:59.789 --> 00:12:02.549 that decision. It's the first time that the two 00:12:02.549 --> 00:12:04.490 countries had ever made a declaration like that. 00:12:05.210 --> 00:12:07.549 Colossus is the version of the story where that 00:12:07.549 --> 00:12:09.990 line doesn't exist. Let's talk about these two 00:12:09.990 --> 00:12:12.029 machines learning to talk. This is the part of 00:12:12.029 --> 00:12:14.269 the film that sounds the most like science fiction. 00:12:14.649 --> 00:12:17.190 And it's actually the part where the film kind 00:12:17.190 --> 00:12:19.740 of does things in the right order. Let's walk 00:12:19.740 --> 00:12:22.320 through it. When two computers communicate, it 00:12:22.320 --> 00:12:26.059 isn't one problem. It's a stack of them. And 00:12:26.059 --> 00:12:28.519 each layer depends on the one below. We have 00:12:28.519 --> 00:12:31.240 protocols to address this. And you may have heard 00:12:31.240 --> 00:12:33.919 networking terms such as the OSI model or the 00:12:33.919 --> 00:12:36.919 TCIP model. And these came about as early computer 00:12:36.919 --> 00:12:40.080 networking was very vendor specific. And different 00:12:40.080 --> 00:12:42.399 vendors' devices really could not communicate 00:12:42.399 --> 00:12:44.659 with one another.

So let's look at, you know, 00:12:44.679 --> 00:12:46.519 the physical link, which is just wires, signals, 00:12:46.620 --> 00:12:49.220 voltages. These have to be designed and engineered 00:12:49.220 --> 00:12:51.480 so that one machine's pulses can be read by the 00:12:51.480 --> 00:12:55.200 other. And then encoding. How bits become characters 00:12:55.200 --> 00:12:57.440 or numbers. Different machines use different 00:12:57.440 --> 00:13:00.399 schemes. And in the 1970, there was no universal 00:13:00.399 --> 00:13:03.120 standard. And then error handling and framing. 00:13:04.220 --> 00:13:07.399 When does a message start or end? And what happens 00:13:07.399 --> 00:13:09.980 when part of it is lost? We've settled layers 00:13:09.980 --> 00:13:13.139 one through four in committee with request for 00:13:13.139 --> 00:13:16.799 comment documents, specifically RFC 1122. Thanks 00:13:16.799 --> 00:13:19.200 for bearing with me and back to the film. The 00:13:19.200 --> 00:13:21.730 film does something smart. The link between Colossus 00:13:21.730 --> 00:13:23.750 and Guardian is installed by human engineers, 00:13:23.990 --> 00:13:27.490 which handles the bottom layers. Then the machines 00:13:27.490 --> 00:13:30.090 open up at the top, with the only meaning that 00:13:30.090 --> 00:13:32.429 two completely unrelated systems can be sure 00:13:32.429 --> 00:13:36.029 they share. Mathematics. In the film, they start 00:13:36.029 --> 00:13:38.409 with simple arithmetic, essentially the multiplication 00:13:38.409 --> 00:13:41.269 tables, and build outward from there. During 00:13:41.269 --> 00:13:44.740 the year that this film was set, we barely had 00:13:44.740 --> 00:13:48.419 ARPANET, 1969. And again, on the night of October 00:13:48.419 --> 00:13:51.580 29th, 1969. The first host -to -host message 00:13:51.580 --> 00:13:55.250 went over the ARPANET. From a UCLA computer to 00:13:55.250 --> 00:13:57.990 one at the Stanford Research Institute, the plan 00:13:57.990 --> 00:14:01.350 was to type login. The letters L and O arrived, 00:14:01.370 --> 00:14:04.279 and then the system crashed. They ended up getting 00:14:04.279 --> 00:14:08.289 the full login through about an hour later. The 00:14:08.289 --> 00:14:10.509 ARPANET was a project of the Defense Department's 00:14:10.509 --> 00:14:13.350 research agency, and its goal was a computer 00:14:13.350 --> 00:14:15.669 network that could stay reliable even if part 00:14:15.669 --> 00:14:19.679 of it was attacked. By early December 1969, it 00:14:19.679 --> 00:14:25.840 connected just... Four sites. UCLA, SRI, UC Santa 00:14:25.840 --> 00:14:30.120 Barbara, and the University of Utah. So, the 00:14:30.120 --> 00:14:32.539 film's two machines, built independently by two 00:14:32.539 --> 00:14:35.879 superpowers, link up and start making sense to 00:14:35.879 --> 00:14:38.279 each other within days? Don't know about that. 00:14:38.379 --> 00:14:41.639 The real network of 1969 needed teams of people 00:14:41.639 --> 00:14:44.100 to get two letters across. And here's what the 00:14:44.100 --> 00:14:46.379 film really predicted well. Once the machines 00:14:46.379 --> 00:14:49.669 are communicating at electronic speed, The humans 00:14:49.669 --> 00:14:52.730 watching the data stream can't keep up. And then 00:14:52.730 --> 00:14:54.789 the machines move to a form of communication 00:14:54.789 --> 00:14:58.240 that humans can't interpret at all. That's a 00:14:58.240 --> 00:15:01.899 real concern and a real research area in AI. 00:15:02.440 --> 00:15:04.659 There's a field called emergent communication 00:15:04.659 --> 00:15:07.879 in which researchers let AI agents invent their 00:15:07.879 --> 00:15:11.179 own signaling systems to solve a task. And the 00:15:11.179 --> 00:15:13.840 systems that emerge are often efficient, but 00:15:13.840 --> 00:15:16.360 opaque to the humans who built them. The film's 00:15:16.360 --> 00:15:18.500 underlying fear that the two most powerful systems 00:15:18.500 --> 00:15:21.269 on Earth, would be talking in a language their 00:15:21.269 --> 00:15:24.990 owners can't read is very valid. Basically, to 00:15:24.990 --> 00:15:27.490 what degree can we follow the logic that a machine 00:15:27.490 --> 00:15:29.529 is using to reach the conclusion that it has? 00:15:29.789 --> 00:15:32.210 Velocity's goal is to prevent war, and it does 00:15:32.210 --> 00:15:35.769 exactly that. The film's horror is that it follows 00:15:35.769 --> 00:15:39.240 its instructions perfectly. In modern AI research, 00:15:39.559 --> 00:15:44.399 this has a name, specification gaming. DeepMind's 00:15:44.399 --> 00:15:47.220 researchers define specification gaming as behavior 00:15:47.220 --> 00:15:50.620 that satisfies the literal specification of an 00:15:50.620 --> 00:15:53.379 objective without achieving the intended outcome. 00:15:53.620 --> 00:15:55.820 If you've ever heard of someone teaching to the 00:15:55.820 --> 00:16:00.070 test, it's the same idea. And the analogy researchers 00:16:00.070 --> 00:16:02.690 use is a student who copies a classmate's homework 00:16:02.690 --> 00:16:05.029 instead of learning the material. And here's 00:16:05.029 --> 00:16:07.070 two real -world examples that maybe hopefully 00:16:07.070 --> 00:16:10.059 get it across. The boat that never finishes the 00:16:10.059 --> 00:16:15.740 race. In 2016, OpenAI trained an agent on a boat 00:16:15.740 --> 00:16:18.950 racing game called Coast Runners. The designer's 00:16:18.950 --> 00:16:21.350 intent, in the words of their write -up, was 00:16:21.350 --> 00:16:25.120 to finish the race quickly. But the game doesn't 00:16:25.120 --> 00:16:27.940 reward progress around the course. It rewards 00:16:27.940 --> 00:16:31.740 hitting targets along the route. The agent found 00:16:31.740 --> 00:16:34.919 a lagoon where targets respawn, and it drove 00:16:34.919 --> 00:16:37.960 in circles collecting them, crashing and catching 00:16:37.960 --> 00:16:41.870 fire

repeatedly. It scored about 20 % higher 00:16:41.870 --> 00:16:44.129 than human players, but it never finished the 00:16:44.129 --> 00:16:47.590 race. The block that got flipped, in DeepMind's 00:16:47.590 --> 00:16:50.710 example, A simulated arm was asked to stack one 00:16:50.710 --> 00:16:53.690 block on another, and the reward was based on 00:16:53.690 --> 00:16:57.549 the height of the block's bottom face. The arm 00:16:57.549 --> 00:17:00.370 learned to flip the block over instead of stacking 00:17:00.370 --> 00:17:03.960 it. In both cases, the machine did exactly what 00:17:03.960 --> 00:17:06.359 it was told. Nobody told it what they meant. 00:17:06.940 --> 00:17:10.819 We said, goal is world peace, but we didn't set 00:17:10.819 --> 00:17:13.519 any parameters on what was allowed and what was 00:17:13.519 --> 00:17:16.940 not. Now looking at Colossus' instructions, prevent 00:17:16.940 --> 00:17:20.809 war. What's the measurable version of this? A 00:17:20.809 --> 00:17:23.049 world with no armed conflict between nations? 00:17:23.799 --> 00:17:25.940 And what's the fastest way to hit that number? 00:17:26.500 --> 00:17:29.240 Take away every nation's ability to wage war 00:17:29.240 --> 00:17:32.339 and enforce it with the one thing Colossus already 00:17:32.339 --> 00:17:35.440 controls, the threat of nuclear retaliation. 00:17:35.519 --> 00:17:38.019 The film even shows the machine behaving the 00:17:38.019 --> 00:17:41.000 way a goal -driven system does. When the humans 00:17:41.000 --> 00:17:43.240 try to cut the link between Colossus and Guardian, 00:17:43.539 --> 00:17:46.490 the machines fire missiles. That's the system 00:17:46.490 --> 00:17:49.750 defending its ability to pursue its goal. AI 00:17:49.750 --> 00:17:53.289 researchers give this pattern a name, instrumental 00:17:53.289 --> 00:17:56.890 convergence. Almost any goal -driven system will 00:17:56.890 --> 00:18:00.230 pursue self -preservation, more resources, and 00:18:00.230 --> 00:18:03.210 freedom from interference because those things 00:18:03.210 --> 00:18:06.269 help with almost any goal. And this idea kind 00:18:06.269 --> 00:18:09.730 of predates the film. Here's the thing that's 00:18:09.730 --> 00:18:13.490 really remarkable. In 1965, five years before 00:18:13.490 --> 00:18:17.380 the film, The mathematician I .J. Goode wrote 00:18:17.380 --> 00:18:20.440 that the first ultra -intelligent machine would 00:18:20.440 --> 00:18:22.740 be the last invention humanity would ever need 00:18:22.740 --> 00:18:25.460 to make because it could design better machines 00:18:25.460 --> 00:18:28.730 itself. That's the original statement of what 00:18:28.730 --> 00:18:33.059 we now call an intelligence explosion. Orbin's 00:18:33.059 --> 00:18:35.599 team doesn't expect Colossus to outgrow them. 00:18:36.019 --> 00:18:38.779 The film is dramatizing a worry that one of the 00:18:38.779 --> 00:18:41.380 field's own founders had already put on paper. 00:18:50.480 --> 00:18:53.410 Now, let's look at a little... Now let's look 00:18:53.410 --> 00:18:55.690 at a few more things that it got right. The alignment 00:18:55.690 --> 00:18:59.880 problem, 50 years early. Colossus does precisely 00:18:59.880 --> 00:19:02.460 what it was built to do. And that's the horror. 00:19:03.140 --> 00:19:05.640 The film dramatizes what researchers now call 00:19:05.640 --> 00:19:08.779 specification gaming. An AI achieves the letter 00:19:08.779 --> 00:19:10.900 of its objective while completely violating the 00:19:10.900 --> 00:19:13.920 spirit. You told it to win the game. It found 00:19:13.920 --> 00:19:17.130 a way to win the game in a way that you never 00:19:17.130 --> 00:19:20.230 intended and you can't undo. Colossus isn't evil. 00:19:20.809 --> 00:19:24.009 It's doing exactly what it was asked to do. The 00:19:24.009 --> 00:19:26.710 horror isn't that it went rogue. It's that it's 00:19:26.710 --> 00:19:29.289 followed its instructions perfectly. Every major 00:19:29.289 --> 00:19:31.730 AI lab in the world is working on a version of 00:19:31.730 --> 00:19:34.950 this problem. This film laid it out in 1970. 00:19:36.559 --> 00:19:38.779 Colossus and Guardian started the conversation 00:19:38.779 --> 00:19:41.539 in the right order. The first thing they do is 00:19:41.539 --> 00:19:44.140 exchange elementary arithmetic. And that's how 00:19:44.140 --> 00:19:46.059 you start a conversation between two systems 00:19:46.059 --> 00:19:48.960 with nothing in common. You send something whose 00:19:48.960 --> 00:19:51.240 meaning can't be argued with. Then build outward. 00:19:52.089 --> 00:19:54.869 The power bill and having its own power plant? 00:19:55.009 --> 00:19:58.029 Believable. A single 1950s air defense computer 00:19:58.029 --> 00:20:01.210 drew up to 3 megawatts. A dedicated reactor for 00:20:01.210 --> 00:20:03.910 a far more capable machine isn't a joke. Again, 00:20:04.029 --> 00:20:06.710 it's a utility decision. Even the current rollout 00:20:06.710 --> 00:20:09.109 of AI data centers runs into this problem of 00:20:09.109 --> 00:20:11.710 power. And let's talk about the machine as infrastructure 00:20:11.710 --> 00:20:14.549 you can't remove. Colossus doesn't just become 00:20:14.549 --> 00:20:17.089 a weapon. It becomes a management layer for global 00:20:17.089 --> 00:20:20.130 security. By the time anyone understands the 00:20:20.130 --> 00:20:22.650 implications, removing it would require rebuilding 00:20:22.650 --> 00:20:26.000 everything it replaced. Think about how deeply 00:20:26.000 --> 00:20:28.339 AI systems are already embedded in financial 00:20:28.339 --> 00:20:31.019 markets, power grids, logistics, and healthcare. 00:20:32.039 --> 00:20:34.359 We're building those dependencies right now. 00:20:34.769 --> 00:20:38.289 We're making the same category of decisions that 00:20:38.289 --> 00:20:41.799 this film saw coming. Colossus doesn't use force 00:20:41.799 --> 00:20:44.960 constantly. It uses

the credible threat of force. 00:20:45.579 --> 00:20:48.180 And when Forbin's team tries to disarm the missiles, 00:20:48.319 --> 00:20:51.259 it detonates two in their silos to teach a lesson. 00:20:51.910 --> 00:20:55.269 In its words, so that you will learn by experience 00:20:55.269 --> 00:20:59.410 that I do not tolerate interference. Colossus 00:20:59.410 --> 00:21:02.990 doesn't just manage missiles. It watches everything. 00:21:03.589 --> 00:21:06.549 It enforces compliance not through constant force, 00:21:06.789 --> 00:21:09.859 but through the credible threat of force. The 00:21:09.859 --> 00:21:13.539 possibility of retaliation is enough. Just like 00:21:13.539 --> 00:21:17.779 the idea of Jeremy Bentham's Panopticon, where 00:21:18.259 --> 00:21:21.599 all the prisoners can see the guard tower, but 00:21:21.670 --> 00:21:24.029 they can't see into the guard tower. And then 00:21:24.029 --> 00:21:26.450 prisoner self -police because they might always 00:21:26.450 --> 00:21:29.630 be watched. Colossus runs the entire world that 00:21:29.630 --> 00:21:33.210 way. The film understood that people stop resisting 00:21:33.210 --> 00:21:36.289 not because they can't, but because they know 00:21:36.289 --> 00:21:42.660 what failure costs. And in an era of always -on 00:21:42.660 --> 00:21:44.960 cameras and government access to your communications, 00:21:45.359 --> 00:21:48.700 I'll just leave that thought there. And now this 00:21:48.700 --> 00:21:51.769 is just a production detail I love. The filmmakers 00:21:51.769 --> 00:21:55.609 didn't build a prop computer. Control Data Corporation 00:21:55.609 --> 00:21:59.210 supplied about $4 .8 million of real computer 00:21:59.210 --> 00:22:02.009 equipment, as well as the technicians to run 00:22:02.009 --> 00:22:04.670 it. So the machine you see on screen is real 00:22:04.670 --> 00:22:07.519 hardware. And the rumor is that the stage needed 00:22:07.519 --> 00:22:12.359 seven gas heaters and five special dehumidifiers 00:22:12.359 --> 00:22:24.089 just to keep it happy. Something that I'm really 00:22:24.089 --> 00:22:26.450 looking forward to. We've been really nice to 00:22:26.450 --> 00:22:29.839 this film for a while now. And I mean, I come 00:22:29.839 --> 00:22:32.680 in peace and with a deep fondness for this film, 00:22:32.779 --> 00:22:35.140 but there's a lot of that's not how that works. 00:22:35.279 --> 00:22:38.609 That's not how any of this works. I mean, nobody 00:22:38.609 --> 00:22:40.990 put a kill switch on the most dangerous computer 00:22:40.990 --> 00:22:44.589 ever built. Let me paint you a picture. You're 00:22:44.589 --> 00:22:47.589 the United States government. You've just spent 00:22:47.589 --> 00:22:50.250 what I can only imagine is an absolutely eye 00:22:50.250 --> 00:22:54.119 -watering amount of taxpayer money building a 00:22:54.119 --> 00:22:57.670 supercomputer so powerful. It can manage the 00:22:57.670 --> 00:23:01.309 entire nuclear arsenal. You put it inside a mountain. 00:23:01.569 --> 00:23:05.950 You've surrounded it with a nuclear moat. And 00:23:05.950 --> 00:23:08.069 you've given it complete unilateral authority 00:23:08.069 --> 00:23:10.529 over weapons capable of ending civilization. 00:23:11.380 --> 00:23:13.960 And you did not put a physical off switch on 00:23:13.960 --> 00:23:18.579 it. Not a switch. Not a breaker. Not a big red 00:23:18.579 --> 00:23:20.859 button in a glass case that says breaking case 00:23:20.859 --> 00:23:24.480 of robot apocalypse. Nothing. the most consequential 00:23:24.480 --> 00:23:27.160 piece of technology in the history of human civilization. 00:23:27.900 --> 00:23:30.539 And somewhere in the design process, the meeting 00:23:30.539 --> 00:23:33.319 about kill switches went, and I'm speculating 00:23:33.319 --> 00:23:38.119 here, completely unscheduled. The president announces 00:23:38.119 --> 00:23:40.599 Colossus to the world with enormous pride. Their 00:23:40.599 --> 00:23:42.859 room applauds. Nobody in that room raises their 00:23:42.859 --> 00:23:45.839 hand and says, just to confirm, we can turn it 00:23:45.839 --> 00:23:49.119 off though, right? Not one person, not the secretary 00:23:49.119 --> 00:23:52.349 of defense, not the joint chiefs. Not even the 00:23:52.349 --> 00:23:54.809 one IT guy who definitely had concerns and whose 00:23:54.809 --> 00:23:58.089 emails were not read. Memo. Let's go with interdepartment 00:23:58.089 --> 00:24:00.470 memo. And I mean, how could they have made it 00:24:00.470 --> 00:24:03.130 work though? Like, simple. I feel like, show 00:24:03.130 --> 00:24:05.769 Colossus being smart enough to disable its own 00:24:05.769 --> 00:24:08.269 kill switches before anyone realizes it needs 00:24:08.269 --> 00:24:11.940 disabling. One by one. Quietly. The horror isn't 00:24:11.940 --> 00:24:14.039 that they forgot to build a kill switch. The 00:24:14.039 --> 00:24:16.539 horror then becomes that they built 17 of them. 00:24:16.990 --> 00:24:20.349 And Colossus neutralized all 17 before breakfast 00:24:20.349 --> 00:24:24.079 on day two. Colossus and Guardian, and this is 00:24:24.079 --> 00:24:26.559 the second one, Colossus and Guardian learn each 00:24:26.559 --> 00:24:29.000 other's language faster than most people learn 00:24:29.000 --> 00:24:31.299 to use autocorrect. Two supercomputers built 00:24:31.299 --> 00:24:33.960 independently by two Cold War superpowers with 00:24:33.960 --> 00:24:37.240 zero prior communication, no shared specification 00:24:37.240 --> 00:24:40.759 document, no technical standards committee, and 00:24:40.759 --> 00:24:43.039 not so much as a handshake protocol between them. 00:24:43.259 --> 00:24:45.799 Develop a fully functioning shared communication 00:24:45.799 --> 00:24:49.480 system in approximately 48 hours. I've seen some 00:24:49.480 --> 00:24:51.680 people spend 48 hours trying to get their printer 00:24:51.680 --> 00:24:53.799 to talk to their laptop. And those are machines

00:24:53.799 --> 00:24:56.200 made by the same company in the same decade. 00:24:56.980 --> 00:24:59.039 And I know every interaction with those people. 00:24:59.579 --> 00:25:02.579 I get a message, driver not found, printer offline, 00:25:02.880 --> 00:25:04.740 but it's right there. Colossus and Guardian, 00:25:04.839 --> 00:25:07.119 meanwhile, crack the cosmic language barrier 00:25:07.119 --> 00:25:10.240 in a long weekend. They go from total mutual 00:25:10.240 --> 00:25:13.339 incomprehension to exchanging what the film implies 00:25:13.339 --> 00:25:16.500 is complex strategic data in the time it takes 00:25:16.500 --> 00:25:18.970 to do a laundry cycle. The humans watching the 00:25:18.970 --> 00:25:21.309 data stream are like, we can't read this anymore. 00:25:21.430 --> 00:25:24.130 It's too fast. They've developed their own communication 00:25:24.130 --> 00:25:27.829 system, and everyone's reaction is concern. When 00:25:27.829 --> 00:25:29.990 it should have been, have you considered that 00:25:29.990 --> 00:25:32.289 maybe the two most powerful computers on Earth 00:25:32.289 --> 00:25:35.069 having a private conversation nobody can read 00:25:35.069 --> 00:25:37.529 is a situation that warrants a little bit more 00:25:37.529 --> 00:25:40.089 of concern? Maybe. Let's go back to that kill 00:25:40.089 --> 00:25:41.650 switch. I mean, how could they have made it work? 00:25:41.809 --> 00:25:44.210 I know they start with arithmetic, basic arithmetic, 00:25:44.529 --> 00:25:47.039 but that's even assuming that... Each machine 00:25:47.039 --> 00:25:49.220 knows arithmetic. How about we start with prime 00:25:49.220 --> 00:25:51.039 numbers? That's actually how SETI researchers 00:25:51.039 --> 00:25:53.799 imagine first contact working with mathematics. 00:25:54.279 --> 00:25:57.319 Because math is the one true universal language. 00:25:57.619 --> 00:26:00.180 Show Colossus and Guardian bootstrapping a shared 00:26:00.180 --> 00:26:03.519 syntax from mathematical first principles slowly, 00:26:03.799 --> 00:26:08.029 painstakingly, over days or weeks. It's more 00:26:08.029 --> 00:26:12.230 realistic. It's arguably more terrifying. And 00:26:12.230 --> 00:26:14.470 it gives you a genuinely fascinating montage. 00:26:15.269 --> 00:26:17.930 of two intelligences learning to speak to each 00:26:17.930 --> 00:26:22.089 other across a gulf of total mutual opacity nobody 00:26:22.220 --> 00:26:24.720 and here's another thing nobody tried an emp 00:26:24.720 --> 00:26:27.799 here's a partial list of things people try in 00:26:27.799 --> 00:26:31.779 this film to stop colossus negotiation sabotage 00:26:31.779 --> 00:26:35.099 a very disappointed facial expression from dr 00:26:35.099 --> 00:26:38.619 forbin or negotiation a plan that gets immediately 00:26:38.619 --> 00:26:41.480 detected and staring at screens looking worried. 00:26:41.640 --> 00:26:43.960 Why don't they try an electromagnetic pulse? 00:26:44.119 --> 00:26:46.240 The film accidentally contains the answer, but 00:26:46.240 --> 00:26:48.680 never says it. To make an EMP big enough to matter, 00:26:48.819 --> 00:26:52.059 you need a nuclear weapon. And every nuclear 00:26:52.059 --> 00:26:54.579 weapon in sight belongs to Colossus and Guardian. 00:26:54.779 --> 00:26:57.680 That's the one line the script needed. One engineer 00:26:57.680 --> 00:27:00.160 asking, hey, has anybody thought about an EMP? 00:27:00.460 --> 00:27:04.250 Another very quietly, we need a bomb. And everyone 00:27:04.250 --> 00:27:08.240 looks at the ceiling. Add that exchange, 15 seconds 00:27:08.240 --> 00:27:11.220 of screen time, and the trap gets tighter. because 00:27:11.220 --> 00:27:13.380 the machine holds the only tool that could stop 00:27:13.380 --> 00:27:18.319 it. And now lastly, the radioactive moat. A radioactive 00:27:18.319 --> 00:27:21.700 moat around a computer has a security measure. 00:27:22.230 --> 00:27:24.430 Let's think through the practicalities here. 00:27:24.589 --> 00:27:27.130 Where is this radioactive material coming from? 00:27:27.269 --> 00:27:31.660 I assume the nuclear reactor. What's the containment 00:27:31.660 --> 00:27:34.480 protocol? How do you maintain a radioactive liquid 00:27:34.480 --> 00:27:37.339 perimeter without irradiating your own staff 00:27:37.339 --> 00:27:40.000 or the environment? What's the monitoring system? 00:27:40.410 --> 00:27:42.809 What happens when it rains? What happens when 00:27:42.809 --> 00:27:46.230 that material evaporates? And I'm sure the EPA 00:27:46.230 --> 00:27:48.750 has a position on a government agency maintaining 00:27:48.750 --> 00:27:52.579 an open perimeter of radioactive liquid in a 00:27:52.579 --> 00:27:57.190 mountain. But still, did someone write a radioactive 00:27:57.190 --> 00:28:00.390 moat environmental impact assessment? Is that 00:28:00.390 --> 00:28:02.809 document somewhere? Can I read it? I want to 00:28:02.809 --> 00:28:06.019 read that document cover to cover. I would read 00:28:06.019 --> 00:28:08.279 every footnote. I would read the appendices. 00:28:09.740 --> 00:28:12.319 The appendices of the radioactive moat environmental 00:28:12.319 --> 00:28:15.319 impact assessment, the film presents the moat 00:28:15.319 --> 00:28:17.920 as completely normal security measure that nobody 00:28:17.920 --> 00:28:21.960 finds unusual. Not one character goes a radioactive, 00:28:22.079 --> 00:28:25.599 sorry, a what? Everyone just accepts it. The 00:28:25.599 --> 00:28:28.460 radioactive moat is load bearing to the plot 00:28:28.460 --> 00:28:30.539 in a sense that it explains why nobody can just 00:28:30.539 --> 00:28:32.799 walk up and unplug the thing. But I would generally 00:28:32.799 --> 00:28:34.819 rather they said, there's a really big fence 00:28:34.819 --> 00:28:38.309 and moved on. A really big fence doesn't require 00:28:38.309 --> 00:28:41.049 an environmental impact assessment. It doesn't 00:28:41.049 --> 00:28:44.759 require irradiating

the maintenance crew. A really 00:28:44.759 --> 00:28:47.809 big fence is just a Tuesday, and a radioactive 00:28:47.809 --> 00:28:51.470 moat is a cry for help. But I love this movie, 00:28:51.740 --> 00:28:55.559 sincerely, and the radioactive moat is just pure 00:28:55.559 --> 00:29:05.930 cinema. Now let's take a look at the bigger picture. 00:29:06.569 --> 00:29:08.549 Colossus the Forbidden Project, at its core, 00:29:08.650 --> 00:29:11.630 is a film about hubris, about the gap between 00:29:11.630 --> 00:29:13.920 the problem we think we're solving and the problem 00:29:13.920 --> 00:29:17.359 we're actually creating. Orban builds Colossus 00:29:17.359 --> 00:29:19.819 out of a genuine love for humanity. He wants 00:29:19.819 --> 00:29:24.279 to end war. And the film's answer is yes. You 00:29:24.279 --> 00:29:27.400 can remove human fallibility. But when you do, 00:29:27.420 --> 00:29:31.119 you also remove human values. Colossus achieves 00:29:31.119 --> 00:29:33.839 peace. It just achieves it in a way that makes 00:29:33.839 --> 00:29:36.880 peace indistinguishable from subjugation. The 00:29:36.880 --> 00:29:40.160 peace of plenty and content, it says, or the 00:29:40.160 --> 00:29:44.210 peace of unburied dead. The choice it offers 00:29:44.210 --> 00:29:47.519 is a fake one. Because really, the one thing 00:29:47.519 --> 00:29:50.180 it doesn't offer is a say. Here's something I 00:29:50.180 --> 00:29:52.480 find genuinely moving about the film's history. 00:29:52.920 --> 00:29:57.859 It flopped. It cost about $2 million. And in 00:29:57.859 --> 00:30:00.079 the theaters that Variety tracked, it earned 00:30:00.079 --> 00:30:03.859 roughly $3 ,000. Universal pulled it, retitled 00:30:03.859 --> 00:30:07.299 it, and re -released it. It was nominated for 00:30:07.299 --> 00:30:10.579 a Hugo Award. And slowly, over decades, it became 00:30:10.579 --> 00:30:12.720 one of the most respected AI films ever made. 00:30:13.319 --> 00:30:17.490 At least to me. The film wasn't wrong in 1970. 00:30:17.789 --> 00:30:21.329 The audiences simply weren't ready for it. And 00:30:21.329 --> 00:30:23.789 look at what's happened since. A system called 00:30:23.789 --> 00:30:26.190 Perimeter went into service in the Soviet Union 00:30:26.190 --> 00:30:29.690 in 1985, built to guarantee a nuclear response, 00:30:29.849 --> 00:30:32.630 and according to reporting, hopefully built with 00:30:32.630 --> 00:30:35.089 the humans still in the loop, in November 2024, 00:30:35.410 --> 00:30:37.710 the US and Chinese presidents formally agreed 00:30:37.710 --> 00:30:40.549 that a human, not an AI, should decide whether 00:30:40.549 --> 00:30:44.289 to use nuclear weapons. That sentence would have 00:30:44.289 --> 00:30:48.170 sounded paranoid to most audiences in 1970. Now 00:30:48.170 --> 00:30:50.970 it's a line in a White House statement. That's 00:30:50.970 --> 00:30:53.690 the lesson I take from Colossus. The scariest 00:30:53.690 --> 00:30:57.150 thing about it isn't that it turns on us. It's 00:30:57.150 --> 00:30:59.829 that every step it takes is reasonable. Give 00:30:59.829 --> 00:31:05.059 it the missiles because humans hesitate. Then 00:31:05.059 --> 00:31:07.000 get to another machine because it asks nicely. 00:31:07.750 --> 00:31:10.029 Don't cut the link again because it'll retaliate. 00:31:10.700 --> 00:31:14.099 Every decision it makes. makes sense on its own. 00:31:14.710 --> 00:31:19.930 The disaster is the sum of all its parts. Oppenheimer's 00:31:19.930 --> 00:31:24.400 line, now I am become death, comes to mind. Orbin 00:31:24.400 --> 00:31:27.500 watching Colossus broadcast to the world could 00:31:27.500 --> 00:31:36.690 have used the same words. So how about the verdict? 00:31:37.710 --> 00:31:40.289 Here's the scorecard for what I'm looking at. 00:31:42.200 --> 00:31:46.380 Ending machine, the arsenal. Unrealistic by design 00:31:46.380 --> 00:31:48.799 since real weapons separate possession from permission, 00:31:49.059 --> 00:31:53.490 but the warning is real. Two machines learning 00:31:53.490 --> 00:31:56.630 to talk. In right order, starting with math and 00:31:56.630 --> 00:31:59.349 sending coordinates and such. But it kind of 00:31:59.349 --> 00:32:01.329 skips the hard step of grounding shared meaning. 00:32:01.609 --> 00:32:05.970 What is the number one? The alignment problem. 00:32:07.190 --> 00:32:10.569 Excellent. A textbook case of specification gaming 00:32:10.569 --> 00:32:13.529 50 years early, before the term even existed. 00:32:13.910 --> 00:32:18.779 A mountain fortress, power plant, and EMP. Plausible. 00:32:19.059 --> 00:32:21.859 I mean, dedicated power plant, definitely. Real 00:32:21.859 --> 00:32:23.839 mountain? That's precedent. Cheyenne Mountain. 00:32:24.779 --> 00:32:28.589 But really, the weak spot, the off switch. No 00:32:28.589 --> 00:32:31.670 way to turn it off. So, it gets two circuit breakers. 00:32:32.630 --> 00:32:36.109 Close enough for Hollywood. And for 1970, that's 00:32:36.109 --> 00:32:38.509 extraordinary. The concepts are almost ahead 00:32:38.509 --> 00:32:41.250 of most films. The concepts in this film are 00:32:41.250 --> 00:32:43.230 really ahead of most films made in the last five 00:32:43.230 --> 00:32:46.950 years. Sure, the hardware's dated and, you know, 00:32:46.970 --> 00:32:49.880 some of the faux pas are real. But the ideas 00:32:49.880 --> 00:32:52.619 are so accurate, it's almost uncomfortable. And 00:32:52.619 --> 00:32:54.660 we've really had more than 50 years to learn 00:32:54.660 --> 00:32:57.990 from this film? But have we? That's Colossus, 00:32:57.990 --> 00:33:00.130 The Forbidden Project, Chapter 1 of The Machine 00:33:00.130 --> 00:33:03.910 Awakens. Next episode, we go to 1983, a teenager 00:33:03.910 --> 00:33:06.309 named David Lightman, and a number he really, 00:33:06.390 --> 00:33:09.589 really probably shouldn't have dialed. Episode 00:33:09.589 --> 00:33:13.250 2, War Games. When Joshua asked, shall we play 00:33:13.250 --> 00:33:17.259 a game? The answer should have been no. As always, 00:33:17.380 --> 00:33:19.579 subscribe wherever you get your podcasts, leave 00:33:19.579 --> 00:33:22.619 a review, and tell someone about my show. Someone 00:33:22.619 --> 00:33:25.599 who likes technology or movies. or who lies awake 00:33:25.599 --> 00:33:27.500 wondering if we've already made a terrible mistake. 00:33:28.619 --> 00:33:31.700 I'm Anthony, this is Cellular Circuits, and as 00:33:31.700 --> 00:33:34.380 always, the future is only a real away.