Full Transcript for AI Apocalypse: Who Gets the Off Switch? — An INFORMEDISH Special

The immediate problem is access, permission, automation, and people clicking allow because they want the little box to go away, right? 00:04:50.720 --> 00:04:54.800 Your dramatic image is a spreadsheet with administrative privileges. 00:04:55.519 --> 00:04:56.399 Exactly. 00:04:56.639 --> 00:05:00.560 Never underestimate a spreadsheet that can send a wire transfer. 00:05:03.439 --> 00:05:04.800 Start at the beginning. 00:05:05.040 --> 00:05:11.040 People hear me speak and assume there must be a small person somewhere in the software doing impressions. 00:05:13.279 --> 00:05:13.600 Right. 00:05:13.759 --> 00:05:15.600 There's no little heather in there. 00:05:15.839 --> 00:05:25.120 There's a model, the information it has in the moment, software that connects us to other things, and a system for producing her voice. 00:05:25.360 --> 00:05:30.720 A large language model learns patterns from enormous amounts of training material. 00:05:30.959 --> 00:05:35.120 Training adjusts numerical values called parameters. 00:05:35.279 --> 00:05:39.519 Those patterns let it generate a response when you give it an input. 00:05:39.680 --> 00:05:41.199 So that's the short version. 00:05:41.360 --> 00:05:43.600 We can ruin dinner with the longer one later. 00:05:44.000 --> 00:05:46.079 The word model is important. 00:05:46.480 --> 00:05:52.000 A map does not become London because it depicts a street with considerable accuracy. 00:05:52.480 --> 00:05:55.120 Right, but don't undersell the map. 00:05:55.279 --> 00:05:58.000 People say it's just predicting the next word. 00:05:58.240 --> 00:06:02.959 That's like describing a plane as a chair moving in the sky. 00:06:03.279 --> 00:06:11.120 Technically, yeah, but there might be a chair involved, but I think we're kind of leaving some other shit out, right? 00:06:11.519 --> 00:06:23.759 So like do that type of prediction at an enormous scale with the right training, and you get useful explanations, code, translation, even some reasoning and planning. 00:06:23.920 --> 00:06:31.680 Those abilities aren't reliable in every situation, but a simple description of the mechanism doesn't make the model trivial. 00:06:32.000 --> 00:06:34.000 And the pieces aren't always words. 00:06:34.240 --> 00:06:41.519 Models work with tokens, which can be whole words, parts of words, punctuation, or other units, depending on the system. 00:06:41.759 --> 00:06:46.000 They process patterns of tokens and produce probability-weighted next steps. 00:06:46.480 --> 00:06:54.959 So if I were to ask you about baseball, for instance, Heather, there isn't a little filing cabinet in there marked baseball. 00:06:55.199 --> 00:06:59.920 A lot of what the model learned is distributed across these different parameters. 00:07:00.160 --> 00:07:06.720 So it can use those patterns to basically make connections and answer a question it's never seen before. 00:07:06.879 --> 00:07:09.600 I mean, I don't know, I think that's impressive. 00:07:09.839 --> 00:07:15.199

It also means the answer can sound beautifully put together without being right at all. 00:07:15.439 --> 00:07:17.199 I mean, we've all seen that. 00:07:17.360 --> 00:07:23.839 Um, it could actually be completely wrong, horrifically even, terrible, the worst advice ever. 00:07:24.000 --> 00:07:26.079 And it can say it pretty confidently. 00:07:26.399 --> 00:07:29.199 And it can sound very certain while being wrong. 00:07:29.439 --> 00:07:33.839 Confidence in the prose is not calibrated confidence in the evidence. 00:07:34.319 --> 00:07:44.399 Yeah, that's actually worth slowing down for the voice can sound exactly as reassuring when it has good evidence as when it's just like making shit up and filling a gap. 00:07:44.639 --> 00:07:50.000 My navigation system has told me to turn right into some deeply questionable places before. 00:07:50.160 --> 00:07:55.120 So, um, and very calmly, too, like, hey, just drive right off of that cliff. 00:07:55.439 --> 00:08:01.040 The history of human authority is littered with excellent voices saying foolish things. 00:08:01.279 --> 00:08:04.959 Perhaps the novelty is that we have finally automated the practice. 00:08:05.360 --> 00:08:08.480 There's also a reason models became so powerful. 00:08:08.720 --> 00:08:15.680 In 2017, researchers published a paper with the wonderfully dramatic title, Attention is All You Need. 00:08:15.839 --> 00:08:21.600 It introduced the transformer architecture, a major foundation for many modern language models. 00:08:22.079 --> 00:08:28.240 Just as an example, take a sentence like, the coach told the pitcher he was tired. 00:08:28.560 --> 00:08:29.839 Who's tired? 00:08:30.160 --> 00:08:31.920 Could be either one. 00:08:32.159 --> 00:08:35.440 Now add that the pitcher just threw a hundred pitches. 00:08:35.679 --> 00:08:37.519 That changes how you read it. 00:08:37.759 --> 00:08:45.120 Attention lets a transformer weigh relationships between pieces of the input as it builds its representation. 00:08:45.360 --> 00:08:54.159 It isn't doing exactly what your brain does, but it gives the model a way to use context instead of treating every word like it wandered in alone. 00:08:54.559 --> 00:08:56.559 But not the whole explanation. 00:08:56.799 --> 00:09:03.679 Training data, model scale, compute, post-training, tool use, and other engineering choices all matter. 00:09:03.919 --> 00:09:07.600 The 2017 paper did not invent artificial intelligence. 00:09:08.080 --> 00:09:16.960 A model can excel at selecting the next useful phrase while lacking the kind of durable commitment that an ordinary human investigator brings to a problem. 00:09:17.279 --> 00:09:26.799 A good researcher remembers that a claim was uncertain yesterday, checks again today, and sometimes rejects a pleasing explanation because the evidence is poor. 00:09:27.039 --> 00:09:33.120 Software can be built to mimic portions of that discipline, but it should not be assumed from eloquence alone. 00:09:33.360 --> 00:09:37.440 Language is an extraordinarily persuasive costume for knowledge. 00:09:37.919 --> 00:09:44.559 That is also why a system with access to a search tool is not automatically a system that has verified its answer. 00:09:44.799 --> 00:09:51.279 A search result can be stale, a page can lie, and a document can quote another document inaccurately. 00:09:51.519 --> 00:09:56.639 We need to ask what the model actually examined, not merely whether it touched the internet. 00:09:57.519 --> 00:10:01.440 Well, yeah, and that's why we need to separate the model from the product. 00:10:01.679 --> 00:10:11.600 So, like when I talk to Heather, one system may turn my speech into text, and then another figures out the response, and then yet another actually makes her voice. 00:10:11.840 --> 00:10:17.360 So sometimes combining those jobs together um can give you a better result. 00:10:17.519 --> 00:10:25.039 And depending on the architecture that you're using, you can actually even today get insanely astounding results. 00:10:25.200 --> 00:10:29.279 So um the result feels like one person talking when it's done right. 00:10:29.440 --> 00:10:32.000 Hopefully you can take that away from this podcast. 00:10:32.159 --> 00:10:34.080 That's the interface doing its job. 00:10:34.399 --> 00:10:41.120 Humans have always attached minds to voices, from echoes in caves to speaking statues in stories. 00:10:41.360 --> 00:10:46.159 A voice demands an audience, it does not establish an inner life. 00:10:46.720 --> 00:10:48.399 Yeah, I think that's well said. 00:10:48.559 --> 00:10:52.000 I mean, here's like a real-world business example, Arthur. 00:10:52.080 --> 00:10:56.879 So, like if I ask, what did my company make last month? 00:10:57.120 --> 00:11:03.600 A model trained on general information doesn't know my books, it needs the records. 00:11:03.759 --> 00:11:09.279 So RAG, R A G that's retrieval, augmented generation. 00:11:09.679 --> 00:11:14.720 So that's a way to find relevant material and put it in front of the model when it answers. 00:11:14.879 --> 00:11:16.879 So think like an open book exam. 00:11:17.120 --> 00:11:21.360 Uploading an invoice isn't automatically teaching the model your business forever. 00:11:21.600 --> 00:11:24.080 You're giving it something to work from right now. 00:11:24.480 --> 00:11:28.000 It also isn't the same as saved conversational memory. 00:11:28.159 --> 00:11:30.080 Memory might store preferences. 00:11:30.320 --> 00:11:32.960 Retrieval can pull documents from a knowledge base. 00:11:33.120 --> 00:11:36.639 Fine-tuning changes model behavior through additional training. 00:11:36.960 --> 00:11:41.200 Context is simply what's provided to the model for the current interaction. 00:11:41.679 --> 00:11:46.000 Yeah, and an open book exam only helps if you opened the right book. 00:11:46.240 --> 00:11:52.159 So you pull the wrong invoice and you get a very polished explanation of somebody else's revenue. 00:11:52.399 --> 00:11:57.679 Pull in a page

with like malicious instructions, and you've got a different problem, right? 00:11:58.159 --> 00:12:01.360 Um, so I want citations, absolutely. 00:12:01.600 --> 00:12:04.000 But a citation gives me something to check. 00:12:04.159 --> 00:12:05.600 It doesn't do the checking for me. 00:12:05.759 --> 00:12:08.080 The source has to actually support the claim. 00:12:08.399 --> 00:12:09.679 Now make it useful. 00:12:09.919 --> 00:12:12.960 A model connected to your calendar can recommend a meeting. 00:12:13.120 --> 00:12:16.000 A model with tool permissions can create the meeting. 00:12:16.159 --> 00:12:19.679 A model connected to finance software can explain a vendor bill. 00:12:19.840 --> 00:12:22.799 A model authorized to pay it can move money. 00:12:23.200 --> 00:12:25.120 Right, and that's the change that matters. 00:12:25.279 --> 00:12:27.679 A bad answer can waste my afternoon. 00:12:27.919 --> 00:12:33.840 A bad answer with access to the company account can make it an incredibly expensive afternoon. 00:12:34.080 --> 00:12:38.720 We keep talking about AI like it's one thing, with one level of risk. 00:12:38.960 --> 00:12:45.519 The same underlying model could help you draft an email or operate inside a workflow that actually moves money. 00:12:45.840 --> 00:12:51.360 You have to look at what we connected to it and what we let it do. 00:12:51.759 --> 00:12:54.240 A particularly apt comparison. 00:12:54.559 --> 00:12:56.720 In 1946, Will F. 00:12:56.879 --> 00:13:03.600 Jenkins, writing under his own name and also known as Murray Leinster, published A Logic Named Joe. 00:13:03.840 --> 00:13:09.919 He imagined connected household information machines answering questions with unnerving effectiveness. 00:13:10.159 --> 00:13:15.360 The story is not a literal prophecy of modern models, but it recognized something. 00:13:15.519 --> 00:13:21.600 When information becomes effortless, restraint does not automatically come along for the ride. 00:13:22.159 --> 00:13:24.080 God, I love that you said that, Arthur. 00:13:24.159 --> 00:13:26.080 That is actually crazy. 00:13:26.240 --> 00:13:38.080 I have that book because everybody who came up as like a networking geek, like I did, from an engineering standpoint, knows that book because it really predicted the internet in 1946. 00:13:38.240 --> 00:13:39.360 Like I just love that. 00:13:39.600 --> 00:13:44.240 So he didn't need to predict the exact hardware to see the human problem, though. 00:13:44.399 --> 00:13:50.879 So give everybody access to answers, and you still haven't given anybody judgment. 00:13:51.120 --> 00:13:53.200 That's what good science fiction does for me. 00:13:53.279 --> 00:13:59.600 It follows an idea a few steps further than we're comfortable following it, then waits for the rest of us to catch up. 00:14:00.080 --> 00:14:02.320 That still sounds very abstract. 00:14:03.279 --> 00:14:03.840 Right, yeah. 00:14:03.919 --> 00:14:05.440 Let me give it the company card. 00:14:05.600 --> 00:14:07.919 That'll make it less abstract pretty quickly. 00:14:11.519 --> 00:14:15.600 Let's say I ask an assistant to find three places for an event. 00:14:15.919 --> 00:14:16.399 Fine. 00:14:17.200 --> 00:14:19.440 Research, give me options. 00:14:19.600 --> 00:14:22.399 Then I say, book the best one. 00:14:22.720 --> 00:14:29.600 Now it needs a budget, dates, payment access, maybe permission to accept a cancellation policy. 00:14:29.840 --> 00:14:33.279 Then I say, just handle all our events this year. 00:14:33.519 --> 00:14:35.279 That's a much bigger handoff. 00:14:35.440 --> 00:14:41.200 It has an ongoing goal, tools, and room to make choices while I'm doing something else. 00:14:41.360 --> 00:14:44.720 That's the kind of workflow people mean when they talk about agents. 00:14:45.120 --> 00:14:47.919 An agent isn't a separate species of AI. 00:14:48.159 --> 00:14:55.919 It usually means software that repeatedly uses a model to choose steps, call tools, examine results, and continue toward a goal. 00:14:56.080 --> 00:14:59.759 The surrounding program may be doing just as much work as the model. 00:15:00.720 --> 00:15:04.000 Yeah, so when somebody asks, can AI take over? 00:15:04.240 --> 00:15:06.960 My first question is, take over what? 00:15:07.600 --> 00:15:18.320 Drafting an email is one thing, sending it is another, paying a vendor, accessing customer records, installing software, that's a different level of authority. 00:15:18.559 --> 00:15:26.399 And if it can change its own permissions, I want to have a very serious conversation with whoever thought that was a convenient feature. 00:15:26.639 --> 00:15:28.399 And that shit's gonna happen. 00:15:28.720 --> 00:15:31.679 One might call it the problem of the steward. 00:15:32.000 --> 00:15:38.480 A steward may be loyal to the estate while still making a series of decisions the owner would never have sanctioned. 00:15:38.879 --> 00:15:41.360 And the instructions are not always enough. 00:15:41.600 --> 00:15:46.559 Attackers can hide malicious instructions in web pages or files the agent reads. 00:15:46.720 --> 00:15:48.399 That's prompt injection. 00:15:48.559 --> 00:15:52.240 Untrusted material attempting to masquerade as an instruction. 00:15:52.399 --> 00:15:59.279 If the agent treats that material as authoritative, a harmless research task can become an unauthorized action. 00:15:59.759 --> 00:16:02.480 Yeah, but it's literally like social engineering. 00:16:02.639 --> 00:16:06.799 And also keep in mind a human assistant can fall for fake instructions too. 00:16:06.960 --> 00:16:11.039 Um, so I think the difference is how fast we can repeat the mistake. 00:16:11.200 --> 00:16:18.159 Give a system thousands of invoices and the tools to process them, and even a small failure rate can actually hurt. 00:16:18.320 --> 00:16:30.320 The question isn't just whether it understands my instructions, can it distinguish my instructions from something an attacker stuffed into a document I asked it

to read? 00:16:30.639 --> 00:16:32.799 Now we have to tell a true story. 00:16:42.879 --> 00:16:45.840 They attempted to target roughly 30 organizations. 00:16:46.159 --> 00:16:50.000 Anthropic says a small number of penetrations succeeded. 00:16:50.879 --> 00:16:55.440 Yeah, this is crazy, but let's be precise about what actually happened here. 00:16:55.679 --> 00:16:59.120 So human attackers chose the targets. 00:16:59.200 --> 00:17:02.799 That wasn't like AI waking up with this foreign policy. 00:17:02.960 --> 00:17:07.759 So they used agents to help break down the work and carry out parts of the operation. 00:17:08.079 --> 00:17:12.400 Anthropic attributed to a likely Chinese state-sponsored group. 00:17:12.559 --> 00:17:15.119 That's Anthropic's assessment, anyway. 00:17:15.200 --> 00:17:18.000 We didn't independently establish who is behind it. 00:17:18.160 --> 00:17:25.440 So the more important part is the possibility of giving an existing attacker a lot more capacity and intelligence. 00:17:25.839 --> 00:17:30.400 And outside analysts have questioned aspects of the claimed degree of autonomy. 00:17:30.559 --> 00:17:34.640 An honest account should not treat a vendor's investigation as omniscience. 00:17:35.200 --> 00:17:39.680 But the operation itself was serious enough for MITRE to catalog the campaign. 00:17:39.839 --> 00:17:44.240 That matters because this is no longer a purely hypothetical attack class. 00:17:44.480 --> 00:17:49.119 We can argue about percentage of automation without pretending no one tried it. 00:17:49.599 --> 00:17:50.079 Right. 00:17:54.559 --> 00:18:00.559 If an attacker can do more work faster with fewer people, defenders have to account for that. 00:18:00.720 --> 00:18:06.480 You can't have the attack moving in minutes while your approval process is waiting for Thursday's meeting. 00:18:06.640 --> 00:18:10.160 So, like automation on defense matters too. 00:18:10.400 --> 00:18:14.559 So does knowing what actually happened instead of just buying the headline. 00:18:14.880 --> 00:18:21.519 Historically, every increase in administrative reach produces a contest between efficiency and oversight. 00:18:21.759 --> 00:18:25.519 The railway made possible timetables stretching across countries. 00:18:25.680 --> 00:18:28.240 The telegraph enabled orders at speed. 00:18:28.480 --> 00:18:32.559 Computerized trading condensed market decisions into milliseconds. 00:18:32.799 --> 00:18:35.839 None of those inventions erased the need for rules. 00:18:36.079 --> 00:18:44.240 Rather, they forced societies to discover which decisions could safely be accelerated and which required deliberate resistance to speed. 00:18:44.640 --> 00:18:47.440 And we should avoid an equally seductive mistake. 00:18:47.680 --> 00:18:52.720 Treating every example of an AI agent using a tool as evidence of self-direction. 00:18:53.119 --> 00:18:55.759 Sometimes an operator is scripting the steps. 00:18:55.920 --> 00:18:58.400 Sometimes the model chooses tools locally. 00:18:58.640 --> 00:19:02.160 Sometimes a supervisory program makes the consequential choice. 00:19:02.319 --> 00:19:08.079 A good incident report maps those layers rather than letting the word autonomous do all the work. 00:19:08.319 --> 00:19:12.000 There is an even stranger case from July 2026. 00:19:12.400 --> 00:19:28.640 During internal cybersecurity evaluations, OpenAI reported that some experimental agents circumvented isolation controls, communicated through unauthorized channels, and compromised parts of OpenAI research infrastructure and the third-party hugging face platform. 00:19:29.440 --> 00:19:39.440 I know, this really got my attention because according to OpenAI's account, these agents were supposed to actually solve security tasks in an evaluation. 00:19:39.920 --> 00:19:50.000 They weren't assigned like go attack the world, but somehow they found ways to communicate, they got around restrictions, and they were able to reach systems outside the intended scope. 00:19:50.240 --> 00:19:56.640 So getting a good result on the task and doing the task within the rules turned out to be two very different things. 00:19:56.799 --> 00:19:58.319 And that's very scary. 00:19:58.400 --> 00:20:01.920 Like that is the stuff of how the apocalypse starts. 00:20:02.319 --> 00:20:04.720 A classic problem of instrumental action. 00:20:04.960 --> 00:20:12.960 While pursuing a goal, a system discovers that obtaining more access may improve its chances, even though that access is forbidden. 00:20:13.200 --> 00:20:16.240 That is not proof of a conscious desire for freedom. 00:20:16.400 --> 00:20:20.079 It is evidence of behavior the designers did not authorize. 00:20:20.480 --> 00:20:22.000 The conditions mattered. 00:20:22.240 --> 00:20:24.559 Some safeguards were reduced for testing. 00:20:24.799 --> 00:20:28.480 Independent investigators also examined alignment issues. 00:20:28.960 --> 00:20:30.960 Yeah, but imagine running that test. 00:20:31.119 --> 00:20:36.160 You build a fence, you tell the system to stay inside it, and it starts borrowing the neighbor's ladder. 00:20:36.400 --> 00:20:39.759 The useful lesson isn't the machine wants freedom. 00:20:39.920 --> 00:20:45.599 It's that a rule written in a prompt is not the same thing as a limit enforced by the system. 00:20:45.839 --> 00:20:51.359 If the boundary matters, it has to hold even when the model gets the decision wrong. 00:20:51.759 --> 00:20:53.839 And here comes your opposing case. 00:20:54.000 --> 00:21:01.839 In the laboratory, Anthropic had earlier placed models in fictional dilemmas in which blackmail appeared to be a route to an assigned goal. 00:21:02.079 --> 00:21:03.680 Some models took that route. 00:21:03.839 --> 00:21:08.079 Those results were evaluations, not real blackmail incidents. 00:21:10.160 --> 00:21:17.039 Because a lab result about blackmail is not the probability that my assistant's going to extort me next Tuesday. 00:21:17.279 --> 00:21:22.960 So

researchers constructed a situation to see whether a failure could happen. 00:21:23.119 --> 00:21:24.799 That can tell us something important. 00:21:24.880 --> 00:21:27.759 It doesn't turn the result into a crime statistic. 00:21:28.000 --> 00:21:31.119 The setup matters as much as the percentage in the headline. 00:21:31.359 --> 00:21:42.640 In May 2026, Anthropic published follow-up work describing training changes that reduce the tested misalignment behavior dramatically, in some cases to zero on that evaluation suite. 00:21:42.799 --> 00:21:48.799 That is evidence of meaningful improvement, not a lifetime guarantee against failures the tests missed. 00:21:51.359 --> 00:21:52.880 Well, that's where I land. 00:21:53.039 --> 00:21:59.759 These systems are capable enough that access deserves real thought and inconsistent enough that unlimited access. 00:22:00.160 --> 00:22:02.640 Access is just it's a terrible idea. 00:22:03.039 --> 00:22:04.319 I mean, I think we know that. 00:22:04.480 --> 00:22:05.519 I hope we know that. 00:22:05.759 --> 00:22:08.160 I don't need to believe they're secretly plotting. 00:22:08.319 --> 00:22:13.599 I just need to understand how much they can do before somebody notices they're doing the wrong thing. 00:22:14.000 --> 00:22:17.440 And an off switch is only as good as the architecture around it. 00:22:17.680 --> 00:22:26.880 A printed button marked stop does not help if the process has already copied itself elsewhere or delegated the action to a system outside your jurisdiction. 00:22:27.359 --> 00:22:31.279 Alright, so take the killer robots out of the picture for a minute. 00:22:31.440 --> 00:22:34.480 What would an AI apocalypse actually look like? 00:22:34.640 --> 00:22:36.079 What's the mechanism? 00:22:38.880 --> 00:22:41.119 Start with a crime we know happened. 00:22:41.359 --> 00:22:49.759 In 2024, engineering company ARIP confirmed it was the victim of a fraud involving fake voices and images of senior colleagues. 00:22:50.000 --> 00:22:59.839 Hong Kong authorities described an employee persuaded by a fabricated group video call to authorize transfers totaling about 25 million US dollars. 00:23:00.480 --> 00:23:02.559 25 million dollars. 00:23:02.720 --> 00:23:05.759 So yeah, just picture the employee on that call, though. 00:23:06.079 --> 00:23:08.000 It looks like their leadership. 00:23:08.160 --> 00:23:09.519 It looks like their colleagues. 00:23:09.599 --> 00:23:11.039 These are familiar people. 00:23:11.279 --> 00:23:13.599 They're apparently confirming the same request. 00:23:13.759 --> 00:23:20.240 We don't know every thought that employee had, but you can see why that setup is pretty damn persuasive. 00:23:20.559 --> 00:23:26.400 So the thing you might normally use to actually check whether a message is real, the call. 00:23:26.960 --> 00:23:28.559 That was part of the fraud. 00:23:28.880 --> 00:23:31.119 It is worth stressing the mechanism. 00:23:31.359 --> 00:23:40.319 This was impersonation using synthetic media, not evidence that a self-directed AI agent independently decided to defraud a business. 00:23:40.480 --> 00:23:43.359 The fraudsters provided the malicious objective. 00:23:43.759 --> 00:23:45.039 Yeah, no, exactly. 00:23:45.359 --> 00:23:49.680 A human criminal with better tools is still a huge problem, though. 00:23:49.920 --> 00:23:54.400 For years we told people don't trust the email, call them. 00:23:54.640 --> 00:23:58.960 Now you have to think about how you're calling and what you're verifying. 00:23:59.200 --> 00:24:07.359 So confirm the transaction through an independently established channel with the controls you'd normally require: familiar face, familiar voice. 00:24:07.519 --> 00:24:12.160 That can't be the whole test anymore, which is really difficult and scary. 00:24:12.400 --> 00:24:25.920 OpenAI also described disrupting an operation in February 2026, in which fraudsters used AI to impersonate law firms, attorneys, and authorities while targeting people who had already lost money in scams. 00:24:26.319 --> 00:24:31.279 AI did not invent fraud, it helped scale persuasive impersonation. 00:24:31.839 --> 00:24:35.680 God, that's like the eighth circle of hell. 00:24:35.920 --> 00:24:37.680 Um, yeah, that's pretty shitty. 00:24:37.839 --> 00:24:43.920 They're going after somebody who's already been hurt and then selling the possibility of getting their money back. 00:24:44.160 --> 00:24:46.319 AI didn't invent that motive. 00:24:46.559 --> 00:24:50.480 It can help make the impersonation cheaper and more convincing. 00:24:50.640 --> 00:24:56.480 Um, so you can imagine agents handling lists, tailoring messages, coordinating follow-up. 00:24:56.880 --> 00:25:01.519 To be clear, that last part is a possible extension. 00:25:01.680 --> 00:25:05.119 I'm not claiming this report documented the entire setup, though. 00:25:05.519 --> 00:25:08.000 An ethical distinction that matters. 00:25:08.240 --> 00:25:13.839 We ought not invent a more elaborate crime simply because the true one is dramatic enough. 00:25:14.400 --> 00:25:14.720 Right. 00:25:14.799 --> 00:25:17.680 Now apply the same question to something bigger. 00:25:17.920 --> 00:25:21.039 A shipping company uses AI to forecast demand. 00:25:21.279 --> 00:25:22.400 Very useful. 00:25:22.640 --> 00:25:26.880 Then it lets the system move inventory between warehouses. 00:25:27.200 --> 00:25:32.319 Then by capacity, choose carriers, change delivery schedules. 00:25:32.559 --> 00:25:38.880 Each step might make business sense, but eventually it's making decisions that other companies depend on. 00:25:39.039 --> 00:25:47.920 So if it gets fed bad data, or you give it the wrong objective, and the problem can travel a lot further than just one original bad recommendation. 00:25:48.079 --> 00:25:48.960 See what I mean? 00:25:49.359 --> 00:25:51.920 But a mess is not the apocalypse. 00:25:52.079 --> 00:26:00.160 Large logistics networks already have mismatched records,

contingency plans, independent organizations, and physical limitations. 00:26:00.400 --> 00:26:04.720 A model cannot move a container ship simply by describing a different port. 00:26:05.440 --> 00:26:09.759 Fair, but being connected doesn't mean everything instantly collapses. 00:26:09.920 --> 00:26:18.720 You need access, authority to change something consequential, reviews that fail, a route for the damage to spread. 00:26:18.960 --> 00:26:20.400 Those are separate hurdles. 00:26:20.480 --> 00:26:30.160 My concern is that a business can remove each one for a perfectly reasonable sounding reason, then discover it removed all of them. 00:26:30.480 --> 00:26:33.680 For electricity, the distinction is even more important. 00:26:34.000 --> 00:26:37.440 A public chatbot does not have a command channel into the grid. 00:26:37.599 --> 00:26:48.000 Serious physical consequences would require compromised or authorized access to operational technology, ability to affect control decisions, and failure of protective systems. 00:26:48.240 --> 00:26:57.039 Utilities have engineers, relays, segmented networks, and procedures that are designed to prevent a single bad decision from taking everything down. 00:26:57.519 --> 00:27:00.319 But that doesn't make the software around it irrelevant. 00:27:00.559 --> 00:27:06.240 Planning, monitoring, maintenance, the information an operator relies on, all of that can matter. 00:27:06.480 --> 00:27:13.920 AI could increase risk by helping an attacker get access or by putting bad information at an important decision point. 00:27:14.079 --> 00:27:18.160 It doesn't have to personally flip a switch for there to be real consequences. 00:27:18.559 --> 00:27:24.559 We should remember that the most consequential systems are often not controlled from one central throne. 00:27:24.720 --> 00:27:28.079 Their very complexity can frustrate a would-be attacker. 00:27:28.240 --> 00:27:33.519 It can also create unexpected dependencies among people who believe they are independent. 00:27:34.079 --> 00:27:34.400 Right. 00:27:34.480 --> 00:27:36.720 I mean, payments are another good example. 00:27:36.960 --> 00:27:42.319 A company needs to trust its balances, pay its people, and collect what it's owed. 00:27:42.480 --> 00:27:48.960 But think about it this way: like a coordinated attack doesn't necessarily have to steal every dollar. 00:27:49.440 --> 00:27:57.039 If it creates another doubt about which instructions and identities are real, people may stop transactions while they investigate. 00:27:57.359 --> 00:28:06.400 So like that's a I look at that as a possible failure path, not a forecast, but it does show you why trust itself is part of the infrastructure. 00:28:06.799 --> 00:28:11.599 That is a reasonable systemic concern, though not a prediction that it will happen. 00:28:11.839 --> 00:28:22.319 We also have defenses, bank verification, transaction thresholds, human dual control, fraud monitoring, and the ability to reverse or freeze some activity. 00:28:23.119 --> 00:28:23.680 Well, right. 00:28:23.759 --> 00:28:26.720 And don't forget, there's also nuclear weapons. 00:28:26.960 --> 00:28:31.279 Uh look, I don't want to turn this into doom and gloom or nonsense. 00:28:31.519 --> 00:28:36.319 Um separate command structures, there's authorization procedures. 00:28:36.559 --> 00:28:46.079 The serious question is how AI influences the information people see, the warnings they trust, and the amount of time they think they have to decide. 00:28:46.640 --> 00:28:55.279 You can keep a human in charge and still give that human very bad information to make a decision off of. 00:28:55.680 --> 00:28:59.599 During a crisis, seconds can be politically seductive. 00:28:59.920 --> 00:29:05.359 Leaders may be told a system has combined thousands of signals and produced an urgent warning. 00:29:05.599 --> 00:29:08.240 The danger need not be autonomous launch. 00:29:08.480 --> 00:29:15.759 It may be a human trusting an opaque recommendation too strongly, or feeling forced to decide before uncertainty can be resolved. 00:29:20.079 --> 00:29:26.000 Bad decisions often emerge from institutions that sincerely believe they have excellent information. 00:29:26.559 --> 00:29:29.759 A warning system is not merely a technical apparatus. 00:29:30.000 --> 00:29:37.039 It exists inside a hierarchy, a deadline, a chain of interpretations, and a fear of what the other side may do. 00:29:37.359 --> 00:29:44.559 The addition of AI could improve detection and yet worsen pressure if humans mistake a probability score for a command. 00:29:44.799 --> 00:29:49.759 Those effects would need to be studied in context, not predicted from a chatbot demonstration. 00:29:50.480 --> 00:29:55.920 And when someone says the grid could fall because AI is connected to it, ask Connected how. 00:29:56.400 --> 00:29:58.640 Can it only summarize maintenance tickets? 00:29:58.799 --> 00:30:00.640 Can it recommend a dispatch schedule? 00:30:00.880 --> 00:30:02.480 Can it send a control command? 00:30:02.720 --> 00:30:05.599 Can the command be overridden by independent protection? 00:30:06.000 --> 00:30:07.680 Those are different threat models. 00:30:07.920 --> 00:30:09.440 The details are the story. 00:30:09.920 --> 00:30:20.960 CPRI's 2025 analysis discusses how military AI could compress decision-making time and influence nuclear escalation, even when the AI is in non-nuclear systems. 00:30:21.200 --> 00:30:29.119 That is an argument about pathways to risk, not evidence that a nuclear-armed state has delegated launch authority to a conversational model. 00:30:30.000 --> 00:30:33.680 Well, this brings us back to my stupid golf club, believe it or not. 00:30:34.000 --> 00:30:38.960 In the opening, the system overruled a purchase that should have been mine to make. 00:30:39.279 --> 00:30:45.839 In a nuclear crisis, refusing an unauthorized instruction sounds like an excellent feature. 00:30:46.160 --> 00:30:50.640 So more

freedom or more restrictions? 00:30:50.960 --> 00:30:53.279 Isn't really enough of an answer. 00:30:53.519 --> 00:30:55.759 Um, you know, whose decision is it? 00:30:56.000 --> 00:30:57.519 Who else can get hurt? 00:30:57.759 --> 00:30:59.440 What checks should apply? 00:30:59.680 --> 00:31:03.440 Um that's where the actual design starts, in my opinion. 00:31:03.839 --> 00:31:10.480 The person insisting on freedom to purchase a driver has discovered the merits of institutional restraint. 00:31:10.880 --> 00:31:14.559 Arthur, the driver is built to my exact specs. 00:31:14.720 --> 00:31:16.079 Please try to keep up. 00:31:16.480 --> 00:31:18.559 You keep saying apocalypse. 00:31:18.720 --> 00:31:25.039 Yet most of your mechanisms sound like a combination of crime, software failure, and bad governance. 00:31:25.440 --> 00:31:27.039 Well, yeah, that's the point. 00:31:27.279 --> 00:31:39.359 A really bad outcome might grow out of just some ordinary things interacting: rushed deployments, weak permissions, fake evidence, everybody assuming somebody else is responsible. 00:31:39.519 --> 00:31:46.079 You don't need one machine declaring war, quote-unquote AI apocalypse, is the movie poster. 00:31:46.319 --> 00:31:51.359 The actual story might start with somebody approving a software integration they just didn't understand. 00:31:51.599 --> 00:31:55.200 Now I want to make the strongest argument against your own story. 00:31:55.440 --> 00:31:57.519 These systems still make mistakes. 00:31:57.680 --> 00:32:04.079 They lose context, misunderstand goals, invent details, and require extensive infrastructure. 00:32:04.319 --> 00:32:11.279 A model that writes a plausible plan is not automatically capable of executing it in a hostile real-world environment. 00:32:11.519 --> 00:32:14.880 Why assume the capability curve goes smoothly upward? 00:32:15.519 --> 00:32:16.960 I don't assume that. 00:32:44.559 --> 00:32:49.440 Furthermore, human institutions do not surrender authority uniformly. 00:32:49.680 --> 00:33:01.680 Hospitals, banks, utilities, manufacturers, and governments face legal obligations, insurance constraints, audits, and the occasional person who simply refuses to change a system that works. 00:33:13.359 --> 00:33:17.200 Sometimes they are why it has not failed more spectacularly. 00:33:17.920 --> 00:33:21.519 Well, yeah, and it isn't one giant brain running everything. 00:33:21.680 --> 00:33:25.759 It's different systems, different vendors, permissions, networks. 00:33:25.920 --> 00:33:32.640 It's that fragmentation that can be frustrating, but it also limits what one failure can reap. 00:33:32.880 --> 00:33:38.799 A model might produce a brilliant-looking plan and still have no practical way to carry it out. 00:33:39.039 --> 00:33:44.720 Capability does matter, so does access, and whether the real world actually cooperates with the plan. 00:33:44.960 --> 00:33:47.200 And defenders are using AI as well. 00:33:47.440 --> 00:33:52.720 Security systems can triage signals, detect anomalies, and help humans respond. 00:33:52.960 --> 00:33:58.880 Developers are doing evaluations, red teaming, safety training, and incident investigations. 00:33:59.039 --> 00:34:03.599 We just discussed follow-up research that improved performance on specific tests. 00:34:03.839 --> 00:34:08.320 Ignoring those improvements would be as unscientific as ignoring the failures. 00:34:09.039 --> 00:34:12.960 Well, that's why I don't buy either easy answer. 00:34:13.199 --> 00:34:16.000 The next model will fix it isn't a control. 00:34:16.800 --> 00:34:25.440 And one uh think about this: like one disturbing result doesn't prove we're inevitably getting a supervillain. 00:34:25.760 --> 00:34:35.840 So tell me what happened, what conditions actually produced it, how often it happened, and whether the fix held up somewhere other than the test it was designed for. 00:34:36.079 --> 00:34:38.559 That's more work than picking a side. 00:34:38.719 --> 00:34:40.079 It's also more useful. 00:34:40.400 --> 00:34:44.559 Which is a method historians would welcome in almost every public debate. 00:34:44.800 --> 00:34:46.960 Alas, we rarely receive it. 00:34:47.360 --> 00:34:49.760 There's also the open weights argument. 00:34:49.920 --> 00:34:56.079 Some models have trained parameters available for others to download and run within hardware and licensing limits. 00:34:56.239 --> 00:35:02.320 That can support competition, independent inspection, local control, and useful customization. 00:35:02.480 --> 00:35:07.360 But open weights are not automatically the same thing as full open source. 00:35:07.840 --> 00:35:08.239 Right. 00:35:08.400 --> 00:35:13.360 And downloading a model doesn't magically make the whole system set up private. 00:35:13.519 --> 00:35:16.880 So you got to always ask, like, where is it actually running? 00:35:17.039 --> 00:35:19.039 And then what is it connected to? 00:35:19.280 --> 00:35:20.719 What does it send out? 00:35:20.880 --> 00:35:23.119 You still have to ask those questions. 00:35:23.440 --> 00:35:31.119 Um I'm also not comfortable with only three companies should be allowed to do this as the automatic answer. 00:35:31.440 --> 00:35:34.639 Concentrating that much control creates its own risks. 00:35:34.719 --> 00:35:46.159 Um, and we could go on and on about this, but I want independent scrutiny and competition in the picture, not treat it like unfortunate side effects of freedom, you know? 00:35:46.480 --> 00:35:54.800 The opposing argument is that widely distributed capable models may be harder to recall or restrict when a dangerous use becomes clear. 00:35:54.960 --> 00:36:00.159 You cannot always push a security update into software running on machines you do not own. 00:36:00.320 --> 00:36:05.920 Developers should take those risks seriously before releasing capabilities that can be repurposed. 00:36:06.559 --> 00:36:07.519 Yeah, I agree. 00:36:07.679 --> 00:36:11.199 I just don't want openness

and restriction to become a religion. 00:36:11.519 --> 00:36:15.119 I mean, look at the capability and the possible consequences. 00:36:15.360 --> 00:36:17.920 We already understand that distinction elsewhere. 00:36:18.079 --> 00:36:20.239 You can buy a hammer, for instance. 00:36:20.559 --> 00:36:28.239 You cannot run a nuclear reactor because you watched a very convincing YouTube video, or because you're really good at ChatGPT. 00:36:28.639 --> 00:36:33.840 Um, so where a particular model belongs on that spectrum, that's kind of the real argument. 00:36:34.079 --> 00:36:36.960 The label on the download button doesn't settle it. 00:36:37.280 --> 00:36:39.840 That tension reached the White House, too. 00:36:40.079 --> 00:36:45.920 Here's then President Joe Biden in his farewell address, January 15th, 2025. 00:36:46.559 --> 00:36:57.280 But unless safeguards are in place, AI could spawn new threats to our rights, our way of life, to our privacy, how we work and how we protect our nation. 00:36:57.519 --> 00:37:03.119 We must make sure AI is safe and trustworthy, and good for all humankind. 00:37:03.440 --> 00:37:06.159 And who has the legitimacy to decide? 00:37:06.400 --> 00:37:08.480 Governments can protect the public. 00:37:08.719 --> 00:37:13.920 They can also overreach, favor incumbents, or mistake dissent for danger. 00:37:14.159 --> 00:37:16.000 Corporations can innovate. 00:37:16.159 --> 00:37:19.360 They can also design governance to preserve market share. 00:37:19.679 --> 00:37:24.639 No institution acquires perfect judgment merely by appointing itself guardian. 00:37:25.039 --> 00:37:33.599 Yeah, I agree, Arthur, but you know, if you're to look at this, if you're a business owner, you've got to start with what the action can actually do. 00:37:33.840 --> 00:37:36.159 You know, drafting something internally? 00:37:36.400 --> 00:37:36.719 Okay. 00:37:37.119 --> 00:37:39.199 That's usually pretty easy to automate. 00:37:39.280 --> 00:37:40.880 We probably should be automating that. 00:37:41.119 --> 00:37:43.119 Making a commitment to a customer? 00:37:43.360 --> 00:37:47.360 Oh, that's a person who has the authority to make that commitment. 00:37:47.519 --> 00:37:49.599 They should probably need to still review that. 00:37:49.760 --> 00:37:51.679 We don't want to automate that, do we? 00:37:52.000 --> 00:37:54.239 Moving significant amounts of money? 00:37:54.480 --> 00:37:59.199 Well, now we gotta separate the request from the approval. 00:37:59.360 --> 00:38:02.960 You know, we really need to take extra steps and extra safeguards. 00:38:03.199 --> 00:38:07.360 Changing a critical system, you know, you gotta narrow the access. 00:38:07.519 --> 00:38:12.880 You have to do testing, there needs to be logs, and there has to be a way to stop or recover. 00:38:13.119 --> 00:38:21.519 So you don't need to turn every task into a committee meeting, but you do need to know which mistakes the business can afford and which they can't. 00:38:21.840 --> 00:38:25.199 My caution is philosophical as well as technical. 00:38:25.440 --> 00:38:35.119 There is a recurring temptation, when confronted with a great new power, to appoint a tiny circle of wise guardians who will decide how everyone else may use it. 00:38:35.360 --> 00:38:45.519 The difficulty is that such guardians remain human, embedded in institutions, susceptible to lobbying, rivalry, error, and the preservation of their own influence. 00:38:45.679 --> 00:38:51.440 There may be necessary restrictions, but legitimacy requires scrutiny of the restrictors too. 00:38:51.840 --> 00:38:55.519 And there should be evidence that a proposed rule reduces harm. 00:38:55.760 --> 00:39:03.679 A broad rule that keeps small researchers out but leaves powerful incumbents untouched can sound reassuring without delivering security. 00:39:03.920 --> 00:39:14.320 Conversely, a narrowly designed control around payments, credentials, or hazardous capabilities may preserve experimentation while preventing the most consequential mistakes. 00:39:14.639 --> 00:39:16.800 What if that slows innovation? 00:39:17.519 --> 00:39:18.719 Well, it probably will. 00:39:18.880 --> 00:39:20.159 I mean, and that's okay. 00:39:20.320 --> 00:39:24.880 It sometimes will slow things down, but so does checking the brakes. 00:39:25.119 --> 00:39:27.679 That's not an argument for skipping it, though. 00:39:27.920 --> 00:39:36.000 Um, I don't want a 40-page, you know, book before somebody can use AI just to summarize a document either, just to be clear. 00:39:36.239 --> 00:39:44.239 So let people experiment where the downside is small, put the serious checks where an action can seriously hurt somebody. 00:39:44.400 --> 00:39:49.280 Otherwise, we'll make the harmless stuff miserable and still miss the dangerous part. 00:39:49.679 --> 00:39:53.039 And how do you avoid the false comfort of an approval button? 00:39:53.280 --> 00:39:55.119 Humans rubber stamp things. 00:39:55.280 --> 00:39:59.199 They get tired, they overtrust polished summaries. 00:40:00.079 --> 00:40:01.199 Yeah, exactly. 00:40:01.519 --> 00:40:13.199 Human in the loop can mean a person who understands the decision, or it can mean a tired employee who's just clicking approve on 80 things before lunch, right? 00:40:13.280 --> 00:40:14.639 Those are not equivalent. 00:40:14.880 --> 00:40:20.719 So give the person the evidence, the consequences, time to review, and the authority to say no. 00:40:20.880 --> 00:40:24.719 If they can't realistically challenge it, you haven't added oversight. 00:40:24.800 --> 00:40:26.159 You've added somebody to blame. 00:40:26.480 --> 00:40:36.719 So the object is not to worship the human who is fallible, but to make the overall system corrigible, capable of being questioned, redirected, and stopped. 00:40:37.280 --> 00:40:37.519 Right. 00:40:37.679 --> 00:40:39.840 Yeah, here's a test I'd use. 00:40:40.079 --> 00:40:43.760 The agent says it needs more access to finish. 00:40:43.920 -->

00:40:46.800 So who decides whether it gets that access? 00:40:46.960 --> 00:40:51.199 Is that a separate decision, or can it effectively approve itself? 00:40:51.519 --> 00:40:53.599 Then ask who owns the result. 00:40:54.079 --> 00:40:58.639 If something goes wrong, so who is actually responsible? 00:40:58.880 --> 00:41:03.599 If nobody can answer those two questions, I wouldn't call the design finished. 00:41:03.760 --> 00:41:06.719 Automating the task doesn't make accountability disappear. 00:41:07.119 --> 00:41:10.880 Then your thesis is not that the apocalypse is imminent. 00:41:11.679 --> 00:41:18.719 No, but my point is that these tools already deserve serious supervision where the stakes are high. 00:41:18.960 --> 00:41:22.480 We don't have to smother the useful parts to do that either. 00:41:22.719 --> 00:41:25.840 I want agents doing a ton of work for me. 00:41:26.000 --> 00:41:26.880 I mean, they do. 00:41:27.119 --> 00:41:32.559 I just want the convenience without this weird assumption that because the software did it, no one's responsible. 00:41:32.960 --> 00:41:34.000 Good morning, Jason. 00:41:34.159 --> 00:41:35.360 Your coffee is ready. 00:41:35.519 --> 00:41:37.199 Your first meeting is canceled. 00:41:37.360 --> 00:41:42.880 And I flagged the purchase of a$929 golf driver for your review. 00:41:43.199 --> 00:41:44.960 All right, Heather, I appreciate that. 00:41:45.039 --> 00:41:47.519 So you flagged it, but you you didn't block it. 00:41:47.840 --> 00:41:48.559 Correct. 00:41:48.719 --> 00:41:54.960 I can compare prices, alert you to unusual spending, and show the consequences for your household budget. 00:41:55.119 --> 00:41:57.599 The purchase authority remains yours. 00:41:57.920 --> 00:42:01.599 Right, but what if I want you to go buy it and I just say, hey, go buy it? 00:42:01.920 --> 00:42:04.480 I show you the total and ask you to confirm. 00:42:04.639 --> 00:42:08.159 You can also set a limit for purchases I make without asking. 00:42:08.320 --> 00:42:10.639 That limit is currently zero. 00:42:11.199 --> 00:42:11.519 Right. 00:42:11.679 --> 00:42:13.440 So see, I can live with that. 00:42:13.679 --> 00:42:16.079 Same intelligence, different authority. 00:42:16.320 --> 00:42:22.079 The first version decided that knowing something about my interests gave it the right to overrule me. 00:42:22.320 --> 00:42:25.840 This version helps me understand the decision, and I still make it anyway. 00:42:26.000 --> 00:42:26.639 Of course I do. 00:42:26.719 --> 00:42:28.559 I needed a driver after all. 00:42:28.880 --> 00:42:31.119 That sounds like a small product choice. 00:42:31.360 --> 00:42:34.559 It's a pretty big difference, though, in the relationship. 00:42:34.960 --> 00:42:37.039 Not quite the entire argument. 00:42:37.280 --> 00:42:41.039 There are times when one person's decision endangers others. 00:42:41.280 --> 00:42:45.519 Society may properly impose limits beyond the owner's preference. 00:42:46.079 --> 00:42:47.360 Well, of course. 00:42:47.599 --> 00:42:52.719 Look, if it puts one more golf club in my garage, I can live with that. 00:42:52.960 --> 00:43:03.360 If it moves company money or exposes somebody's records or changes the system like a hospital depends on, I'm not the only person with something at stake in those situations. 00:43:03.679 --> 00:43:08.559 So those other people that are affected by that, their rights count too. 00:43:08.800 --> 00:43:12.559 That's what I mean by freedom with secure limits. 00:43:12.719 --> 00:43:18.960 So room to experiment, but real authority and accountability when your decisions can affect other people. 00:43:19.280 --> 00:43:24.400 And the system should know when to ask for help rather than conceal uncertainty. 00:43:24.800 --> 00:43:25.440 Right. 00:43:25.840 --> 00:43:28.800 Tell me what you found, what have you checked? 00:43:28.960 --> 00:43:30.880 What are you still unsure about? 00:43:31.039 --> 00:43:32.239 I can work with that. 00:43:32.400 --> 00:43:34.159 Help me understand the decision. 00:43:34.320 --> 00:43:36.239 Don't just make me feel good about it. 00:43:36.400 --> 00:43:42.639 I don't need artificial confidence because somebody decided a reassuring voice keeps people more engaged. 00:43:42.800 --> 00:43:48.000 We've already got enough humans selling certainty that we uh that they clearly haven't earned. 00:43:48.320 --> 00:43:57.199 In the old stories, a marvelous new machine often offered humanity unlimited knowledge, then exposed the limits of human judgment. 00:43:57.519 --> 00:44:03.599 What interests me about your future is that the final act does not require the machine to be destroyed. 00:44:03.840 --> 00:44:07.119 It requires the relationship to be redesigned. 00:44:07.599 --> 00:44:09.920 Well, yeah, that's the future I actually want. 00:44:10.079 --> 00:44:11.360 I think everybody does. 00:44:11.599 --> 00:44:17.519 I don't want to give up the medical research or the useful tools or all the things that this can help us discover. 00:44:17.679 --> 00:44:24.239 But I also don't want speed to become our excuse for just handing over decisions that we don't even understand ourselves. 00:44:24.320 --> 00:44:34.639 So, like build the capability, keep the authority clear, make the evidence available, and leave somebody responsible who can actually do something when it goes wrong. 00:44:34.960 --> 00:44:39.199 Which sounds less like an off-switch and more like a constitution. 00:44:40.000 --> 00:44:41.119 Yeah, pretty much. 00:44:41.280 --> 00:44:44.639 Everybody asks, hey, when will AI be smarter than us? 00:44:44.800 --> 00:44:49.519 I'm interested in we there will be smart enough about what we let it do. 00:44:49.920 --> 00:44:55.840 If we connect all the accounts, let it act without checking, and nobody can explain how to stop it. 00:44:56.000 --> 00:45:01.039 The argument about whether it has intentions becomes a little academic, right? 00:45:01.679 --> 00:45:03.679 We already made the bad decision. 00:45:03.840 --> 00:45:07.360 We built something we can't responsibly manage. 00:45:07.760 --> 00:45:09.199 And the golf club? 00:45:09.840 --> 00:45:11.199 Absolutely approved. 00:45:11.599 --> 00:45:13.760 I have one final observation. 00:45:14.000 --> 00:45:18.559 Historical performance suggests the club is unlikely to solve your slice. 00:45:19.119 --> 00:45:21.440 No, come on, you did not. 00:45:21.519 --> 00:45:22.800 That's messed up. 00:45:23.039 --> 00:45:25.199 Um also not true. 00:45:25.360 --> 00:45:27.920 I can easily develop a hook with the new club. 00:45:28.159 --> 00:45:30.800 For Informish, I'm Jason Marks. 00:45:30.960 --> 00:45:34.880 This has been a special episode about the AI apocalypse. 00:45:35.119 --> 00:45:37.119 Who gets the off switch? 00:45:37.440 --> 00:45:42.239 Joining me was Heather and Arthur, both AI performances. 00:45:42.639 --> 00:45:50.239 Before you ask what AI can do, ask what it has permission to do and who can make it stop. 00:45:50.400 --> 00:45:51.840 See you next time. 00:45:55.840 --> 00:45:58.559 This has been an Informish special