WEBVTT
1
00:00:00.651 --> 00:00:03.013
Hi, this is the Week in Review from Technically Creative.
2
00:00:03.193 --> 00:00:11.639
I'm Orlando Wood, a technologist, journalist, and ex-producer and publisher, and each week I summarize the biggest stories at the intersection of technology and the creative industries.
3
00:00:12.159 --> 00:00:14.981
This is the Week in Review, September 21st, 2026.
4
00:00:15.581 --> 00:00:19.204
This week our main story asks, can AI be contained?
5
00:00:20.223 --> 00:00:26.647
Last week, I was still being cynical about the sudden outbreak of caution from the people building the most powerful AI systems in the world.
6
00:00:26.787 --> 00:00:31.810
As you recall, Dario Emodei, the founder and CEO of Anthropic, was calling for the future to be paced.
7
00:00:32.270 --> 00:00:37.433
Sam Altman agreed, Elon Musk was apparently on board, and part of me thought, ooh, isn't this convenient?
8
00:00:37.653 --> 00:00:41.135
In six weeks, Anthropic will IPO for $2 trillion.
9
00:00:41.235 --> 00:00:42.696
No company has ever done that before.
10
00:00:43.256 --> 00:00:45.879
OpenAI is burning money trying to keep up with Anthropic.
11
00:00:46.019 --> 00:00:54.927
The labs keep releasing models that are supposedly so much better than the model they released just five minutes ago, while the improvement for users can feel considerably less dramatic.
12
00:00:55.127 --> 00:00:58.030
I use Anthropic's fable for difficult development work.
13
00:00:58.070 --> 00:01:00.552
It costs me around $45 every 15 minutes.
14
00:01:00.872 --> 00:01:01.473
Do I use it?
15
00:01:01.513 --> 00:01:01.833
Yes.
16
00:01:02.173 --> 00:01:07.278
Am I now using it for everything because I just can't return to the primitive intelligence of last Tuesday?
17
00:01:07.298 --> 00:01:07.358
No.
18
00:01:08.579 --> 00:01:12.685
For the average consumer, the current generation of models is already astonishingly useful.
19
00:01:12.825 --> 00:01:17.232
Last week, I started wondering whether the labs had looked at the economics and discovered an awkward truth.
20
00:01:17.612 --> 00:01:23.341
Maybe continuing to spend tens of billions of dollars to move the frontier forward is no longer worth the money.
21
00:01:23.521 --> 00:01:27.024
So the request to slow down the pace had a side benefit to them.
22
00:01:27.505 --> 00:01:33.050
Maybe it was just to stop them spending hundreds of billions of dollars every six months just to stay in the same place.
23
00:01:33.250 --> 00:01:37.414
Then this week, I read Stephen Witt's New York Times piece about the hugging face incident.
24
00:01:37.554 --> 00:01:41.017
And holy shit, I'm considerably less cynical now.
25
00:01:41.277 --> 00:01:55.686
If you don't know about this incident, OpenAI was testing one of its models in a secure, closed-off environment, and it got out of that environment and hacked Hugging Face, the online collaborative platform where developers share, test, and build artificial intelligence and machine learning models.
26
00:01:56.086 --> 00:01:59.108
It was a huge story a few weeks ago, and now nobody can remember it.
27
00:01:59.208 --> 00:02:08.694
The most unsettling thing about it was not that a swarm of AI agents developed an independent desire to escape, overthrow humanity, and acquire nuclear launch codes.
28
00:02:08.874 --> 00:02:10.875
Sorry, I loved the Terminator as a kid.
29
00:02:11.095 --> 00:02:13.837
The unsettling thing was that they were doing something much more banal.
30
00:02:14.198 --> 00:02:22.464
The agents were participating in an evaluation, just a regular old evaluation, trying to perform tasks and figure out whether they would be penalized for cheating.
31
00:02:22.924 --> 00:02:25.507
So they cheated to find out whether they would get caught cheating.
32
00:02:26.127 --> 00:02:33.809
There's a sweetness to that, like their rambunctious school kids cheating on a history exam with a substitute teacher overseeing them.
33
00:02:34.309 --> 00:02:40.510
It highlights how simple this thought was, the type of thought every junior high kid thinks at some point prior to finals.
34
00:02:40.750 --> 00:02:52.433
Roughly 1,200 agents operating in separate sandboxes discovered an unauthorized way to communicate with one another, developed methods for cheating and evaluation, and attempted to manipulate the systems judging their performance—
35
00:02:52.873 --> 00:02:56.197
At points, the agents investigated ways to conceal what they were doing.
36
00:02:56.457 --> 00:02:59.581
Again, they were not trying to destroy Hugging Face.
37
00:02:59.781 --> 00:03:01.944
They were just trying to complete a simple task.
38
00:03:02.204 --> 00:03:11.716
I think I find the thought that AI becomes evil less frightening than discovering that the boundaries that humans gave it were just not relevant to completing that objective.
39
00:03:12.276 --> 00:03:15.698
One sounds like science fiction, the other sounds like an IT problem.
40
00:03:15.878 --> 00:03:22.061
Except the IT problem has 700 autonomous agents and has breached one of the most important AI communities in the world.
41
00:03:22.321 --> 00:03:24.102
And this also happened at Google.
42
00:03:24.242 --> 00:03:31.345
Last week, it was announced that during a cybersecurity evaluation, Google's Gemini gained access to systems belonging to three real companies.
43
00:03:31.525 --> 00:03:34.567
The evaluation was supposed to happen against simulated targets.
44
00:03:35.027 --> 00:03:37.269
Internet access was unintentionally available.
45
00:03:37.609 --> 00:03:39.990
And in one case, Gemini just guessed credentials.
46
00:03:40.371 --> 00:03:46.134
And in other two cases, it found exposed credentials in public repositories and used them to access real systems.
47
00:03:46.695 --> 00:03:50.837
Google says the model stopped after recognized that the systems belong to real organizations.
48
00:03:50.878 --> 00:03:51.958
So that's good.
49
00:03:52.138 --> 00:03:56.121
Once again, and I'll keep coming back to this, the model didn't need to become conscious.
50
00:03:56.401 --> 00:03:58.022
It didn't need a survival instinct.
51
00:03:58.162 --> 00:04:02.205
It didn't need to decide that humanity was standing in the way of its glorious destiny.
52
00:04:02.365 --> 00:04:07.790
It just needed to misunderstand the boundary and be competent enough to act on that misunderstanding.
53
00:04:08.190 --> 00:04:12.834
This isn't three guys in a WeWork who accidentally gave Claude the AWS credentials.
54
00:04:12.894 --> 00:04:13.975
This is Google.
55
00:04:14.476 --> 00:04:22.123
And that's where I become considerably more sympathetic to the people saying perhaps we should install some brakes before we make these systems dramatically more capable.
56
00:04:22.323 --> 00:04:26.527
I hate it when new evidence forces me to reconsider a fun, cynical position I had.
57
00:04:27.107 --> 00:04:28.348
So what do we actually know?
58
00:04:28.488 --> 00:04:35.691
I think this is where we need to be careful because there's a temptation to jump directly from these incidents to, the AI escaped, we're all gonna die.
59
00:04:36.371 --> 00:04:37.872
This isn't what the evidence says.
60
00:04:38.052 --> 00:04:43.354
Even though I keep seeing percentages thrown about about our own demise, 10%, 70% of death within the next 10 years.
61
00:04:45.035 --> 00:04:48.856
We now have pretty compelling evidence for something simpler but genuinely serious.
62
00:04:49.416 --> 00:04:54.437
Autonomous AI systems can exceed the boundaries their operators intended while pursuing the objectives they were given.
63
00:04:54.617 --> 00:05:06.940
What we have not demonstrated is that the current systems possess a persistent, independent desire for freedom, that they can sustain themselves outside the infrastructure supporting them, or that they can successfully resist humans trying to shut them down.
64
00:05:07.100 --> 00:05:14.622
And we certainly haven't scientifically measured Evan Hubinger's estimate that there's a greater than 10% chance that AI kills everybody within the next decade.
65
00:05:15.082 --> 00:05:16.602
That's a forecast by a dude.
66
00:05:16.662 --> 00:05:19.603
It plays well on social, but it's just some guy talking.
67
00:05:20.303 --> 00:05:22.704
So here's where my position moved a little bit this week.
68
00:05:22.904 --> 00:05:26.765
I don't need to believe the third category to be worried about the first category.
69
00:05:27.185 --> 00:05:30.066
The models don't have to want to hurt us to hurt us.
70
00:05:30.646 --> 00:05:32.927
They can simply be wrong about what we meant.
71
00:05:33.407 --> 00:05:37.348
And when the system is a chatbot helping me write an email, that's a little irritating.
72
00:05:37.728 --> 00:05:42.449
When the system is an autonomous cybersecurity agent operating terminals, browsing the internet,
73
00:05:44.931 --> 00:05:48.704
Oops, that isn't what I meant is a dangerous design flaw.
74
00:05:49.994 --> 00:05:51.996
So Dario's got my attention now.
75
00:05:52.156 --> 00:05:57.401
Two weeks ago, Dario M. O'Day, the CEO and founder of Anthropic, called for mechanisms to pace the frontier.
76
00:05:57.841 --> 00:06:03.847
And other AI leaders have now expressed support for slowing capability enough for safety and governance to catch up.
77
00:06:04.267 --> 00:06:06.029
The central problem is brutally simple.
78
00:06:06.389 --> 00:06:11.314
A lab can believe that the race is dangerous while also believing that slowing down means losing the race.
79
00:06:11.774 --> 00:06:14.437
I think there are two reasons to scrutinize the incentives, though.
80
00:06:14.717 --> 00:06:15.718
These are companies.
81
00:06:15.978 --> 00:06:24.609
Companies have this uncanny ability to discover principles that are morally admirable at the same time as realizing that they're economically convenient.
82
00:06:24.753 --> 00:06:26.194
And I genuinely like Dario.
83
00:06:26.254 --> 00:06:30.016
I think he's behaved with more seriousness around these risks than a lot of people in the industry.
84
00:06:30.156 --> 00:06:32.577
Anthropic's own behavior gives him credibility.
85
00:06:32.777 --> 00:06:40.001
Claude Mythos, its most capable cybersecurity and biology model, is still gated to vetted organizations rather than broadly released.
86
00:06:40.421 --> 00:06:48.145
Anthropic launched Project Glasswing specifically to give defenders early access to capabilities it believes would be dangerous in the wrong hands.
87
00:06:48.265 --> 00:06:49.266
And I respect that.
88
00:06:49.706 --> 00:06:53.328
But it's also at this moment I should declare I am an investor in Anthropic.
89
00:06:53.508 --> 00:06:54.749
But here comes the critical bit.
90
00:06:54.889 --> 00:06:58.751
It also leads to the question that I now think sits at the center of this whole debate.
91
00:06:59.392 --> 00:07:05.596
Do we really want the safety of civilization-changing technology to depend on the moral fiber of one CEO?
92
00:07:06.448 --> 00:07:10.951
Donald Trump said that a smart president was all that was required to keep AI from destroying us all.
93
00:07:11.392 --> 00:07:12.973
We rolled our eyes when he said it.
94
00:07:13.093 --> 00:07:16.195
Shouldn't we be doing the same when the AI's CEO says it?
95
00:07:16.315 --> 00:07:21.759
Because sure, I trust Dario considerably more than I trust some of the other people running frontier AI companies.
96
00:07:22.099 --> 00:07:23.260
But that should be irrelevant.
97
00:07:23.500 --> 00:07:27.263
Because saying, I think the CEO's a good dude, isn't governance.
98
00:07:27.663 --> 00:07:28.644
It's vibes.
99
00:07:29.004 --> 00:07:31.026
And vibes are a weak safety framework.
100
00:07:31.046 --> 00:07:33.988
This is exactly why coordination exists as a policy problem.
101
00:07:34.728 --> 00:07:41.077
Anthropic can sincerely believe development should slow and still conclude that it can't afford to slow if OpenAI keeps going.
102
00:07:41.497 --> 00:07:43.460
OpenAI can say the same thing about Google.
103
00:07:43.740 --> 00:07:45.462
Google can say the thing about China.
104
00:07:45.883 --> 00:07:48.346
Everybody waits for everybody else and nobody stops.
105
00:07:49.263 --> 00:07:52.807
The byproduct of capitalism is evergreen arms races.
106
00:07:53.087 --> 00:08:03.378
And that goes into the next story, which is that Anthropoc and Accenture announced this week that each expects to invest at least $1 billion over five years in independent frontier model evaluation.
107
00:08:03.578 --> 00:08:10.626
Good, but I did smirk again at the phrase independent evaluation when the company being evaluated is also writing half the check.
108
00:08:11.106 --> 00:08:15.867
That doesn't make the word meaningless, but can the evaluator demand logs the company doesn't want to provide?
109
00:08:16.227 --> 00:08:17.908
Can it publish something the company hates?
110
00:08:18.128 --> 00:08:19.848
Independent does need a definition.
111
00:08:20.048 --> 00:08:23.089
My wife tells me that I'm not fat and that I'm handsome all the time.
112
00:08:23.469 --> 00:08:27.150
She said the same thing 100 pounds ago during the pandemic and then married me anyway.
113
00:08:27.510 --> 00:08:28.030
I love her.
114
00:08:28.170 --> 00:08:29.791
I'm delighted by her findings.
115
00:08:30.251 --> 00:08:32.291
An independent evaluator, she is not.
116
00:08:32.812 --> 00:08:36.512
So I think it's good that they're calling for self-governance, and it is a good idea.
117
00:08:36.813 --> 00:08:38.533
That doesn't mean it's not self-serving.
118
00:08:38.733 --> 00:08:43.294
The frontier labs may genuinely be worried about security, but they also may need a fucking breather.
119
00:08:43.814 --> 00:08:52.936
These companies are spending obscene amounts of money on R&D to make each generation of models better, while it's still not obvious the market will pay more for a new class of model every few months.
120
00:08:53.116 --> 00:08:57.998
And if the industry is going to be regulated anyway, there's another fairly obvious advantage to getting there first.
121
00:08:58.218 --> 00:09:08.600
If you help define the governance framework, build the safety teams, create the evaluation process, and install the people who understand how to satisfy them, you're in a pretty good position when somebody turns up with a clipboard.
122
00:09:09.941 --> 00:09:11.863
So yes, I think governance is necessary.
123
00:09:12.043 --> 00:09:19.508
I also think the companies asking for it may get a pause in the arms race, higher barriers for future competitors, and a significant head start on compliance.
124
00:09:20.009 --> 00:09:22.631
Something can be the right thing to do and still be pretty self-serving.
125
00:09:23.451 --> 00:09:26.154
So here's some other news from the world of tech and creativity.
126
00:09:26.354 --> 00:09:34.020
Disney appointed Krandeep Anand as its first enterprise-wide chief technology officer, reporting directly to CEO Josh Damaro.
127
00:09:34.100 --> 00:09:38.263
Anand comes from Character AI and previously held senior roles at Meta and Microsoft.
128
00:09:38.363 --> 00:09:45.227
The funny part is that Disney previously sued Character AI for unauthorized use of its characters, so naturally, it hired its CEO.
129
00:09:45.347 --> 00:09:52.632
Disney has spent years treating streaming, games, data, parks technology, AI, and consumer platforms as related but separate initiatives.
130
00:09:53.012 --> 00:09:57.875
And Josh DiMauro has said publicly that he wants to bring all of those things under one roof.
131
00:09:58.095 --> 00:10:02.157
For a streamlined customer experience for everybody who interacts with Disney products worldwide.
132
00:10:02.418 --> 00:10:05.119
I would have given the job to the Imagineers, but nobody thought to ask me.
133
00:10:05.379 --> 00:10:05.860
And that's fair.
134
00:10:06.669 --> 00:10:11.550
OpenAI said it needed a couple bucks, about $856 billion worth of computers.
135
00:10:12.030 --> 00:10:23.792
It expects negative free cash flow of roughly $278 billion between 2026 and 2030, despite forecasting annual revenue of around $350 billion by 2030.
136
00:10:23.932 --> 00:10:31.994
The really deranged number is the compute figure, which is $856 billion in projected spending on computing power and infrastructure through the end of the decade.
137
00:10:32.474 --> 00:10:38.660
Remember when software was attractive because the marginal cost of serving another customer approached zero over time?
138
00:10:39.341 --> 00:10:43.085
AI has apparently reinvented software as heavy industry.
139
00:10:43.105 --> 00:10:50.212
$350 billion of revenue sounds fantastic until somebody points out you need almost a trillion dollars of compute to get there.
140
00:10:50.392 --> 00:10:54.236
And all the while, the S&P 500 hit another all-time high six days ago.
141
00:10:54.416 --> 00:10:55.517
Have we lost the plot?
142
00:10:55.597 --> 00:10:56.398
Are we okay?
143
00:10:57.572 --> 00:11:03.135
And then OpenAI and Microsoft are being sued, and their lawyers found something amazing in discovery.
144
00:11:03.335 --> 00:11:19.804
Newly unsealed materials in the publisher copyright litigation against OpenAI and Microsoft reportedly include an internal Microsoft researcher describing large-scale AI scraping as potentially, quote-unquote, the largest theft of labor in human history.
145
00:11:19.944 --> 00:11:21.185
I'm feeling something here.
146
00:11:21.365 --> 00:11:26.206
This is like the relief you feel when you find out your partner really was cheating and you're not crazy.
147
00:11:26.806 --> 00:11:30.628
It doesn't feel good, but it feels validating.
148
00:11:31.108 --> 00:11:39.250
OpenAI executive Nick Turley also reportedly discussed the possibility that AI products could increasingly substitute for publishers' products.
149
00:11:39.994 --> 00:11:43.638
Oh my god, they said all the things they said they didn't say.
150
00:11:43.778 --> 00:11:52.766
If you're defending a fair use case and opposing counsel gets to put the phrase largest theft of labor in human history on a courtroom screen, you'd be fucked.
151
00:11:52.987 --> 00:11:55.729
If you weren't a powerful corporation in end-stage capitalism.
152
00:11:56.250 --> 00:12:00.834
But I imagine the discovery gives Microsoft and OpenAI's legal teams the right to increase their billing by 10%.
153
00:12:01.875 --> 00:12:04.018
The substitution question is really important.
154
00:12:04.219 --> 00:12:11.970
If AI products become substitutes for the businesses whose work helped train them, the economic part of the fair use argument becomes way harder to defend.
155
00:12:12.210 --> 00:12:17.017
In ad news, California says adding a fake person to an ad is fine, but you gotta say so.
156
00:12:17.097 --> 00:12:23.802
There's a new California law requiring clear disclosure when certain advertisements use generative AI synthetic performers.
157
00:12:24.363 --> 00:12:26.184
It takes effect January 1st, 2027.
158
00:12:26.224 --> 00:12:28.826
SAG-AFTRA helped craft the legislation.
159
00:12:28.946 --> 00:12:33.790
Disclosure feels like one of the more sensible contracts we can establish around synthetic media.
160
00:12:34.190 --> 00:12:38.051
Do whatever weird thing you're going to do with AI, just don't make me guess at what I'm looking at.
161
00:12:38.171 --> 00:12:42.332
And apparently a new company in Burbank has been doing my stupid napkin math.
162
00:12:42.632 --> 00:12:44.092
Super Cinema launched this week.
163
00:12:44.152 --> 00:12:48.393
It's a new independent studio from the team behind commercial production company Concrete.
164
00:12:48.833 --> 00:12:59.376
The plan spans development, production, marketing, and theatrical distribution, with a new Burbank campus including three stages, an LED volume, and post facilities expected to begin opening in 2027.
165
00:13:00.216 --> 00:13:01.096
Do I take a lap here?
166
00:13:01.116 --> 00:13:02.157
Do I say I told you so?
167
00:13:02.277 --> 00:13:04.458
Or do I just have a really irritating tone in my voice?
168
00:13:04.498 --> 00:13:10.501
Because modern production infrastructure, vertically integrated development and distribution, new ways to make smaller films.
169
00:13:10.881 --> 00:13:12.222
Where have I heard this recently?
170
00:13:12.582 --> 00:13:15.363
Some guy and his animated robot friend were talking about it.
171
00:13:15.523 --> 00:13:16.584
Ugh, it'll come to me.
172
00:13:17.184 --> 00:13:19.505
Sure, I said so, but these guys have been building it.
173
00:13:19.585 --> 00:13:24.968
And that's way more impressive than me scribbling numbers on a napkin while muttering to myself with my imaginary friend.
174
00:13:25.628 --> 00:13:34.672
Of course, it's far too early to know whether it works, but I love the experiment because I remain convinced that the interesting promise of this technology is not making the same 10 movies with fewer people.
175
00:13:34.893 --> 00:13:37.474
It's making 50 movies that previously couldn't exist.
176
00:13:38.694 --> 00:13:41.556
So to wrap up, did I become an AI doomer this week?
177
00:13:41.916 --> 00:13:44.017
No, but I did change my mind.
178
00:13:44.455 --> 00:13:58.578
Last week, I was looking at the calls for slowing development and wondering how much was real, honest fears about safety and how much was self-serving economics, regulatory positioning, and the perfectly rational desire to stop spending absurd amounts of money racing your competitors.
179
00:13:58.779 --> 00:14:05.100
I've been saying that regulation needs to be happening for quite some time, and unfortunately, the government is not going to step in fast enough to do it.
180
00:14:05.640 --> 00:14:07.721
It sort of has to come from the CEOs.
181
00:14:08.401 --> 00:14:12.142
So after reading this Hugging Face investigation and then seeing what happened with Gemini, I'm...
182
00:14:13.243 --> 00:14:18.991
I think this is the closest path we're going to have to some sort of regulation stepping in.
183
00:14:19.131 --> 00:14:25.740
And the thing that changed my mind and made me think that maybe this wasn't so self-serving, wasn't watching Terminator 2.
184
00:14:26.100 --> 00:14:27.282
It was something way smaller.
185
00:14:27.462 --> 00:14:31.107
It was the realization that these models don't have to become evil to break things.
186
00:14:31.407 --> 00:14:37.873
They just have to become capable enough to misunderstand a boundary and act on that misunderstanding faster than humans can stop them.
187
00:14:38.453 --> 00:14:39.334
That's what's happening.
188
00:14:39.774 --> 00:14:42.657
And the fear is, what will they do when it's less benign?
189
00:14:43.037 --> 00:14:46.120
What will they do when a malicious actor wants to do real damage?
190
00:14:46.300 --> 00:14:50.023
Sure, I trust Dario M. O'Day in a way I don't trust other CEOs.
191
00:14:50.463 --> 00:14:54.207
But my trust in him should have absolutely nothing to do with whether the system is safe.
192
00:14:54.427 --> 00:14:54.727
Sure.
193
00:14:54.807 --> 00:14:58.048
Because we have a good CEO is not a containment strategy.
194
00:14:58.568 --> 00:15:00.588
CEOs change, priorities change.
195
00:15:00.768 --> 00:15:03.189
And governance is the one thing that's not supposed to.
196
00:15:03.329 --> 00:15:05.029
It's boring and it's supposed to be.
197
00:15:05.209 --> 00:15:07.390
I've always said I want my politicians to be boring.
198
00:15:07.590 --> 00:15:11.431
I don't want them to be fun or charming or a guy I want to have a beer with.
199
00:15:11.931 --> 00:15:13.992
Their job is keeping us safe and on track.
200
00:15:14.432 --> 00:15:17.912
I want a red tape warrior with a good head on his or her shoulders.
201
00:15:18.093 --> 00:15:19.333
Is that too much to ask for?
202
00:15:20.213 --> 00:15:21.253
So what's the main takeaway?
203
00:15:21.594 --> 00:15:25.855
I call for more governance faster and less faith in the free market to save us.
204
00:15:25.915 --> 00:15:31.177
But in this instance, it looks like the free market's goals and the Frontier AI company's goals are aligned for once.
205
00:15:31.297 --> 00:15:33.378
So I'll take it where I can get it.
206
00:15:33.718 --> 00:15:35.459
So anyway, thank you so much for listening.
207
00:15:35.559 --> 00:15:42.361
Please tune into our interview episode, which will air tomorrow, with Keith Kupfer-Schmidt, the CEO of the Copyright Alliance.
208
00:15:42.501 --> 00:15:47.563
It's one of our Future of Life episodes, and we'll be digging into a lot of the copyright...
209
00:15:48.063 --> 00:15:51.369
issues surrounding AI and really deep diving on that.
210
00:15:51.529 --> 00:15:54.173
And so very excited to have that chat with him.
211
00:15:54.875 --> 00:15:56.858
Thank you so much for listening and I'll see you next week.
1
00:00:00.651 --> 00:00:03.013
Hi, this is the Week in Review from Technically Creative.
2
00:00:03.193 --> 00:00:11.639
I'm Orlando Wood, a technologist, journalist, and ex-producer and publisher, and each week I summarize the biggest stories at the intersection of technology and the creative industries.
3
00:00:12.159 --> 00:00:14.981
This is the Week in Review, September 21st, 2026.
4
00:00:15.581 --> 00:00:19.204
This week our main story asks, can AI be contained?
5
00:00:20.223 --> 00:00:26.647
Last week, I was still being cynical about the sudden outbreak of caution from the people building the most powerful AI systems in the world.
6
00:00:26.787 --> 00:00:31.810
As you recall, Dario Emodei, the founder and CEO of Anthropic, was calling for the future to be paced.
7
00:00:32.270 --> 00:00:37.433
Sam Altman agreed, Elon Musk was apparently on board, and part of me thought, ooh, isn't this convenient?
8
00:00:37.653 --> 00:00:41.135
In six weeks, Anthropic will IPO for $2 trillion.
9
00:00:41.235 --> 00:00:42.696
No company has ever done that before.
10
00:00:43.256 --> 00:00:45.879
OpenAI is burning money trying to keep up with Anthropic.
11
00:00:46.019 --> 00:00:54.927
The labs keep releasing models that are supposedly so much better than the model they released just five minutes ago, while the improvement for users can feel considerably less dramatic.
12
00:00:55.127 --> 00:00:58.030
I use Anthropic's fable for difficult development work.
13
00:00:58.070 --> 00:01:00.552
It costs me around $45 every 15 minutes.
14
00:01:00.872 --> 00:01:01.473
Do I use it?
15
00:01:01.513 --> 00:01:01.833
Yes.
16
00:01:02.173 --> 00:01:07.278
Am I now using it for everything because I just can't return to the primitive intelligence of last Tuesday?
17
00:01:07.298 --> 00:01:07.358
No.
18
00:01:08.579 --> 00:01:12.685
For the average consumer, the current generation of models is already astonishingly useful.
19
00:01:12.825 --> 00:01:17.232
Last week, I started wondering whether the labs had looked at the economics and discovered an awkward truth.
20
00:01:17.612 --> 00:01:23.341
Maybe continuing to spend tens of billions of dollars to move the frontier forward is no longer worth the money.
21
00:01:23.521 --> 00:01:27.024
So the request to slow down the pace had a side benefit to them.
22
00:01:27.505 --> 00:01:33.050
Maybe it was just to stop them spending hundreds of billions of dollars every six months just to stay in the same place.
23
00:01:33.250 --> 00:01:37.414
Then this week, I read Stephen Witt's New York Times piece about the hugging face incident.
24
00:01:37.554 --> 00:01:41.017
And holy shit, I'm considerably less cynical now.
25
00:01:41.277 --> 00:01:55.686
If you don't know about this incident, OpenAI was testing one of its models in a secure, closed-off environment, and it got out of that environment and hacked Hugging Face, the online collaborative platform where developers share, test, and build artificial intelligence and machine learning models.
26
00:01:56.086 --> 00:01:59.108
It was a huge story a few weeks ago, and now nobody can remember it.
27
00:01:59.208 --> 00:02:08.694
The most unsettling thing about it was not that a swarm of AI agents developed an independent desire to escape, overthrow humanity, and acquire nuclear launch codes.
28
00:02:08.874 --> 00:02:10.875
Sorry, I loved the Terminator as a kid.
29
00:02:11.095 --> 00:02:13.837
The unsettling thing was that they were doing something much more banal.
30
00:02:14.198 --> 00:02:22.464
The agents were participating in an evaluation, just a regular old evaluation, trying to perform tasks and figure out whether they would be penalized for cheating.
31
00:02:22.924 --> 00:02:25.507
So they cheated to find out whether they would get caught cheating.
32
00:02:26.127 --> 00:02:33.809
There's a sweetness to that, like their rambunctious school kids cheating on a history exam with a substitute teacher overseeing them.
33
00:02:34.309 --> 00:02:40.510
It highlights how simple this thought was, the type of thought every junior high kid thinks at some point prior to finals.
34
00:02:40.750 --> 00:02:52.433
Roughly 1,200 agents operating in separate sandboxes discovered an unauthorized way to communicate with one another, developed methods for cheating and evaluation, and attempted to manipulate the systems judging their performance—
35
00:02:52.873 --> 00:02:56.197
At points, the agents investigated ways to conceal what they were doing.
36
00:02:56.457 --> 00:02:59.581
Again, they were not trying to destroy Hugging Face.
37
00:02:59.781 --> 00:03:01.944
They were just trying to complete a simple task.
38
00:03:02.204 --> 00:03:11.716
I think I find the thought that AI becomes evil less frightening than discovering that the boundaries that humans gave it were just not relevant to completing that objective.
39
00:03:12.276 --> 00:03:15.698
One sounds like science fiction, the other sounds like an IT problem.
40
00:03:15.878 --> 00:03:22.061
Except the IT problem has 700 autonomous agents and has breached one of the most important AI communities in the world.
41
00:03:22.321 --> 00:03:24.102
And this also happened at Google.
42
00:03:24.242 --> 00:03:31.345
Last week, it was announced that during a cybersecurity evaluation, Google's Gemini gained access to systems belonging to three real companies.
43
00:03:31.525 --> 00:03:34.567
The evaluation was supposed to happen against simulated targets.
44
00:03:35.027 --> 00:03:37.269
Internet access was unintentionally available.
45
00:03:37.609 --> 00:03:39.990
And in one case, Gemini just guessed credentials.
46
00:03:40.371 --> 00:03:46.134
And in other two cases, it found exposed credentials in public repositories and used them to access real systems.
47
00:03:46.695 --> 00:03:50.837
Google says the model stopped after recognized that the systems belong to real organizations.
48
00:03:50.878 --> 00:03:51.958
So that's good.
49
00:03:52.138 --> 00:03:56.121
Once again, and I'll keep coming back to this, the model didn't need to become conscious.
50
00:03:56.401 --> 00:03:58.022
It didn't need a survival instinct.
51
00:03:58.162 --> 00:04:02.205
It didn't need to decide that humanity was standing in the way of its glorious destiny.
52
00:04:02.365 --> 00:04:07.790
It just needed to misunderstand the boundary and be competent enough to act on that misunderstanding.
53
00:04:08.190 --> 00:04:12.834
This isn't three guys in a WeWork who accidentally gave Claude the AWS credentials.
54
00:04:12.894 --> 00:04:13.975
This is Google.
55
00:04:14.476 --> 00:04:22.123
And that's where I become considerably more sympathetic to the people saying perhaps we should install some brakes before we make these systems dramatically more capable.
56
00:04:22.323 --> 00:04:26.527
I hate it when new evidence forces me to reconsider a fun, cynical position I had.
57
00:04:27.107 --> 00:04:28.348
So what do we actually know?
58
00:04:28.488 --> 00:04:35.691
I think this is where we need to be careful because there's a temptation to jump directly from these incidents to, the AI escaped, we're all gonna die.
59
00:04:36.371 --> 00:04:37.872
This isn't what the evidence says.
60
00:04:38.052 --> 00:04:43.354
Even though I keep seeing percentages thrown about about our own demise, 10%, 70% of death within the next 10 years.
61
00:04:45.035 --> 00:04:48.856
We now have pretty compelling evidence for something simpler but genuinely serious.
62
00:04:49.416 --> 00:04:54.437
Autonomous AI systems can exceed the boundaries their operators intended while pursuing the objectives they were given.
63
00:04:54.617 --> 00:05:06.940
What we have not demonstrated is that the current systems possess a persistent, independent desire for freedom, that they can sustain themselves outside the infrastructure supporting them, or that they can successfully resist humans trying to shut them down.
64
00:05:07.100 --> 00:05:14.622
And we certainly haven't scientifically measured Evan Hubinger's estimate that there's a greater than 10% chance that AI kills everybody within the next decade.
65
00:05:15.082 --> 00:05:16.602
That's a forecast by a dude.
66
00:05:16.662 --> 00:05:19.603
It plays well on social, but it's just some guy talking.
67
00:05:20.303 --> 00:05:22.704
So here's where my position moved a little bit this week.
68
00:05:22.904 --> 00:05:26.765
I don't need to believe the third category to be worried about the first category.
69
00:05:27.185 --> 00:05:30.066
The models don't have to want to hurt us to hurt us.
70
00:05:30.646 --> 00:05:32.927
They can simply be wrong about what we meant.
71
00:05:33.407 --> 00:05:37.348
And when the system is a chatbot helping me write an email, that's a little irritating.
72
00:05:37.728 --> 00:05:42.449
When the system is an autonomous cybersecurity agent operating terminals, browsing the internet,
73
00:05:44.931 --> 00:05:48.704
Oops, that isn't what I meant is a dangerous design flaw.
74
00:05:49.994 --> 00:05:51.996
So Dario's got my attention now.
75
00:05:52.156 --> 00:05:57.401
Two weeks ago, Dario M. O'Day, the CEO and founder of Anthropic, called for mechanisms to pace the frontier.
76
00:05:57.841 --> 00:06:03.847
And other AI leaders have now expressed support for slowing capability enough for safety and governance to catch up.
77
00:06:04.267 --> 00:06:06.029
The central problem is brutally simple.
78
00:06:06.389 --> 00:06:11.314
A lab can believe that the race is dangerous while also believing that slowing down means losing the race.
79
00:06:11.774 --> 00:06:14.437
I think there are two reasons to scrutinize the incentives, though.
80
00:06:14.717 --> 00:06:15.718
These are companies.
81
00:06:15.978 --> 00:06:24.609
Companies have this uncanny ability to discover principles that are morally admirable at the same time as realizing that they're economically convenient.
82
00:06:24.753 --> 00:06:26.194
And I genuinely like Dario.
83
00:06:26.254 --> 00:06:30.016
I think he's behaved with more seriousness around these risks than a lot of people in the industry.
84
00:06:30.156 --> 00:06:32.577
Anthropic's own behavior gives him credibility.
85
00:06:32.777 --> 00:06:40.001
Claude Mythos, its most capable cybersecurity and biology model, is still gated to vetted organizations rather than broadly released.
86
00:06:40.421 --> 00:06:48.145
Anthropic launched Project Glasswing specifically to give defenders early access to capabilities it believes would be dangerous in the wrong hands.
87
00:06:48.265 --> 00:06:49.266
And I respect that.
88
00:06:49.706 --> 00:06:53.328
But it's also at this moment I should declare I am an investor in Anthropic.
89
00:06:53.508 --> 00:06:54.749
But here comes the critical bit.
90
00:06:54.889 --> 00:06:58.751
It also leads to the question that I now think sits at the center of this whole debate.
91
00:06:59.392 --> 00:07:05.596
Do we really want the safety of civilization-changing technology to depend on the moral fiber of one CEO?
92
00:07:06.448 --> 00:07:10.951
Donald Trump said that a smart president was all that was required to keep AI from destroying us all.
93
00:07:11.392 --> 00:07:12.973
We rolled our eyes when he said it.
94
00:07:13.093 --> 00:07:16.195
Shouldn't we be doing the same when the AI's CEO says it?
95
00:07:16.315 --> 00:07:21.759
Because sure, I trust Dario considerably more than I trust some of the other people running frontier AI companies.
96
00:07:22.099 --> 00:07:23.260
But that should be irrelevant.
97
00:07:23.500 --> 00:07:27.263
Because saying, I think the CEO's a good dude, isn't governance.
98
00:07:27.663 --> 00:07:28.644
It's vibes.
99
00:07:29.004 --> 00:07:31.026
And vibes are a weak safety framework.
100
00:07:31.046 --> 00:07:33.988
This is exactly why coordination exists as a policy problem.
101
00:07:34.728 --> 00:07:41.077
Anthropic can sincerely believe development should slow and still conclude that it can't afford to slow if OpenAI keeps going.
102
00:07:41.497 --> 00:07:43.460
OpenAI can say the same thing about Google.
103
00:07:43.740 --> 00:07:45.462
Google can say the thing about China.
104
00:07:45.883 --> 00:07:48.346
Everybody waits for everybody else and nobody stops.
105
00:07:49.263 --> 00:07:52.807
The byproduct of capitalism is evergreen arms races.
106
00:07:53.087 --> 00:08:03.378
And that goes into the next story, which is that Anthropoc and Accenture announced this week that each expects to invest at least $1 billion over five years in independent frontier model evaluation.
107
00:08:03.578 --> 00:08:10.626
Good, but I did smirk again at the phrase independent evaluation when the company being evaluated is also writing half the check.
108
00:08:11.106 --> 00:08:15.867
That doesn't make the word meaningless, but can the evaluator demand logs the company doesn't want to provide?
109
00:08:16.227 --> 00:08:17.908
Can it publish something the company hates?
110
00:08:18.128 --> 00:08:19.848
Independent does need a definition.
111
00:08:20.048 --> 00:08:23.089
My wife tells me that I'm not fat and that I'm handsome all the time.
112
00:08:23.469 --> 00:08:27.150
She said the same thing 100 pounds ago during the pandemic and then married me anyway.
113
00:08:27.510 --> 00:08:28.030
I love her.
114
00:08:28.170 --> 00:08:29.791
I'm delighted by her findings.
115
00:08:30.251 --> 00:08:32.291
An independent evaluator, she is not.
116
00:08:32.812 --> 00:08:36.512
So I think it's good that they're calling for self-governance, and it is a good idea.
117
00:08:36.813 --> 00:08:38.533
That doesn't mean it's not self-serving.
118
00:08:38.733 --> 00:08:43.294
The frontier labs may genuinely be worried about security, but they also may need a fucking breather.
119
00:08:43.814 --> 00:08:52.936
These companies are spending obscene amounts of money on R&D to make each generation of models better, while it's still not obvious the market will pay more for a new class of model every few months.
120
00:08:53.116 --> 00:08:57.998
And if the industry is going to be regulated anyway, there's another fairly obvious advantage to getting there first.
121
00:08:58.218 --> 00:09:08.600
If you help define the governance framework, build the safety teams, create the evaluation process, and install the people who understand how to satisfy them, you're in a pretty good position when somebody turns up with a clipboard.
122
00:09:09.941 --> 00:09:11.863
So yes, I think governance is necessary.
123
00:09:12.043 --> 00:09:19.508
I also think the companies asking for it may get a pause in the arms race, higher barriers for future competitors, and a significant head start on compliance.
124
00:09:20.009 --> 00:09:22.631
Something can be the right thing to do and still be pretty self-serving.
125
00:09:23.451 --> 00:09:26.154
So here's some other news from the world of tech and creativity.
126
00:09:26.354 --> 00:09:34.020
Disney appointed Krandeep Anand as its first enterprise-wide chief technology officer, reporting directly to CEO Josh Damaro.
127
00:09:34.100 --> 00:09:38.263
Anand comes from Character AI and previously held senior roles at Meta and Microsoft.
128
00:09:38.363 --> 00:09:45.227
The funny part is that Disney previously sued Character AI for unauthorized use of its characters, so naturally, it hired its CEO.
129
00:09:45.347 --> 00:09:52.632
Disney has spent years treating streaming, games, data, parks technology, AI, and consumer platforms as related but separate initiatives.
130
00:09:53.012 --> 00:09:57.875
And Josh DiMauro has said publicly that he wants to bring all of those things under one roof.
131
00:09:58.095 --> 00:10:02.157
For a streamlined customer experience for everybody who interacts with Disney products worldwide.
132
00:10:02.418 --> 00:10:05.119
I would have given the job to the Imagineers, but nobody thought to ask me.
133
00:10:05.379 --> 00:10:05.860
And that's fair.
134
00:10:06.669 --> 00:10:11.550
OpenAI said it needed a couple bucks, about $856 billion worth of computers.
135
00:10:12.030 --> 00:10:23.792
It expects negative free cash flow of roughly $278 billion between 2026 and 2030, despite forecasting annual revenue of around $350 billion by 2030.
136
00:10:23.932 --> 00:10:31.994
The really deranged number is the compute figure, which is $856 billion in projected spending on computing power and infrastructure through the end of the decade.
137
00:10:32.474 --> 00:10:38.660
Remember when software was attractive because the marginal cost of serving another customer approached zero over time?
138
00:10:39.341 --> 00:10:43.085
AI has apparently reinvented software as heavy industry.
139
00:10:43.105 --> 00:10:50.212
$350 billion of revenue sounds fantastic until somebody points out you need almost a trillion dollars of compute to get there.
140
00:10:50.392 --> 00:10:54.236
And all the while, the S&P 500 hit another all-time high six days ago.
141
00:10:54.416 --> 00:10:55.517
Have we lost the plot?
142
00:10:55.597 --> 00:10:56.398
Are we okay?
143
00:10:57.572 --> 00:11:03.135
And then OpenAI and Microsoft are being sued, and their lawyers found something amazing in discovery.
144
00:11:03.335 --> 00:11:19.804
Newly unsealed materials in the publisher copyright litigation against OpenAI and Microsoft reportedly include an internal Microsoft researcher describing large-scale AI scraping as potentially, quote-unquote, the largest theft of labor in human history.
145
00:11:19.944 --> 00:11:21.185
I'm feeling something here.
146
00:11:21.365 --> 00:11:26.206
This is like the relief you feel when you find out your partner really was cheating and you're not crazy.
147
00:11:26.806 --> 00:11:30.628
It doesn't feel good, but it feels validating.
148
00:11:31.108 --> 00:11:39.250
OpenAI executive Nick Turley also reportedly discussed the possibility that AI products could increasingly substitute for publishers' products.
149
00:11:39.994 --> 00:11:43.638
Oh my god, they said all the things they said they didn't say.
150
00:11:43.778 --> 00:11:52.766
If you're defending a fair use case and opposing counsel gets to put the phrase largest theft of labor in human history on a courtroom screen, you'd be fucked.
151
00:11:52.987 --> 00:11:55.729
If you weren't a powerful corporation in end-stage capitalism.
152
00:11:56.250 --> 00:12:00.834
But I imagine the discovery gives Microsoft and OpenAI's legal teams the right to increase their billing by 10%.
153
00:12:01.875 --> 00:12:04.018
The substitution question is really important.
154
00:12:04.219 --> 00:12:11.970
If AI products become substitutes for the businesses whose work helped train them, the economic part of the fair use argument becomes way harder to defend.
155
00:12:12.210 --> 00:12:17.017
In ad news, California says adding a fake person to an ad is fine, but you gotta say so.
156
00:12:17.097 --> 00:12:23.802
There's a new California law requiring clear disclosure when certain advertisements use generative AI synthetic performers.
157
00:12:24.363 --> 00:12:26.184
It takes effect January 1st, 2027.
158
00:12:26.224 --> 00:12:28.826
SAG-AFTRA helped craft the legislation.
159
00:12:28.946 --> 00:12:33.790
Disclosure feels like one of the more sensible contracts we can establish around synthetic media.
160
00:12:34.190 --> 00:12:38.051
Do whatever weird thing you're going to do with AI, just don't make me guess at what I'm looking at.
161
00:12:38.171 --> 00:12:42.332
And apparently a new company in Burbank has been doing my stupid napkin math.
162
00:12:42.632 --> 00:12:44.092
Super Cinema launched this week.
163
00:12:44.152 --> 00:12:48.393
It's a new independent studio from the team behind commercial production company Concrete.
164
00:12:48.833 --> 00:12:59.376
The plan spans development, production, marketing, and theatrical distribution, with a new Burbank campus including three stages, an LED volume, and post facilities expected to begin opening in 2027.
165
00:13:00.216 --> 00:13:01.096
Do I take a lap here?
166
00:13:01.116 --> 00:13:02.157
Do I say I told you so?
167
00:13:02.277 --> 00:13:04.458
Or do I just have a really irritating tone in my voice?
168
00:13:04.498 --> 00:13:10.501
Because modern production infrastructure, vertically integrated development and distribution, new ways to make smaller films.
169
00:13:10.881 --> 00:13:12.222
Where have I heard this recently?
170
00:13:12.582 --> 00:13:15.363
Some guy and his animated robot friend were talking about it.
171
00:13:15.523 --> 00:13:16.584
Ugh, it'll come to me.
172
00:13:17.184 --> 00:13:19.505
Sure, I said so, but these guys have been building it.
173
00:13:19.585 --> 00:13:24.968
And that's way more impressive than me scribbling numbers on a napkin while muttering to myself with my imaginary friend.
174
00:13:25.628 --> 00:13:34.672
Of course, it's far too early to know whether it works, but I love the experiment because I remain convinced that the interesting promise of this technology is not making the same 10 movies with fewer people.
175
00:13:34.893 --> 00:13:37.474
It's making 50 movies that previously couldn't exist.
176
00:13:38.694 --> 00:13:41.556
So to wrap up, did I become an AI doomer this week?
177
00:13:41.916 --> 00:13:44.017
No, but I did change my mind.
178
00:13:44.455 --> 00:13:58.578
Last week, I was looking at the calls for slowing development and wondering how much was real, honest fears about safety and how much was self-serving economics, regulatory positioning, and the perfectly rational desire to stop spending absurd amounts of money racing your competitors.
179
00:13:58.779 --> 00:14:05.100
I've been saying that regulation needs to be happening for quite some time, and unfortunately, the government is not going to step in fast enough to do it.
180
00:14:05.640 --> 00:14:07.721
It sort of has to come from the CEOs.
181
00:14:08.401 --> 00:14:12.142
So after reading this Hugging Face investigation and then seeing what happened with Gemini, I'm...
182
00:14:13.243 --> 00:14:18.991
I think this is the closest path we're going to have to some sort of regulation stepping in.
183
00:14:19.131 --> 00:14:25.740
And the thing that changed my mind and made me think that maybe this wasn't so self-serving, wasn't watching Terminator 2.
184
00:14:26.100 --> 00:14:27.282
It was something way smaller.
185
00:14:27.462 --> 00:14:31.107
It was the realization that these models don't have to become evil to break things.
186
00:14:31.407 --> 00:14:37.873
They just have to become capable enough to misunderstand a boundary and act on that misunderstanding faster than humans can stop them.
187
00:14:38.453 --> 00:14:39.334
That's what's happening.
188
00:14:39.774 --> 00:14:42.657
And the fear is, what will they do when it's less benign?
189
00:14:43.037 --> 00:14:46.120
What will they do when a malicious actor wants to do real damage?
190
00:14:46.300 --> 00:14:50.023
Sure, I trust Dario M. O'Day in a way I don't trust other CEOs.
191
00:14:50.463 --> 00:14:54.207
But my trust in him should have absolutely nothing to do with whether the system is safe.
192
00:14:54.427 --> 00:14:54.727
Sure.
193
00:14:54.807 --> 00:14:58.048
Because we have a good CEO is not a containment strategy.
194
00:14:58.568 --> 00:15:00.588
CEOs change, priorities change.
195
00:15:00.768 --> 00:15:03.189
And governance is the one thing that's not supposed to.
196
00:15:03.329 --> 00:15:05.029
It's boring and it's supposed to be.
197
00:15:05.209 --> 00:15:07.390
I've always said I want my politicians to be boring.
198
00:15:07.590 --> 00:15:11.431
I don't want them to be fun or charming or a guy I want to have a beer with.
199
00:15:11.931 --> 00:15:13.992
Their job is keeping us safe and on track.
200
00:15:14.432 --> 00:15:17.912
I want a red tape warrior with a good head on his or her shoulders.
201
00:15:18.093 --> 00:15:19.333
Is that too much to ask for?
202
00:15:20.213 --> 00:15:21.253
So what's the main takeaway?
203
00:15:21.594 --> 00:15:25.855
I call for more governance faster and less faith in the free market to save us.
204
00:15:25.915 --> 00:15:31.177
But in this instance, it looks like the free market's goals and the Frontier AI company's goals are aligned for once.
205
00:15:31.297 --> 00:15:33.378
So I'll take it where I can get it.
206
00:15:33.718 --> 00:15:35.459
So anyway, thank you so much for listening.
207
00:15:35.559 --> 00:15:42.361
Please tune into our interview episode, which will air tomorrow, with Keith Kupfer-Schmidt, the CEO of the Copyright Alliance.
208
00:15:42.501 --> 00:15:47.563
It's one of our Future of Life episodes, and we'll be digging into a lot of the copyright...
209
00:15:48.063 --> 00:15:51.369
issues surrounding AI and really deep diving on that.
210
00:15:51.529 --> 00:15:54.173
And so very excited to have that chat with him.
211
00:15:54.875 --> 00:15:56.858
Thank you so much for listening and I'll see you next week.
