TENTANG EPISODE INI
Could slowing AI development make superintelligence safer? Daniel Kokotajlo and Thomas Larsen of the AI Futures Project join Tim Scarfe to examine AI 2040: Plan A, a proposal to buy time before AI exceeds human control.
SPONSOR:
---
Cyber Fund built the Monastery to help founders ship products that were impossible a year ago.
Apply now: https://cyber.fund
---
After revisiting AI 2027 and the limits of forecasting, they ask what happens when AI can automate research and sustain an economy without human workers. Tim challenges the case for general models and asks whether intelligence alone explains power. Plan A proposes an initial pause to build safety infrastructure, then cautious development up to the strongest AI that can still be reliably controlled. The discussion tests the distinction between control and alignment, the case for public AI research, and whether the US and China could enforce a slowdown. It ends with the evidence that would change their forecasts.
---
TIMESTAMPS:
00:00:00 AI 2040: a slower route to superintelligence
00:01:34 Sponsor: Cyber Fund
00:02:12 From OpenAI to AI 2027
00:06:58 Forecasts, war games and self-fulfilling prophecies
00:17:44 Why AI sceptics are changing their minds
00:23:04 When AI can replace its own researchers
00:28:45 Could an AI economy grow without human workers?
00:37:32 One general model or a society of specialists?
00:47:43 Brains, machines and collective intelligence
00:56:12 Plan A: buy time at the controllable frontier
01:00:02 Why control buys time but cannot replace alignment
01:06:36 Why AI research should be public
01:10:32 Can the US and China enforce an AI slowdown?
01:19:04 Why AI policy debates miss the technology
01:21:56 Is AI normal technology? The remaining disagreement
Many thanks to James Wilken-Smith for helping with show research.
---
REFERENCES:
other:
[00:00:01] AI 2040: Plan A
https://ai-2040.com/
[00:03:27] AI 2027
https://ai-2027.com/
[00:13:47] Scenario Scrutiny for AI Policy
https://blog.aifutures.org/p/scenario-scrutiny-for-ai-policy
[00:33:11] The 2028 Global Intelligence Crisis
https://www.citriniresearch.com/p/2028gic
[01:00:40] Brief independent investigation of agents' behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident
https://www.redwoodresearch.org/research/hugging-face-incident
[01:09:21] The Hugging Face incident and the road ahead
https://openai.com/index/hugging-face-incident-and-the-road-ahead/
[01:22:01] AI as Normal Technology
https://www.normaltech.ai/p/ai-as-normal-technology
[01:22:51] Common Ground between AI 2027 & AI as Normal Technology
https://asteriskmag.substack.com/p/common-ground-between-ai-2027-and
person:
[00:19:43] Geoffrey Hinton
https://www.cs.toronto.edu/~hinton/
[00:20:07] Ryan Greenblatt
https://www.lesswrong.com/users/ryan_greenblatt
[00:26:06] Elon Musk
https://www.tesla.com/elon-musk
tool:
[00:21:46] ARC-AGI-3
https://arcprize.org/arc-agi/3
[00:21:53] AlphaGo and Move 37
https://deepmind.google/research/alphago/
[00:39:41] Claude
https://claude.com/product/overview
[00:39:58] NVIDIA H100 GPU
https://www.nvidia.com/en-us/data-center/h100/
paper:
[00:24:42] Training AI Scientists to Replicate Research
https://arxiv.org/abs/2608.13331v1
[01:27:19] Validity of the single processor approach to achieving large scale computing capabilities
https://www.cs.cmu.edu/~18742/papers/Amdahl1967.pdf
book:
[00:28:52] Bullshit Jobs: A Theory
https://www.simonandschuster.com/books/Bullshit-Jobs/David-Graeber/9781501143335
organization:
[01:05:09] Redwood Research
https://www.redwoodresearch.org/
---
RESCRIPT:
https://app.rescript.info/public/share/33d1a58fa8f307ae7dfd504d4fdaa9d5
