このエピソードについて
Written as part of the MATS 9.1 extension program, mentored by Richard Ngo. Additional thanks to Andrew Wu, Maria Kostylew, and Lennie Wells for helpful draft feedback and editing.
This post reflects on the tortured distinction between "safety" and "capabilities" in AI research.
Richard Ngo has written about why the alignment vs. capabilities ontology is conceptually fraught, and is currently arguing that key strategic decision-makers in and around "AI safety" have brought about the AI labs' stampede towards Artificial Superintelligence (ASI). This post instead looks at the following problem: how does one conduct alignment research without contributing to capabilities? It proposes decisions an individual or a small research group can take to do good work in AI.
At the end, I discuss possible objections: namely, that my proposals fail to 'maximise impact'. I lay out why this meme is poisonous and usually backfires, and conclude by rejecting it entirely.
Two examples of failure
My first claim is that 'safety' and 'research' are two concepts that are in routine tension with one another. I illustrate this through examples of work that did too much of one at the expense of the other.
Example: (mechanistic) interpretability
In limiting its scope [...]
---
Outline:
(01:12) Two examples of failure
(01:27) Example: (mechanistic) interpretability
(04:30) Example: MIRI and Recursive Self-Improvement
(09:49) The curse of science
(12:11) A note on the AI labs
(15:53) So what do you do?
(17:00) The information you give away
(20:16) The information you let in
(21:46) But what about impact?
(22:42) The virtue of taking things slow
(26:50) Appendix: caveat for policy work
The original text contained 19 footnotes which were omitted from this narration.
---
First published:
September 23rd, 2026
Source:
https://www.lesswrong.com/posts/HekpnSkrt89tMm3Dc/i-am-an-ai-safety-researcher
---
Narrated by TYPE III AUDIO.
This post reflects on the tortured distinction between "safety" and "capabilities" in AI research.
Richard Ngo has written about why the alignment vs. capabilities ontology is conceptually fraught, and is currently arguing that key strategic decision-makers in and around "AI safety" have brought about the AI labs' stampede towards Artificial Superintelligence (ASI). This post instead looks at the following problem: how does one conduct alignment research without contributing to capabilities? It proposes decisions an individual or a small research group can take to do good work in AI.
At the end, I discuss possible objections: namely, that my proposals fail to 'maximise impact'. I lay out why this meme is poisonous and usually backfires, and conclude by rejecting it entirely.
Two examples of failure
My first claim is that 'safety' and 'research' are two concepts that are in routine tension with one another. I illustrate this through examples of work that did too much of one at the expense of the other.
Example: (mechanistic) interpretability
In limiting its scope [...]
---
Outline:
(01:12) Two examples of failure
(01:27) Example: (mechanistic) interpretability
(04:30) Example: MIRI and Recursive Self-Improvement
(09:49) The curse of science
(12:11) A note on the AI labs
(15:53) So what do you do?
(17:00) The information you give away
(20:16) The information you let in
(21:46) But what about impact?
(22:42) The virtue of taking things slow
(26:50) Appendix: caveat for policy work
The original text contained 19 footnotes which were omitted from this narration.
---
First published:
September 23rd, 2026
Source:
https://www.lesswrong.com/posts/HekpnSkrt89tMm3Dc/i-am-an-ai-safety-researcher
---
Narrated by TYPE III AUDIO.
英語
アメリカ合衆国
文字起こし 🔗
Are you the producer of this podcast?
Add a podcast transcript
Need Audio-to-Text?
Transcribe with Listen411 in Just 60 Seconds
このポッドキャストの他のエピソード
By Alex Turner, Amber Scorah, and Karl Koch. Op-ed written for Fortune. Giving independent evaluators “employee-like access” is “a great idea”, OpenAI CEO Sam Altman said on September 12th. Two weeks later, OpenAI president Greg Brockman signed the White House Accord on Super Intelligence which out…
Good empirical safety research is heavily bottlenecked on thinking more clearly. This post shares a few frames and tips I found useful, illustrated with concrete examples. I spent the better part of 2026 in empirical AI safety research apprenticeships. I was mentored by Alex Turner and Alex Cloud a…
Tl;dr: The frontier AI companies’ implicit alignment plans center on rushing towards RSI with something like current models. This plan is (at best) contentious, and many believe it will have catastrophic consequences. Nonetheless, it is the track we are on. It is therefore important to collect evid…
TL;DR: This document provides an overview of publicly available evidence regarding the monitoring practices of internally deployed agents within frontier AI companies (focusing on OAI, Anthropic, and GDM). This is intended as a reference for factual claims about monitoring practices. In general, th…
Robin Hanson didn't invent the idea of a martingale process. The lilim say that Ville in 1939 gave the term its mathematical meaning and Doob turned it into the modern theory. But to state this is not to realize that your own probability assignments over time ought to form a martingale in your…
免責: このページに埋め込まれているポッドキャストとアートワークは LessWrong からのものであり、その所有者の所有物であり、Listen Notes, Inc. と提携または承認されていません。
