ABOUT THIS EPISODE
Alignment is not on track
Artificial superintelligence (ASI) may be developed in the next few years. It is unclear whether alignment is on track to be ready on the same timeframe. At a minimum, the empirical programs at AI labs are unlikely to deliver a priori confidence, before training ASI, that things will go well. We are starting a large nonprofit research organization, Sequent, that aims to clear a higher bar:
---
Outline:
(00:21) Alignment is not on track
(02:40) Aiming at higher confidence
(05:30) Why a new big organization
(07:35) Different lines of research will interact
(11:35) Amortizing security and funding
(12:47) Automated alignment is possible, if not necessarily in time
(17:39) Federated structure to preserve research diversity
(18:38) Field building and broader alignment scale-up
(21:07) Independence is important
(22:40) Join us!
The original text contained 1 footnote which was omitted from this narration.
---
First published:
June 10th, 2026
Source:
https://www.lesswrong.com/posts/AP7YDke5jjY4v3X9Z/sequent-scale-and-automation-for-higher-confidence-in-1
---
Narrated by TYPE III AUDIO.
---
Artificial superintelligence (ASI) may be developed in the next few years. It is unclear whether alignment is on track to be ready on the same timeframe. At a minimum, the empirical programs at AI labs are unlikely to deliver a priori confidence, before training ASI, that things will go well. We are starting a large nonprofit research organization, Sequent, that aims to clear a higher bar:
- We are aiming at higher confidence via a portfolio of theory and empirics bets, all of which could fail, such that if any succeed, they would give us more a priori confidence in aligned outcomes.
- We are investing heavily in automation to accelerate progress on these bets.
- We believe that theory unlocks higher automation. Taking a more principled approach offers better filters for deciding which directions of automated research are promising (a proof is worth a thousand experiments, and even a pseudo-proof is worth hundreds).
---
Outline:
(00:21) Alignment is not on track
(02:40) Aiming at higher confidence
(05:30) Why a new big organization
(07:35) Different lines of research will interact
(11:35) Amortizing security and funding
(12:47) Automated alignment is possible, if not necessarily in time
(17:39) Federated structure to preserve research diversity
(18:38) Field building and broader alignment scale-up
(21:07) Independence is important
(22:40) Join us!
The original text contained 1 footnote which was omitted from this narration.
---
First published:
June 10th, 2026
Source:
https://www.lesswrong.com/posts/AP7YDke5jjY4v3X9Z/sequent-scale-and-automation-for-higher-confidence-in-1
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
English
United States
TRANSCRIPT 🔗
Are you the producer of this podcast?
Add a podcast transcript
Need Audio-to-Text?
Transcribe with Listen411 in Just 60 Seconds
SEARCH PAST EPISODES
Search past episodes of LessWrong (Curated & Popular).
OTHER EPISODES IN THIS PODCAST
In February 2020, LessWrong hosted Coronavirus: Justified Practical Advice Thread . It was a gathering place for people to share and debate small pieces of the overall survival policy and let the wisdom of crowds lead to better outcomes than anyone could get alone. In that spirit, I notice The Gre…
Audio note: this post contains a lot of mathematical notation. We narrate the simpler formulas and flag the complex ones, so you may want the original text to hand. Heavily inspired by a conversation with Keshav Srinivasan and Richard Ngo during MATS 10.0. The week ahead of you is packed. This is…
Summary
Timelines to superintelligence without any slowdown interventions look very short now, perhaps with a median in early 2028. AI will likely dramatically automate and accelerate AI research in 2027.
This means that in a substantial share of worlds where the leading AGI company doesn’t solv…
Science is caught between Law and Chaos. On the side of Law:
Things should make explicit sense to your deliberate reasoning ("system-two"). If you just have a "vibe" about how things work, that's bad. Measure everything. Use numbers whenever possible. You want things to b…
How do you get a friend to leave an abusive relationship? If you talk to people who work at women's shelters, they will tell you, but there is something resigned about the way they do it: they know that it's too difficult an ask. Those who can do it reliably are often people who are unluc…
Disclaimer: The podcast and artwork embedded on this page are from LessWrong, which is the property of its owner and not affiliated with or endorsed by Listen Notes, Inc.
EDIT
Thank you for helping to keep the podcast database up to date.