EA - Long Reflection Reading List by Will Aldred

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Long Reflection Reading List, published by Will Aldred on March 25, 2024 on The Effective Altruism Forum.
This is a reading list on the following cluster of notions: "the long reflection", "the deployment problem", "structural risk", "post AGI governance", "ASI governance", "reflective governance", "metaphilosophy", "AI philosophical competence", "trajectory change", "macrostrategy", "worldview investigations", "grand challenges" and "the political philosophy of AI".
I claim that this area outscores regular AI safety on importance[1] while being significantly more neglected (and roughly the same in terms of tractability), making it perhaps the highest priority EA cause area.
I don't claim to be the ideal person to have made this reading list. The story behind how it came about is that two months ago, Will MacAskill wrote: "I think there's a lot of excitement about work in this broad area that isn't yet being represented in places like the Forum. I'd be keen for more people to start learning about and thinking about these issues." Intrigued, I spent some time trying to learn about the issues he was pointing to.
I then figured I'd channel the spirit of "EAs should post more summaries and collections": this reading list is an attempt to make the path easier for others to follow. Accordingly, it starts at the introductory level, but by the end the reader will be at the frontier of publicly available knowledge. (The frontier at the time of writing, at least.[2])
Note: in some places where I write "the long reflection," I'm using the term as shorthand to refer to the above cluster of notions.
Intro
Quotes about the long reflection - MichaelA (2020)[3]
The Precipice - Ord (2020)
Just chapter 7, including endnotes.
Beyond Maxipok - good reflective governance as a target for action - Cotton-Barratt (2024)
New Frontiers in Effective Altruism - MacAskill (2024)
This was a talk given at EAG Bay Area 2024. It doesn't appear to be available as a recording yet, but I'll add it if and when it goes up.
Quick take on Grand Challenges - MacAskill (2024)
The part about hiring is no longer relevant, but the research projects MacAskill outlines still give a sense for what good future work on grand challenges / the long reflection might look like.
Criticism of the long reflection idea:
'Long Reflection' Is Crazy Bad Idea - Hanson (2021)
Objections: What about "long reflection" and the division of labor? - Vinding (2022)
Just the highlighted section.
A comment by Wei Dai (2019a)
What might we be aiming for?
Is there moral truth? What should we do if not? What are human values, and how do they fit in?
Moral Uncertainty and the Path to AI Alignment with William MacAskill - AI Alignment Podcast by the Future of Life Institute (2018)
See also Shah (2018)'s summary and commentary.
See also this comment exchange between Michael Aird and Lukas Gloor (2020), which zooms in on the realism vs. antirealism wager and how it relates to the long reflection.
Complexity of value - LessWrong Wiki
Moral ~realism - Cotton-Barratt (2024)
Why should ethical anti-realists do ethics? - Carlsmith (2023)
Coherent extrapolated volition - Arbital
How to think about utopia?
Hedonium and computronium - EA Forum Wiki
Terms that tend to come up in discussions of utopia.
Why Describing Utopia Goes Badly - Karnofsky (2021)
Visualizing Utopia - Karnofsky (2021)
Characterising utopia - Ngo (2020)
Actually possible: thoughts on Utopia - Carlsmith (2021)
Deep Utopia - Bostrom (2024)
(If and when someone writes a summary of this book I'll add it to this reading list.)
Ideally, I would include at this point some readings on how aggregation might work for building a utopia, since this seems like an obvious and important point.
For instance, should the light cone be divided such that every person (or every moral patient more broad...