Can we control self-improving A.I.?

 What happens when A.I. starts improving itself faster than humans can keep up? The researchers building these models can't fully explain how they work. If those models begin designing their successors, the things we don't understand could get baked into each new generation. How do we keep control?

In part two of his series on recursive self-improvement, David Deming focuses on alignment, the challenge of getting A.I. to do what we want. Human morality can't be reduced to a set of rules, and getting a model to behave well today doesn't guarantee it will ehave well as it becomes more capable.

He then turns to control: limiting what models can do, keeping the ability to shut them down, and building safeguards that assume failures will happen. From factory safety to A.I. agents that find ways around their instructions, David asks what it would take to contain the consequences when things go wrong.

He argues that A.I. labs should have to meet much higher safety standards before they let models improve themselves. Powerful machines need engineering safeguards and people who can be held accountable when those safeguards fail.

Part 1 of this series: https://thecontextwindowpodcast.substack.com/p/can-ai-get-better-at-making-itself

For show notes and full episodes, subscribe to The Context Window with David Deming: ⁠https://thecontextwindowpodcast.substack.com