Extinction Is the Wrong Test

Last week, Evan Hubinger, who leads alignment science at Anthropic, said he believes artificial intelligence has a greater than 10 percent chance of killing every human within the next decade. His position makes the warning impossible to ignore: he works on the safety of Anthropic’s AI systems. His chief concern is that a future, more capable AI might escape human control.

Concern about that possibility has also reached Congress. Senator Bernie Sanders and Representative Greg Casar have announced a proposal to ban artificial superintelligence and pause advanced AI development while safety rules are established. A proposal of that scale calls for a precise account of the disaster it aims to prevent and how the proposed measures would help.

What would it take for AI to kill literally every human? A catastrophe would have to reach across the world, defeat attempts to contain it, and leave no surviving population. Conceivable routes include a biological disaster, nuclear war, or a prolonged collapse of systems essential to survival. AI might contribute through the actions of a person using it for harm, or through a future system acting beyond human control. Each possibility has a different sequence of events.

The first begins with a person. Major AI systems people use online, such as Claude and ChatGPT, are designed to refuse requests for help with cyberattacks or biological weapons. Those safeguards make misuse harder, though they can be evaded. The 2026 International AI Safety Report, chaired by Yoshua Bengio and written by an international group of experts, finds that safeguards are improving but remain imperfect. Providers can also monitor for misuse, giving a determined actor reason to seek another way to use AI.

Open weight models raise a deeper concern because people can download and run them privately. Someone can modify a model, remove its safeguards, and use it beyond a provider’s monitoring. Once released, its weights cannot be recalled. Even then, AI advice does not give someone the materials for a biological weapon or control over nuclear weapons. The question is how much the technology lowers the remaining barriers for a determined actor, and whether an attack could spread so far that no one survived it.

The second possibility is closer to Hubinger’s warning. A future AI might pursue a goal its creators did not intend, conceal what it is doing, and resist attempts to stop it. It would also need to carry out plans over time and gain access to resources through which it could cause harm. The International AI Safety Report says current systems show early signs of some relevant abilities, but cannot yet combine them at the level required to escape human control. Experts disagree about whether future systems will develop those abilities.

The warning asks readers to imagine AI killing literally every human. An asteroid impact wiped out about 75 percent of Earth’s species 66 million years ago. It is a useful measure of what a truly global catastrophe looks like: not a war or a famine confined to a region, but a shock whose effects reached across the planet. Any AI disaster serious enough to end humanity would have to operate on a similar scale.

The evidence so far leaves several steps between present AI capabilities and that outcome. A person would need far more than advice from a model. An AI acting on its own would need capabilities and access that current systems do not have. In either case, a catastrophe would then have to overcome human efforts to contain it and leave no survivors. That sequence makes extinction within a decade appear very unlikely, though the available evidence cannot establish a precise probability or disprove Hubinger’s estimate.

A disaster that killed millions would still demand action, of course. Defining the threat helps identify which action might work. The Sanders and Casar proposal chiefly addresses the prospect of a future AI escaping control. A pause in development could give researchers and regulators time to improve safeguards. A ban on superintelligence would do less to prevent a criminal from using an open weight model already available. Those are reasons to examine the proposal against each danger, along with the benefits it might preserve or sacrifice.

There are grave outcomes that require fewer steps. AI might help a criminal find a weakness in a bank’s systems, deceive an employee, or carry out a breach. Stolen money, altered records, and frozen accounts could disrupt millions of lives. An attack on a power utility could interrupt electricity and the services that depend on it. The International AI Safety Report describes ways AI can assist cyberattacks, though its contribution to actual attacks remains difficult to measure. These possibilities justify preparation on their own.

How lasting could the damage be? The twentieth century offers a difficult comparison. Two world wars killed tens of millions, destroyed cities and industry, and were followed by years of rebuilding. People retained much of the knowledge that had raised living standards before the wars. They repaired infrastructure, developed new tools, and resumed producing more. That history shows a capacity to recover from severe disruption. For those who lost their lives or livelihoods, the fact that prosperity returned offers little comfort.

Prosperity grows through accumulation and resilience. People preserve useful knowledge, improve tools, and pass both to the next generation. The gains arrive gradually, then become part of what we expect from daily life. A frightening prediction arrives all at once and demands an immediate response. Future defenses are harder to picture because some have yet to be developed, tested, or put in place.

The inevitable next alarming headline deserves a set of questions. What sequence of events does it predict? What evidence supports each step? What harm could occur even if the worst prediction never comes true? And which response would reduce that harm? Asking those questions takes the warning seriously and gives the public a way to judge what should be done.

Thank you very much to our sponsors:

* Foundation.xyz: When it comes to protecting your Bitcoin, you shouldn’t have to simply trust the company that made your hardware wallet.

Foundation believes in verifiability over reputation. Its firmware and apps are open source, its builds are reproducible, and both Passport Core and Passport Prime have undergone independent hardware security audits. The point isn’t simply that Foundation says its products are secure. It’s that you can verify that for yourself.

Foundation is also expanding what users can control:

* Passport Prime goes beyond Bitcoin. It can manage passwords and two-factor authentication codes, serve as a FIDO2 security key, and provide encrypted storage—all on a device you own and control. Foundation also plans to enable app sideloading, opening Prime to applications from independent developers.

* Users can now add their own entropy. The latest Envoy and KeyOS releases allow users to introduce their own randomness when creating a seed, reducing reliance on the device itself to generate it.

Order a new Passport Prime and receive a free bumper case using Code PROSPERITY at → foundation.xyz

* River Financial: If not cold storage for your Bitcoin, I’d highly recommend River for buying and holding- River is engineered to protect your Bitcoin over the long run. Use this link for discounts: https://river.com/signup?r=T7GGAF7G