ABOUT THIS EPISODE
In the movie about the bin Laden raid, an analyst says "one hundred percent he's there." The real room gave Leon Panetta a spread — 90, 60, below 50 — and he had to make the call on a gut. Around that same time, American intelligence quietly ran a four-year tournament to find out whether anyone can actually predict the future. The winners were a pharmacist and a retired programmer with a duck for an avatar.
This episode teaches the method they used, move by move: start from the base rate, break impossible questions into estimable pieces, commit to precise numbers — 63%, not "probably" — update small and often, team up and let math sharpen the crowd's answer, and keep score with Brier scores, the scoreboard the whole field runs on.
Then the part most tellings skip: where it breaks. The curated questions; Taleb's rare-event attack and Tetlock's own concession; the Ukraine and Brexit misses; the AI progress the superforecasters underpriced — priced at 2.3%, resolved yes.
And the 2026 twist, in two industries. A betting-market war: Kalshi reportedly chasing a $40 billion price tag while New York sues and a federal regulator invokes emergency powers to keep it open. And AI systems that reached statistical parity with the superforecaster median — by having the method typed into their prompts: "You are an expert superforecaster… pick the base rate." The superforecasters' own median forecast says AI beats them by 2028. The best calibrators on Earth, calibrated about losing.
You can practice the skill free — Good Judgment Open and Metaculus run public tournaments with real scoreboards. The skill was never seeing the future. It was keeping score on yourself, before reality does it for you.
RELATED EPISODES
Will AI Replace Software Engineers? What the Data Actually Says — the same benchmark-asterisk test this episode runs on AI forecasting parity
How an AI Agent Hacked Hugging Face — the AI-capability arc the superforecasters underpriced
How GPS Actually Works — the sibling mechanism explainer: a system you use daily, taught from the inside
CHAPTERS
00:00 The room that ran on gut calls
01:28 'A fair chance' — what vague words cost
03:35 The tournament the government could lose
06:17 Surgery on the famous 30% claim
08:43 The method, move by move
14:41 Where it breaks: Taleb, Ukraine, and the AI miss
18:18 The ox, the whale, and the $40 billion war
23:44 The machines type in the method
27:31 How you practice this free
SOURCES
Philip Tetlock & Dan Gardner, Superforecasting (2015); Tetlock, Expert Political Judgment (the 1984–2004 study)
Mellers et al., Psychological Science 2014 & Perspectives on Psychological Science 2015 — tournament design, granularity (57 values; rounding test), persistence, 300-vs-60 days
Goldstein et al. (MITRE, declassified) — Good Judgment vs the Intelligence Community Prediction Market: 0.15 vs 0.23 on 139 shared questions; the secrecy heuristic
Friedman et al., International Studies Quarterly 2018 — 888,328 forecasts; the cost of verbal probability labels; the Bay of Pigs 'fair chance' record
Chang et al. 2016 — the one-hour training effect (+6–11%); Hauenstein et al., Psychological Science 2025 — the reanalysis contesting it
Forecasting Research Institute — ForecastBench (forecastbench.org) and the XPT near-term accuracy report (Sept 2025); Good Judgment's rebuttals (July 2026)
Wallis, Statistical Science 2014 — Galton's 1906 ox, corrected from the archives
NY Attorney General release (July 31st, 2026); The Block (Aug 11th, 2026 CFTC emergency order); WSJ 'The Economics Of' & CoinDesk market coverage
On the record: Leon Panetta (Choiceology); Warren Hatch (Cassandra Forum, Aug 2026); Luana Lopes Lara (WSJ Events, May 2026); Tetlock (80,000 Hours 2019; Harvard Kennedy School, June 2026)
