SELLEST OSAST
Why are people at AI companies worried about what they're building? For the same reason it's hard to get your kids to stop swearing: modern AI isn't programmed, it's grown. Nate Nichols, who has spent 20 years building AI, explains what that means (with help from Mario, a lot of goblins, and nuclear power plants), why it makes AI so hard to control, and one two-minute thing you can do about it: https://kitchentable.show/act?utm_campaign=ep03-pod
Every claim in this episode is sourced. Full show notes with links below.
Opinions are Nate's own and do not represent his employer.
Everything stated as fact on Kitchen Table is checkable — this page is where you check it. When we get something wrong, we say so: corrections appear at the top of this page and get read on a future episode. Opinions and forecasts are labeled as opinions and forecasts.
Recorded October 14, 2026. AI moves fast; figures below are as of recording unless noted. Time words in the episode ("last year," "in September") are relative to that date.
CORRECTIONS
None yet.
SOURCES, IN EPISODE ORDER
Kids, swearing, and the people building AI
• The kids-and-swearing comparison is Nate's analogy, not a finding.
• People at AI companies who worry about catastrophe. In September 2026, Anthropic's alignment-science lead Evan Hubinger said he personally puts the chance of AI causing human extinction within the next decade at over 10%, while saying today's models are low-risk; his concern is future, self-improving systems. He was responding to former OpenAI/Anthropic researcher Jacob Coxon's public resignation: Newsweek (https://newsweek.com/anthropic-researcher-quits-warns-ai-could-kill-everyone-12418798), Fox LA (https://www.foxla.com/news/anthropic-researcher-ai-10-percent-chance-kill-humans).
• "Grown, not built." Anthropic CEO Dario Amodei, "The Urgency of Interpretability" (https://www.darioamodei.com/post/the-urgency-of-interpretability) (April 2025), describes AI systems as grown more than built, and argues that's why they're hard to understand and control.
Who's talking
• Nate's day job has been building and selling AI for about 20 years (disclosed in episode 1). Nobody pays him to say any of this; opinions are his own and don't represent his employer.
How normal software gets made
• The Mario "10% chance" line is a joke. The closest real number from someone at an AI company is Hubinger's, above.
Fifty years of "good old-fashioned AI"
• AI as a field dates to the mid-1950s. The 1955 Dartmouth proposal coined "artificial intelligence"; the workshop ran in summer 1956: Wikipedia — Dartmouth workshop (https://en.wikipedia.org/wiki/Dartmouth_workshop).
• Rules-and-facts AI, a.k.a. expert systems. The term "good old-fashioned AI" was coined by philosopher John Haugeland in 1985: Wikipedia — GOFAI (https://en.wikipedia.org/wiki/GOFAI). Expert systems could show which rules produced an answer — e.g. Stanford's MYCIN, which diagnosed blood infections in the 1970s and was never used on real patients: Wikipedia — MYCIN (https://en.wikipedia.org/wiki/Mycin). A few found commercial use, like DEC's XCON for configuring computer orders: Wikipedia — XCON (https://en.wikipedia.org/wiki/Xcon).
• "None of it worked very well" about language (2005–2020) is Nate's own experience from his work in the field.
The internet, and a breakthrough at Google
• The Transformer (2017). Eight Google researchers published the design behind today's chatbots in June 2017: Vaswani et al., "Attention Is All You Need" (arXiv) (https://arxiv.org/abs/1706.03762), Google Research blog (https://research.google/blog/transformer-a-novel-neural-network-architecture-for-language-understanding/).
• What they're trained on. A huge, filtered chunk of the public web plus books: GPT-3's training data, for example, was mostly filtered web pages from Common Crawl, plus books and Wikipedia: Brown et al., "Language Models are Few-Shot Learners" (2020) (https://arxiv.org/abs/2005.14165). In a 2025 court ruling, a federal judge described Anthropic buying and scanning millions of print books (and downloading millions of pirated ones) to train Claude: The Register (https://www.theregister.com/2025/06/24/anthropic_book_llm_training_ok/), Goodwin (https://www.goodwinlaw.com/en/insights/publications/2025/06/alerts-practices-aiml-district-court-issues-ai-fair-use-decision). Many newer models also learn from images, audio and video, e.g. Google's Gemini (2023) (https://arxiv.org/abs/2312.11805).
Inside the room: pre-training and reinforcement learning
• Feedback-based training ("reinforcement learning"). OpenAI — Aligning language models to follow instructions (https://openai.com/index/instruction-following/); much of the grading is done by other AI systems, e.g. Anthropic — Constitutional AI: Harmlessness from AI Feedback (https://www.anthropic.com/research/constitutional-ai-harmlessness-from-ai-feedback).
• The lawyer friend paid to grade AI answers is Nate's anecdote.
• What these models can be misused for (fraud, hacking, and more): Anthropic publishes regular threat intelligence reports (https://www.anthropic.com/threat-intelligence); see also Anthropic — Disrupting an AI-orchestrated cyber espionage campaign (Nov 2025) (https://www.anthropic.com/news/disrupting-AI-espionage).
Advantage one, gone: accuracy and consistency
• "Claude is accurate like human experts are accurate" is Nate's characterization.
• Asking an opinion question twice is something you can test yourself; the weekend-getaway answers are from Nate's own try.
Advantage two, gone: transparency
• Expert systems could show their work; today's AI mostly can't. Even the companies building these models have only a limited ability to see why they do what they do, and researchers from OpenAI, Google DeepMind and Anthropic have warned that one of the best current windows (reading a model's step-by-step reasoning) is fragile: Korbak et al., "Chain of Thought Monitorability: A New and Fragile Opportunity" (July 2025) (https://arxiv.org/abs/2507.11473); see also Dario Amodei, "The Urgency of Interpretability" (April 2025) (https://www.darioamodei.com/post/the-urgency-of-interpretability). More in a future episode.
Advantage three, gone: control
• Companies can't rule out bioweapons help. Anthropic switched on extra protections for Claude Opus 4 because it couldn't rule out meaningful help to someone trying to build biological weapons: Anthropic — Activating ASL-3 protections (May 2025) (https://www.anthropic.com/news/activating-asl3-protections).
• Safety filters are AI too. Models' own training is backed up by separate classifier systems: same link as above.
• The Yemen case. On September 10, 2026, Anthropic reported that a cell in northern Yemen used Claude in place of software engineers to write guidance software for rockets and missiles; a guided rocket was test-fired, the test appeared to fail, and the group returned to Claude to troubleshoot. Anthropic found no evidence of an operational weapon, and banned the accounts once detected. Anthropic did not name the group; press reports linked it to the Houthis: Anthropic — Detecting and countering misuse of AI: September 2026 (https://www.anthropic.com/threat-intelligence-report-september-2026), Boston Globe (https://www.bostonglobe.com/2026/09/11/business/ai-anthropic-yemen-houthi/), Times of Israel (https://www.timesofisrael.com/anthropic-says-it-foiled-yemini-effort-to-use-its-ai-chatbot-to-develop-guided-missiles/).
• ChatGPT and the goblins. OpenAI traced ChatGPT's goblin habit to an automated grading system (a reward model for its "Nerdy" personality) that scored goblin-mentioning answers higher. OpenAI retired that personality, removed the grader's goblin preference from future training, and patched the already-released model with a line in its instructions b...
