ABOUT THIS EPISODE
Does AI Make Developers More Productive? The Evidence
AI can help developers finish tasks in some settings. This episode compares task time, completed pull requests, survey responses and delivery outcomes, then follows our own agent-run video pipeline from a checked script toward release.
WHAT WE FOUND:
METR's July 10, 2025 experiment covered 16 experienced developers and 246 real issues using early-2025 tools. AI-allowed tasks took 19% longer; submitted pull requests had similar quality. METR explicitly says that historical finding is out of date.
Its February 24, 2026 update covered 57 developers, 143 repositories and over 800 tasks in an experiment begun in August 2025. Estimated task-time changes were −18% for returning developers and −4% for new participants. Both confidence intervals included no change and some slowdown. METR calls the signal unreliable because task selection, participation and concurrent agents complicated measurement.
Three randomized company experiments pooled 4,867 developers. Their preferred estimate was 26.08% more completed tasks among tool users, measured as completed pull requests per week. This accounts for incomplete tool uptake; offering everyone access is a separate question. The trials began in 2022 and 2023 with GitHub Copilot code completion, with journal publication in February 2026.
Stack Overflow's 2025 survey recorded almost-right AI answers as a frustration for 66% of respondents to that question. A separate question found 52% agreeing AI tools or agents helped productivity. These are self-reports.
Faros's June 2025 observational analysis associated high adoption with 21% more completed tasks, 98% more merged pull requests and 91% longer review time. DORA's March 10, 2026 analysis also associated adoption with more delivery throughput and greater instability, drawing on 2025 observations. Neither establishes causation. Echoes of AI, revised February 26, 2026, detected no significant difference in later code-change time or quality in its randomized second phase; longer-term maintenance remains open.
Our November 14, 2026 snapshot covered nine earlier production runs. All saved final script checks said PASS; five runs were marked released. Across 25 fact-check rounds, the checker recorded 50 required fixes, including wording and attribution. In the five runs with call logs, checking calls outnumbered writing calls, 64 to 43. These are broad production categories in a small media workflow with no human-only comparison.
My opinion: define acceptance criteria before generation, inspect the evidence attached to a change, and track active human time, elapsed time, review waits, rework and the behavior delivered to the user.
In your own work with AI, which step now takes the longest between asking for a change and getting something usable?
Full write-up: https://www.500xengineer.com/blog/ai-coding-evidence
Join the free Skool community: https://www.skool.com/500xengineer
SOURCES: the full source list is in the YouTube description: https://www.youtube.com/watch?v=FKJJ16fNP4w
Question that prompted the review: The Students podcast episode, November 13, 2026, “Jonathan Blow — If AI Solved Coding, Why Is My Software Still the Same?”
https://www.youtube.com/watch?v=IRK8h67ol7Q
Our production figures come from saved aggregate run and fact-check records; they describe a media workflow and cannot establish AI's effect on software-development speed.
DISCLOSURE:
Study dates, tools, outcomes and limitations differ. Our pipeline record is a small descriptive example without a human-only baseline. Narration uses an AI clone of my voice; the short opening clip is an AI-generated avatar. Opinions are mine.