TRANSKRIPT 🔗
WEBVTT
1
00:00:00.000 --> 00:00:10.004
My review queue has never been this full of code nobody wrote, and this year an OpenAI model overturned a conjecture Paul Erdős made in 1946. Nine mathematicians published a human-verified version.
2
00:00:10.004 --> 00:00:17.195
And in October the same company posted seven hundred and twenty-two machine-written papers, then pulled three the next day over a sign error.
3
00:00:17.195 --> 00:00:23.448
Same company, two very different results. The difference is the referee, and I'm about to bet my roadmap on it.
4
00:00:23.448 --> 00:00:26.887
And I'm about to tell you why that's a bad bet.
5
00:00:26.887 --> 00:00:35.641
Before we start, my week. Digiday ran a piece on agencies finding that AI overuse is leading to what they call brief bloat. Briefs getting longer, not better.
6
00:00:35.641 --> 00:00:37.517
Longer, not better. For a brief.
7
00:00:37.517 --> 00:00:50.022
And I send briefs to two agencies, so I read that and thought, am I the problem? I went back through last month's, and one had a section called "strategic north star" that I did not write and cannot explain.
8
00:00:50.022 --> 00:00:54.086
Nobody can explain a north star. That's what makes it a north star.
9
00:00:54.086 --> 00:01:00.652
Who reads page forty? Who checks it? Nothing fails when a brief is bad. It just quietly costs us a month.
10
00:01:00.652 --> 00:01:05.029
Which is the whole episode, and you handed it to me before the welcome.
11
00:01:05.029 --> 00:01:22.224
Welcome to Dan's AI Intel, where we try to get past the hype and the fear, and ask what the AI revolution means for the economy, for politics and for the race between the labs. I'm Alex, I run engineering at an activewear brand, and nobody has ever explained a north star to me either.
12
00:01:22.224 --> 00:01:31.603
I'm Sam, I run brand and lifecycle marketing at an online pet retailer, and I've now been told I wrote my own north star. Follow the show wherever you listen.
13
00:01:31.603 --> 00:01:50.674
Today's question is why the strangest AI results of the last few years keep landing in maths. On the map: which fields AI has really transformed, the climb from guessing formulas to overturning a famous conjecture, the moment the bottleneck flipped to checking, why a referee turns guessing into learning, and how much of the maths lead is just a scoreboard.
14
00:01:50.674 --> 00:02:06.306
My honest starting position is that a lot of this is scoreboard. I've been burned by a vendor deck that only reported the wins, and maths is the one field where you can count wins on a public list. Wake me when it ships in something my team can buy.
15
00:02:06.306 --> 00:02:16.936
Mine is the opposite. I think the lead is real, the reason is a referee, and once you see the reason you can predict which fields fall next. I'm planning my roadmap around it.
16
00:02:16.936 --> 00:02:19.437
Your roadmap, built on a theory about Erdős.
17
00:02:19.437 --> 00:02:21.000
On a theory about checkers.
18
00:02:21.000 --> 00:02:25.240
Then start with my objection. Whose checker, and who checked the checker?
19
00:02:25.240 --> 00:02:30.540
Start with the title. "The secret is the referee." Slogan, or does it mean something?
20
00:02:30.540 --> 00:02:50.327
It means something, and it's personal for me. Every pitch I sit through says AI will speed up science, and maths is the first place that visibly happened. If the cause is verifiability, I can tell my board which of our jobs are next and which wait, and that's a budget conversation, not a science one.
21
00:02:50.327 --> 00:02:55.980
But hold on. Isn't the biggest AI science result a protein thing? Not maths at all.
22
00:02:55.980 --> 00:03:06.580
AlphaFold. Demis Hassabis and John Jumper shared half the Nobel Prize in Chemistry for it. So maths leads on volume, but AlphaFold is the single most consequential result. Both true.
23
00:03:06.580 --> 00:03:11.527
So why does one field get a Nobel and the other gets a list?
24
00:03:11.527 --> 00:03:23.893
AlphaFold was graded in a blind contest against structures labs had solved but not yet published, and it scored above ninety, which is competitive with experiment. A prediction could be checked against a known answer.
25
00:03:23.893 --> 00:03:24.600
A referee.
26
00:03:24.600 --> 00:03:40.853
A referee, plus decades of banked answers to learn from. Weather is the same. Google DeepMind's GraphCast beat the leading conventional forecast on most test targets and makes a full forecast in under a minute on one machine. And the atmosphere grades every forecast within days.
27
00:03:40.853 --> 00:03:44.033
Weather's the easy one. Show me where it breaks.
28
00:03:44.033 --> 00:03:47.567
Materials. DeepMind's GNoME predicted hundreds of thousands of stable crystals.
29
00:03:47.567 --> 00:03:58.167
I did see that one. And then two materials scientists, Anthony Cheetham and Ram Seshadri, looked at a sample and found nothing strikingly new and no use for the list.
30
00:03:58.167 --> 00:04:09.120
DeepMind stands by it and says hundreds have since been made in labs. But a material has to be made before anyone knows. The checking runs at the speed of chemistry.
31
00:04:09.120 --> 00:04:15.127
This is where I get impatient. A list of predictions sounds like a demo that never shipped.
32
00:04:15.127 --> 00:04:26.433
Then medicine will make you furious. No AI-discovered drug has been approved. Insilico's rentosertib, for lung fibrosis, entered a late-stage trial in July, and the company's own timeline says approval around 2030.
33
00:04:26.433 --> 00:04:34.913
That's a brand campaign whose verdict lands two quarters later, and I've been the person waiting on it. Days to design, years to know.
34
00:04:34.913 --> 00:04:45.867
And no amount of computing shortens the trial. Which is why I'd never let an agent near our returns forecasting. Our checker is next season, and I can't retry a season.
35
00:04:45.867 --> 00:04:56.113
Which gives me a better
Load full transcript
question for the next time a vendor tells me AI will fix everything. I ask what the checker is and how long it takes.
36
00:04:56.113 --> 00:05:00.000
That's a better question than most of the slides I get.
37
00:05:00.000 --> 00:05:04.545
But AlphaFold's checks took a lab months. That breaks your own rule.
38
00:05:04.545 --> 00:05:16.667
It does, and I should have caught it. The answers had been banked for decades before it arrived. So the rule isn't just fast checks. It's abundant, trustworthy answers to learn against.
39
00:05:16.667 --> 00:05:21.591
Fine. I'll take the shape. I'm not taking that it's the whole story.
40
00:05:21.591 --> 00:05:37.879
The maths record is a ladder, and each rung deserves its own grade. Bottom rung, the Ramanujan Machine, from the Technion in Israel with Google, published in Nature. It matched numbers to guess new formulas for constants like pi and e. No proofs.
41
00:05:37.879 --> 00:05:39.015
So it guessed.
42
00:05:39.015 --> 00:05:44.318
Some guesses were later proved, some were easy in hindsight. Modest result, real idea.
43
00:05:44.318 --> 00:05:54.167
Participation trophy. In my world that's the vendor with a clever demo and no way to measure whether it helped. We hand those out every quarter.
44
00:05:54.167 --> 00:06:05.152
Next rung, AlphaTensor. DeepMind treated matrix multiplication as a game and found a way to multiply small binary matrices in fewer steps than the classic method from decades back.
45
00:06:05.152 --> 00:06:07.424
Why do I care about matrices?
46
00:06:07.424 --> 00:06:21.818
They sit underneath almost everything, including the computers that train AI. But it was narrow. The trick only works for binary arithmetic, and within days two mathematicians in Linz beat one of its records with ordinary computer search.
47
00:06:21.818 --> 00:06:28.636
So a machine got outrun by ordinary search in a week. Very on brand for my budget meetings.
48
00:06:28.636 --> 00:06:33.939
The next one is my favourite, and you'll see why in about ten seconds.
49
00:06:33.939 --> 00:06:37.348
I can already see it. Your shoulders just moved.
50
00:06:37.348 --> 00:06:53.258
FunSearch. A language model writes programs, a scorer runs them, the best survive. It found a set of points in eight dimensions where no three line up, and DeepMind called it the biggest jump in that kind of set in twenty years.
51
00:06:53.258 --> 00:06:55.152
There it is. The face.
52
00:06:55.152 --> 00:06:56.288
There's no face.
53
00:06:56.288 --> 00:07:00.455
There's a face. You get faster and your hands start moving.
54
00:07:00.455 --> 00:07:11.439
Fine, the face is because it's elegant. Nobody proved a theorem. The scorer just wouldn't let a bad program through, and that scorer, in DeepMind's words, guards against hallucinations.
55
00:07:11.439 --> 00:07:14.470
A nervous intern with a very strict manager.
56
00:07:14.470 --> 00:07:18.258
I'm using that with my board and not crediting you.
57
00:07:18.258 --> 00:07:30.000
And the
strict manager is the part I'd pay for. I don't want a cleverer intern. I want somebody who reads every page and says no, quietly, before it reaches me.
58
00:07:30.000 --> 00:07:39.470
Except the manager has to say no in seconds. A manager who takes a week is just a slower intern, and I've hired that manager.
59
00:07:39.470 --> 00:07:44.015
So the speed of the no matters as much as the no.
60
00:07:44.015 --> 00:07:47.803
That's the whole thesis, yes. I'll stop saying it now.
61
00:07:47.803 --> 00:07:55.000
Where do the contests come in? I keep hearing gold medal and I can't tell if it's a stunt.
62
00:07:55.000 --> 00:08:08.071
AlphaGeometry solved more than double the olympiad geometry problems the previous best had. And it trained on millions of theorems it generated itself, because a geometry engine could certify each one. No human examples.
63
00:08:08.071 --> 00:08:11.147
It made its own homework and marked it.
64
00:08:11.147 --> 00:08:20.758
Then AlphaProof and a second AlphaGeometry reached the top of silver at the Olympiad, and AlphaProof solved the hardest problem, which only five humans cracked.
65
00:08:20.758 --> 00:08:23.450
Where's the catch? There's always a catch.
66
00:08:23.450 --> 00:08:36.905
Humans translated the problems into a proof language called Lean first, and some took days of computing. I've sat through that pitch from vendors: a landmark, not something my engineers could pick up and use.
67
00:08:36.905 --> 00:08:40.366
A landmark that needs a translator is a demo.
68
00:08:40.366 --> 00:08:42.288
For about a year, yes.
69
00:08:42.288 --> 00:08:53.821
Let me tally the ladder so far. Guessed formulas, a record, a scorer, a contest. Every rung has somebody else doing the grading. Never the model marking its own paper.
70
00:08:53.821 --> 00:08:56.897
Hold that thought for later. It gets weirder.
71
00:08:56.897 --> 00:09:01.895
So a year ago that was a demo with a translator. What changed?
72
00:09:01.895 --> 00:09:24.578
Last year, AlphaEvolve. Think of matrix multiplication as a recipe where every step costs real money on a chip. It beat the classic recipe for complex numbers, the first improvement in fifty-six years. Across a big set of open problems it matched the best known answer about three times in four and beat it about one time in five.
73
00:09:24.578 --> 00:09:26.116
Matching is the baseline?
74
00:09:26.116 --> 00:09:32.267
Matching is my baseline vendor. Beating it one time in five gets a pilot from me.
75
00:09:32.267 --> 00:09:38.418
My CMO forwarded me the gold medal headline with three exclamation marks. Is that a stunt?
76
00:09:38.418 --> 00:09:50.336
Later that summer, an advanced Gemini Deep Think scored gold, officially confirmed, and wrote in plain English inside the time limit. An OpenAI model matched the score, outside the official contest.
77
00:09:50.336 --> 00:09:55.334
Plain English, not Lean. I'll file that, because I think it matters later.
78
00:09:55.334 --> 00:09:56.872
It does.
Hold it.
79
00:09:56.872 --> 00:10:15.326
Then the Erdős problems. I saw two people outside any lab solved one with Gemini. No, sorry, the OpenAI one. I mix the names up, it's practically my job. That's the part I care about, because every small shop I know assumes this stuff needs a lab's budget.
80
00:10:15.326 --> 00:10:39.931
Kevin Barreto and Liam Price, in January, with GPT-5.2 Pro. The proof was checked in Lean by Harmonic's system, Aristotle, and Terence Tao called it the first solved largely autonomously, in the spirit the problem intended. In May, DeepMind's AlphaProof Nexus, a language model looping against Lean, resolved a batch more. Days later an OpenAI model disproved Erdős's old conjecture about distances between points.
81
00:10:39.931 --> 00:10:43.006
The one from the cold open. Disproved how?
82
00:10:43.006 --> 00:10:58.000
Picture dots on a page, with a stick of exactly one length joining any two that are that far apart. Erdős guessed you can have only barely more than one stick per dot. The model built arrangements with more.
83
00:10:58.000 --> 00:10:59.752
And nine mathematicians checked it.
84
00:10:59.752 --> 00:11:09.915
Including the Fields medallist Timothy Gowers. They traced its key moves to algebraic number theory, a field far from the problem, and within weeks humans had improved the construction.
85
00:11:09.915 --> 00:11:13.068
So it disproved Erdős's bound. Not the whole problem.
86
00:11:13.068 --> 00:11:21.128
Not the whole problem. But a famous conjecture, overturned by a machine and confirmed by outsiders. That made me rewrite my roadmap slide.
87
00:11:21.128 --> 00:11:25.684
So what makes you less excited? You've been suspiciously happy this whole time.
88
00:11:25.684 --> 00:11:43.205
The checking. Lean is a language where a proof either compiles or fails. Tao says that means you don't need to trust the people you're working with. He calls AI plus formal verification water filters that filter out undrinkable water, and his summary of the models is unreliable but powerful.
89
00:11:43.205 --> 00:11:56.872
Unreliable but powerful. That's my entire vendor list. And water filters I understand. Mine are legal and brand safety, they slow everything down, and nobody loves them. But I'd take a slow filter over a pulled paper any day.
90
00:11:56.872 --> 00:12:05.983
Right, and that's the catch for me: it filters, it doesn't find. So somebody still has to produce the thing worth filtering, and the strain shows.
91
00:12:05.983 --> 00:12:15.094
Twenty-five Fields medallists signed a statement in September saying the labs' goals and mathematics' goals are severely misaligned. And then October, the papers that got pulled.
92
00:12:15.094 --> 00:12:36.470
Only some of them formalised in Lean. So the machines write proofs faster than people can read them. I know that from my side. Addy Osmani put it as the bottleneck moving from writing code to proving it works, and a study this year found most pull requests from AI agents, sixty-one percent, got no review before they merged or closed.
93
00:12:36.470 --> 00:12:37.872
No review at all.
94
00:12:37.872 --> 00:12:55.393
My engineers write less and read more now, and reading someone else's diff is harder than writing your own. If the referee is a person, the referee is the line that never shrinks. That keeps me up, because my review queue is the one thing I can't buy more of.
95
00:12:55.393 --> 00:13:08.009
Mine is the copy queue. My two directors can't read as fast as the tools write, and I've caught myself approving things I skimmed. That's worse than not reading, because now there's a signature on it.
96
00:13:08.009 --> 00:13:12.214
And I explain to the board why we shipped something nobody read.
97
00:13:12.214 --> 00:13:17.470
You love the tech and you're the one with the most to lose from it.
98
00:13:17.470 --> 00:13:20.274
That is a fair description of my job.
99
00:13:20.274 --> 00:13:24.128
Does it get worse with practice? People get tired of checking.
100
00:13:24.128 --> 00:13:42.000
Elena says exactly that, and she'd know. She's an emergency doctor and she proofreads her AI scribe's notes after every shift. A big UC Davis pilot found a small number of notes with errors that could do serious harm if nobody caught them, and the authors said clinician review stays imperative.
101
00:13:42.000 --> 00:13:45.546
Which is why you build it so tiredness doesn't matter.
102
00:13:45.546 --> 00:14:02.923
Robert Wachter's worry was that after a long run of accurate notes, the reviewer stops really looking. I know that one. My reviewers at six on a Friday are not the same readers as at ten on a Tuesday, and a proof checker is the same reader every hour.
103
00:14:02.923 --> 00:14:20.300
In English, please. Why does a checker make a model better, not just safer? Let me test it on my own desk first. If I had a thousand subject-line drafts a day and a button that told me instantly which ones were garbage, I'd be a genius by Friday.
104
00:14:20.300 --> 00:14:39.095
That's it exactly, you've just described the loop. A language model is fluent and unreliable, my junior engineer on a Friday afternoon. Alone, it hallucinates. Paired with a checker, every hallucination gets thrown away and every success becomes training data. AlphaProof Nexus attempted three hundred and fifty-three open Erdős problems and proved nine.
105
00:14:39.095 --> 00:14:41.222
Nine. That's a miserable batting average.
106
00:14:41.222 --> 00:14:55.053
And the failures cost nothing but computing time, because Lean rejected every wrong proof before a human had to read it. My engineers' wrong guesses cost me a review slot each. That's the difference I keep losing sleep over.
107
00:14:55.053 --> 00:14:57.890
Okay, but who pays for that computing time?
108
00:14:57.890 --> 00:15:04.628
The lab. Which is the point. They can afford a million bad guesses because nobody has to read them.
109
00:15:04.628 --> 00:15:15.267
That I can follow. A bad
guess that's free is a different business from a bad send that lands in twenty million inboxes and gets my name on the complaint.
110
00:15:15.267 --> 00:15:34.416
Every system we've named runs that loop. AlphaEvolve verifies, runs and scores proposed programs automatically, and DeepMind says that's why it works best where progress can be measured. AlphaGeometry manufactured its own practice theorems. DeepSeek-R1, the Chinese reasoning model, was trained with a reward that simply checked its final answer against the correct one.
111
00:15:34.416 --> 00:15:39.381
And there's a name for this. I swear I saw it on a slide.
112
00:15:39.381 --> 00:15:53.211
Jason Wei, the OpenAI researcher, called it verifier's law. The ease of training AI to solve a task is proportional to how verifiable the task is. Andrej Karpathy put it as, Software 2.0 easily automates what you can verify.
113
00:15:53.211 --> 00:15:56.048
So where does my desk sit on that?
114
00:15:56.048 --> 00:15:57.112
You tell me.
115
00:15:57.112 --> 00:16:08.815
Both ways. My lifecycle team, the autoship emails, we have a referee. People open, click, unsubscribe, and we see it within days. My brand team writing the voice of the company has none.
116
00:16:08.815 --> 00:16:13.780
Which means by your own logic the first one gets automated before the second.
117
00:16:13.780 --> 00:16:23.000
I hate that you just did that. But yes. And it's the other half of the brief problem, because nobody can check a strategic north star.
118
00:16:23.000 --> 00:16:31.781
There's a number behind that. A big survey of marketers found most of them use AI to produce content and well under half say it improved performance.
119
00:16:31.781 --> 00:16:45.115
Faster, not better. That's my quarterly review in one sentence. My brand strategy is the same, isn't it? Experts disagree what good looks like, the verdict arrives years later, and I'm the one who has to defend it in the meantime.
120
00:16:45.115 --> 00:16:53.246
And that's the part I'd hate to be in your chair for. You'd be defending a call that no loop can grade for two quarters.
121
00:16:53.246 --> 00:17:00.726
My daughter Maya has been living this. She quizzes herself with ChatGPT every day, and she's started screenshotting the times it's confidently wrong.
122
00:17:00.726 --> 00:17:03.002
What does she do with the screenshots?
123
00:17:03.002 --> 00:17:12.434
Sends them to me, with a skull emoji. But she knows a wrong answer that sounds specific and logical is the worst kind, because only a referee catches it.
124
00:17:12.434 --> 00:17:16.987
She's sixteen and already working out that the checker matters more than the confidence.
125
00:17:16.987 --> 00:17:31.947
Here's the weirder part I promised. Once a model can generate its own problems and grade its own answers, its training data is limited by computing power, not by human experts. Code is the second domain like that, because a program can be run against tests.
126
00:17:31.947 --> 00:17:38.126
So the model marks its own paper after all. That's the bit that makes me want an outside examiner.
127
00:17:38.126 --> 00:17:51.785
In maths the examiner is Lean, and it doesn't care how confident the model sounds. In code it's the test suite, and that's why I plan around it. Where we have tests, I let the agents run. Where we don't, I don't.
128
00:17:51.785 --> 00:17:56.338
And that I can take to finance. A rule for where agents are allowed.
129
00:17:56.338 --> 00:17:58.940
Boring is good. Boring means it's a rule.
130
00:17:58.940 --> 00:18:08.372
But here's my worry for the people under me. If the cheap-to-check work goes first, that's the work I hand my juniors. Subject-line variants, send copy, the first rung.
131
00:18:08.372 --> 00:18:13.250
Same fight I have with my engineers. The junior rung is the one with tests.
132
00:18:13.250 --> 00:18:17.803
So I have to build them a different first rung, and nobody sells that.
133
00:18:17.803 --> 00:18:27.235
Nobody sells it because I can't write a test for the review I give a junior engineer on a Friday. It's the part of my job no suite grades.
134
00:18:27.235 --> 00:18:30.812
Put that on a slide for my CFO. She'd frame it.
135
00:18:30.812 --> 00:18:50.000
Now my turn, and I'm going to argue this the way I argue a launch deck. Maths wins are cheap to produce, checkable in days, and sit on a public list. A biology win is none of those. So when I see a flood of maths announcements, I count it like a sale email that only reports the opens.
136
00:18:50.000 --> 00:18:58.925
And I'd say the check is the whole point. My agents' tests passing is not a scoreboard. It's a real referee, and it's why I let them run.
137
00:18:58.925 --> 00:19:14.544
Your tests cover what you thought to test. A list of solved problems covers what someone bothered to pose. Tao says AI is clearing the attention-starved long tail, problems posed once and never followed up. One claimed solution turned out to be in a 1977 paper by Erdős himself.
138
00:19:14.544 --> 00:19:17.413
The machine solved something the author solved decades ago.
139
00:19:17.413 --> 00:19:27.931
Princeton's Noga Alon has stopped working on Erdős problems. His line was, once AI started to solve them, there is no point anymore. Picture my junior copywriters hearing that about the easy briefs.
140
00:19:27.931 --> 00:19:37.175
That one lands on the headline, not on the method. My review queue is full of easy pickings too, and the tests still tell me which ones are real.
141
00:19:37.175 --> 00:19:43.550
Easy picking that passes the test is still a pick. I'll give you that. I won't give you the headline.
142
00:19:43.550 --> 00:19:49.606
Next, reporting bias. Labs publish wins, never misses. I know that move, I've sent it in a launch email.
143
00:19:49.606 --> 00:20:08.094
And that's where I have to give ground. Tao put the real success rate on Erdős problems in the low single digits in January, and not much higher later. The cleanest
test is First Proof, where mathematicians set unpublished research problems the models can't have seen. In June, the best of four public systems solved six of ten.
144
00:20:08.094 --> 00:20:13.831
Six of ten is great, and nowhere near what the announcements made me tell my CMO last month.
145
00:20:13.831 --> 00:20:18.294
I came in counting the wins. I'd have hated that in a vendor deck.
146
00:20:18.294 --> 00:20:34.231
You told me to hold the English thing, so here it is. The gold medals were written in plain English, not Lean. The voice of the company has no Lean either, and if the models got good at fuzzy work anyway, your referee theory is a theory about the past.
147
00:20:34.231 --> 00:20:47.938
That's the one that could make me wrong in front of my board. OpenAI did say its model used general techniques for hard-to-verify tasks. The referee still mattered in training, but if general reinforcement learning keeps improving on fuzzy tasks, maths' advantage shrinks.
148
00:20:47.938 --> 00:20:51.763
So what you're betting your roadmap on might not be what wins.
149
00:20:51.763 --> 00:20:59.094
I could be wrong, and I've been wrong before. In 2015 I ran a startup whose entire referee was me saying "seems fine."
150
00:20:59.094 --> 00:21:00.050
Every founder's referee.
151
00:21:00.050 --> 00:21:04.513
It failed spectacularly. That's why I trust a real checker over my own confidence.
152
00:21:04.513 --> 00:21:06.744
And the other half of the story?
153
00:21:06.744 --> 00:21:23.000
Maths needs no lab, so a failed attempt costs seconds, and decades of written maths gave the models plenty to learn from. I feel that one in my own shop: my agents retry for pennies, and that matters to me as much as the tests do. So verifiability alone isn't it.
154
00:21:23.000 --> 00:21:39.702
Here's my grade, from my side of the P&L. The referee idea is the best explanation I've heard, and it tells me which of my teams has a checker and which doesn't. But the headline volume is part substance and part scoreboard, and I'm not paying for the scoreboard half.
155
00:21:39.702 --> 00:21:48.053
Substance plus cheap retries plus banked answers. I'd say that grade is fair, and I'd say it's mostly substance. You'd say it's closer to half.
156
00:21:48.053 --> 00:21:48.721
I would.
157
00:21:48.721 --> 00:22:07.094
So where do I land. I came in saying scoreboard. I'm leaving saying half. What moved me is that failure is free when the referee is automatic. Nothing on my P&L works like that, and it tells me which of my teams gets squeezed first. I'll take the thesis and stay annoyed about the scoreboard.
158
00:22:07.094 --> 00:22:23.128
I came in saying the referee explains it. What moved me is the success rate. Low single digits, not much higher later, and a bit over half on problems nobody had seen. I'd been counting wins. That's benchmark theatre, I hate benchmark theatre, and I was doing it.
159
00:22:23.128 --> 00:22:25.132
I'd like that on the record.
160
00:22:25.132 --> 00:22:31.145
You get one. Where I won't move: the order of fields is predictable by who has a checker.
161
00:22:31.145 --> 00:22:37.157
And I still want proof it ships. So the honest split is how much of this is substance.
162
00:22:37.157 --> 00:22:46.177
On Monday I'm listing what my team has a referee for and what it doesn't. Tests, yes. Anything that ends in a person reading a diff, no.
163
00:22:46.177 --> 00:23:04.549
Mine's the same list. Send performance, I'll let agents near it. Brand voice, nobody touches it until someone shows me a way to check it. And when finance asks next quarter what the AI line bought us, my answer is that we pay for what has a referee and we stop paying for what doesn't.
164
00:23:04.549 --> 00:23:06.219
That's a sentence I'd sign.
165
00:23:06.219 --> 00:23:09.226
Then sign it before the holidays, when nobody's reading.
166
00:23:09.226 --> 00:23:21.251
One thing we haven't said. AlphaEvolve's scheduling rule already recovers a fraction of one percent of Google's worldwide computing. That's boring, and it's the kind of work that gets automated first, because the check is cheap.
167
00:23:21.251 --> 00:23:32.274
And the fight that actually worries me is the Fields medallists. They weren't mostly complaining about wrong answers. They were complaining about labs announcing results faster than anyone can check or cite them.
168
00:23:32.274 --> 00:23:41.962
Which means whoever controls verification decides who gets the credit. And that's my vendor problem in a lab coat. If one company owns the checker, it owns the scoreboard.
169
00:23:41.962 --> 00:23:48.643
In my world that's the sending platform. Whoever owns the open rates owns the story of whether my campaign worked.
170
00:23:48.643 --> 00:24:00.000
So I'll be watching who builds referees for biology, materials and medicine, because whoever does will decide where the next big results land, and I'd rather be buying from them than competing with them.
171
00:24:00.000 --> 00:24:14.596
We're an AI-voiced show, and we want to be clear about that. Dan builds a custom stack of AI tools to produce this show, and every claim in it is checked against sources, but the hosts and their lives are fiction, and nothing here is advice.
172
00:24:14.596 --> 00:24:28.875
Follow Dan's AI Intel on Spotify or Apple Podcasts so the next episode finds you. And if someone in your life keeps asking where AI is heading, sending them this one is the kindest thing you can do for both of them and the show.
173
00:24:28.875 --> 00:24:29.827
Thanks for listening.
174
00:24:29.827 --> 00:24:33.000
And I still want to know who checks page forty.