WEBVTT
00:00:00.000 --> 00:00:02.879
Imagine hiring like a tier one financial analyst.
00:00:03.060 --> 00:00:05.240
Okay. Right. Like the best of the best. Right.
00:00:05.339 --> 00:00:07.879
They can synthesize complex market data in seconds.
00:00:07.940 --> 00:00:10.679
They never sleep and they have, you know, the
00:00:10.679 --> 00:00:13.199
entire history of the SEC's Edgar database just
00:00:13.199 --> 00:00:15.460
completely memorized. Sounds like an absolute
00:00:15.460 --> 00:00:17.760
dream hire, honestly. It does, but there's a
00:00:17.760 --> 00:00:20.359
catch. Oh. And the catch is their billing structure.
00:00:20.859 --> 00:00:24.269
Of course. Exactly. There's always a catch. So
00:00:24.269 --> 00:00:26.670
you pay this analyst for every single word their
00:00:26.670 --> 00:00:30.870
eyes scan. Oh, wow. Yeah. If they read one short
00:00:30.870 --> 00:00:32.810
summary paragraph to answer your question, the
00:00:32.810 --> 00:00:36.210
bill is negligible. But if they decide to pull
00:00:36.210 --> 00:00:38.649
five thick 10K filings off the shelf to cross
00:00:38.649 --> 00:00:41.250
-reference a single revenue line, you get hit
00:00:41.250 --> 00:00:43.310
with a massive invoice. Right, because they read
00:00:43.310 --> 00:00:45.770
so much more text. Exactly. Now scale that up
00:00:45.770 --> 00:00:48.070
to an enterprise making thousands of queries
00:00:48.070 --> 00:00:51.369
a day, and you are staring down a runaway operational
00:00:51.369 --> 00:00:54.380
expense. just to maintain basic accuracy. Yeah,
00:00:54.420 --> 00:00:56.520
and that really highlights a critical shift in
00:00:56.520 --> 00:00:58.520
the industry right now. I mean, we spent the
00:00:58.520 --> 00:01:01.520
last few years totally obsessed with the ceiling
00:01:01.520 --> 00:01:04.680
of model intelligence. Like, just how smart can
00:01:04.680 --> 00:01:08.810
we make it? Exactly. And now the hangover is
00:01:08.810 --> 00:01:11.129
sort of setting in. The conversation has completely
00:01:11.129 --> 00:01:13.829
moved from, you know, how smart is this system
00:01:13.829 --> 00:01:17.269
to how much does this system's intelligence actually
00:01:17.269 --> 00:01:20.900
cost us at scale? Because every single context
00:01:20.900 --> 00:01:23.900
window expansion has a literal quantifiable price
00:01:23.900 --> 00:01:26.200
tag attached to it. It really does. Every single
00:01:26.200 --> 00:01:29.079
token. Which brings us to the focus of today's
00:01:29.079 --> 00:01:32.040
deep dive. Welcome in, everyone. Today, we are
00:01:32.040 --> 00:01:34.319
looking at a highly focused paper from October
00:01:34.319 --> 00:01:38.459
2026. It's titled Token Budgeted Escalation for
00:01:38.459 --> 00:01:41.420
Financial Document QA. Yep. Published on Archive
00:01:41.420 --> 00:01:44.099
by Junru Zhu and their research team. That's
00:01:44.099 --> 00:01:46.120
the one. And for you listening, our mission today
00:01:46.120 --> 00:01:48.719
is to really map out the hidden economics of
00:01:48.719 --> 00:01:52.299
AI. We are bypassing all the basics to look directly
00:01:52.299 --> 00:01:55.099
at the tension between maximizing retrieval quality
00:01:55.099 --> 00:01:58.340
and, well, surviving your cloud provider's billing
00:01:58.340 --> 00:02:01.280
cycle. Which is a very real struggle. I mean,
00:02:01.280 --> 00:02:03.159
it forces us to view system architecture through
00:02:03.159 --> 00:02:05.859
this lens of microeconomics. Right. What Zhu
00:02:05.859 --> 00:02:07.959
and the team are doing here is treating the language
00:02:07.959 --> 00:02:10.919
model not just as a reasoning engine, but as
00:02:10.919 --> 00:02:13.550
a resource allocation problem. Because every
00:02:13.550 --> 00:02:15.469
additional document you feed into the prompt
00:02:15.469 --> 00:02:18.550
has a marginal cost and a marginal benefit. Exactly.
00:02:18.750 --> 00:02:21.449
You have to weigh those against each other. So
00:02:21.449 --> 00:02:22.930
let's ground this in the enterprise reality.
00:02:23.310 --> 00:02:25.990
We all know how standard RAG pipelines operate
00:02:25.990 --> 00:02:27.750
at this point, right? Right. Retrieval augmented
00:02:27.750 --> 00:02:30.250
generation. Yeah. Think of RAG like an open book
00:02:30.250 --> 00:02:32.870
test for an AI. It looks at your company's documents
00:02:32.870 --> 00:02:35.569
to find the answer. You retrieve the most semantically
00:02:35.569 --> 00:02:37.750
relevant chunk, you generate the answer, you're
00:02:37.750 --> 00:02:40.919
done. Usually, yeah. But when you hit a highly
00:02:40.919 --> 00:02:44.159
complex query, say you're asking an AI to reconcile
00:02:44.159 --> 00:02:47.159
a company's adjusted EBITDA across three turbulent
00:02:47.159 --> 00:02:50.099
quarters, that top one retrieval, just reading
00:02:50.099 --> 00:02:53.479
that one best document, it often fails. It just
00:02:53.479 --> 00:02:56.259
doesn't have enough context. Exactly. The system
00:02:56.259 --> 00:02:59.060
needs to broaden its search. It needs what we
00:02:59.060 --> 00:03:01.379
call escalation. Right. And the problem with
00:03:01.379 --> 00:03:03.979
escalation is that it's usually deployed as a
00:03:03.979 --> 00:03:06.620
blunt instrument. What do you mean by that? Well,
00:03:06.719 --> 00:03:09.520
if a company realizes their top one retrieval
00:03:09.520 --> 00:03:12.340
is failing on these hard questions, the engineering
00:03:12.340 --> 00:03:14.919
team's default reaction is to just crank the
00:03:14.919 --> 00:03:18.360
dial. They just default to top five or top ten
00:03:18.360 --> 00:03:20.979
retrieval for everything. Oh, I see. So they
00:03:20.979 --> 00:03:23.360
just expand the context window to brute force
00:03:23.360 --> 00:03:25.719
the accuracy for every single question? Exactly.
00:03:25.919 --> 00:03:29.240
Just read everything, every time. But comparing
00:03:29.240 --> 00:03:32.300
that to our analyst analogy from earlier, that's
00:03:32.300 --> 00:03:34.979
like sending an entire geographical survey team
00:03:34.979 --> 00:03:37.319
to check if it's raining instead of just sending
00:03:37.319 --> 00:03:39.860
one scout to look out the window. That's a great
00:03:39.860 --> 00:03:42.180
way to put it. I mean, sure, the survey team
00:03:42.180 --> 00:03:44.060
will give you a highly accurate answer about
00:03:44.060 --> 00:03:46.139
the rain, but you've wasted an enormous amount
00:03:46.139 --> 00:03:48.800
of resources. Yeah, when you escalate from top
00:03:48.800 --> 00:03:51.599
one to top five retrieval, you are essentially
00:03:51.599 --> 00:03:54.780
multiplying your token consumption by five for
00:03:54.780 --> 00:03:57.219
that specific query. And if you have a fixed
00:03:57.219 --> 00:03:59.520
enterprise budget, you simply cannot afford to
00:03:59.520 --> 00:04:01.580
do that across the board. You'd go bankrupt.
00:04:01.800 --> 00:04:04.280
And that constraint is really the catalyst for
00:04:04.280 --> 00:04:06.840
this entire paper. So if you have a fixed budget,
00:04:07.080 --> 00:04:10.039
how do you decide which questions actually deserve
00:04:10.039 --> 00:04:13.439
the extra spend? Right. If you only have the
00:04:13.439 --> 00:04:16.319
compute budget to escalate, say, 10 % of your
00:04:16.319 --> 00:04:19.379
daily queries to a top five retrieval, how do
00:04:19.379 --> 00:04:21.860
you mathematically identify the specific 10 %
00:04:21.860 --> 00:04:25.319
that actually need that extra context? Well,
00:04:25.379 --> 00:04:27.800
looking at their methodology, they didn't test
00:04:27.800 --> 00:04:30.160
this on general trivia, right? They went straight
00:04:30.160 --> 00:04:33.759
for the hard stuff. Oh, yeah. They used FinanceBench.
00:04:33.860 --> 00:04:36.620
Which is pretty notoriously unforgiving, from
00:04:36.620 --> 00:04:38.779
what I understand. Extremely unforgiving. It's
00:04:38.779 --> 00:04:41.180
designed specifically to stress test QA systems
00:04:41.180 --> 00:04:44.579
on dense, high -stakes financial material. We're
00:04:44.579 --> 00:04:47.600
talking earnings reports, balance sheets, complex
00:04:47.600 --> 00:04:50.579
tabular data. Not exactly light reading. No,
00:04:50.639 --> 00:04:54.470
not at all. So they pulled 150 questions from
00:04:54.470 --> 00:04:56.410
this benchmark to serve as their testing ground.
00:04:56.850 --> 00:04:59.029
Okay, let's unpack this. I want to break down
00:04:59.029 --> 00:05:01.509
the mechanics of how they actually built this
00:05:01.509 --> 00:05:03.790
trap for the token burn. They set up a two -stage
00:05:03.790 --> 00:05:06.250
process, right? Yeah, exactly. Two stages. So
00:05:06.250 --> 00:05:08.930
stage one is the baseline, top one retrieval.
00:05:08.990 --> 00:05:11.290
The model gets the single best document. Right.
00:05:11.350 --> 00:05:13.870
And stage two is the optional escalation to top
00:05:13.870 --> 00:05:16.689
five retrieval, where it reads the top five documents.
00:05:17.050 --> 00:05:19.269
But the filter between stage one and stage two
00:05:19.269 --> 00:05:20.910
is what we really care about here. How does it
00:05:20.910 --> 00:05:23.449
decide? So they rank every query using a metric
00:05:23.449 --> 00:05:27.189
they call gain per token. Gain per token. That
00:05:27.189 --> 00:05:29.930
sounds like a pure ROI calculation. It basically
00:05:29.930 --> 00:05:33.970
is. That metric is the engine of the paper. Before
00:05:33.970 --> 00:05:36.269
the system commits to the expensive top five
00:05:36.269 --> 00:05:38.930
retrieval, it runs a much smaller, lightweight
00:05:38.930 --> 00:05:41.689
prediction model. Like a proxy model. Exactly.
00:05:41.910 --> 00:05:45.209
And the goal of this proxy model is to estimate
00:05:45.209 --> 00:05:47.949
two distinct variables. First, the denominator.
00:05:48.519 --> 00:05:51.240
How many extra tokens will this specific escalation
00:05:51.240 --> 00:05:54.060
cost? Okay, the cost. And second, the numerator.
00:05:54.519 --> 00:05:57.560
What is the actual quantitative benefit to the
00:05:57.560 --> 00:06:00.839
answer quality if we spend those tokens? The
00:06:00.839 --> 00:06:03.540
logic makes total sense on paper. You divide
00:06:03.540 --> 00:06:05.379
the predicted benefit by the predicted cost,
00:06:05.639 --> 00:06:07.939
generate a ranked list based on ROI, and just
00:06:07.939 --> 00:06:10.120
buy the most profitable escalations until your
00:06:10.120 --> 00:06:12.579
token budget hits zero. That's the dream. But
00:06:12.579 --> 00:06:14.339
there's a massive friction point in the data
00:06:14.339 --> 00:06:16.560
they published. Predicting the token cost is
00:06:16.560 --> 00:06:18.920
simple math. The paper shows an R -squared of
00:06:18.920 --> 00:06:21.639
0 .93 for cost prediction. Right. They knew almost
00:06:21.639 --> 00:06:24.360
exactly what the bill will be. But the benefit
00:06:24.360 --> 00:06:26.600
prediction model, guessing whether reading those
00:06:26.600 --> 00:06:29.319
extra documents will actually fix a broken answer,
00:06:29.560 --> 00:06:35.040
that scored an AUROC of just 0 .60. Yeah, and
00:06:35.040 --> 00:06:36.699
you're touching on the core vulnerability of
00:06:36.699 --> 00:06:38.879
the entire system right there. Let's define that
00:06:38.879 --> 00:06:42.779
jargon real quick for people. AUROC. Sure. So
00:06:42.779 --> 00:06:45.829
to put that... 0 .60 AUROC into perspective.
00:06:46.250 --> 00:06:50.170
An AUROC of 0 .50 means the model's predictions
00:06:50.170 --> 00:06:52.610
are literally indistinguishable from a coin flip.
00:06:52.769 --> 00:06:55.829
Just totally random chance. Right. And an AUROC
00:06:55.829 --> 00:06:58.930
of 1 .0 is a perfect oracle. It knows with absolute
00:06:58.930 --> 00:07:03.170
certainty. Wait, so 0 .60 is barely better than
00:07:03.170 --> 00:07:05.430
a coin flip. It's very weak. Yeah. That's like
00:07:05.430 --> 00:07:08.230
knowing exactly down to the penny what a fancy
00:07:08.230 --> 00:07:09.970
five course dinner will cost, but having basically
00:07:09.970 --> 00:07:12.269
no idea if you'll actually enjoy it. That is
00:07:12.269 --> 00:07:15.110
the perfect analogy. And that friction is the
00:07:15.110 --> 00:07:17.709
central technical problem of the paper. So 0
00:07:17.709 --> 00:07:19.850
.60 means they are basically glancing at the
00:07:19.850 --> 00:07:22.029
query, glancing at the documents, and throwing
00:07:22.029 --> 00:07:25.050
a slightly weighted dart at a board. How does
00:07:25.050 --> 00:07:27.389
a lightweight proxy model even attempt to guess
00:07:27.389 --> 00:07:29.649
if a massive language model is going to experience
00:07:29.649 --> 00:07:32.769
an aha moment? Well, it has to rely on heuristic
00:07:32.769 --> 00:07:35.430
signals rather than actual comprehension. Like
00:07:35.430 --> 00:07:38.050
what? The smaller model might look at, say, the
00:07:38.050 --> 00:07:40.209
density of numerical data in the retrieved chunks.
00:07:40.689 --> 00:07:43.050
or measure the semantic distance between the
00:07:43.050 --> 00:07:45.389
phrasing of the question and the phrasing in
00:07:45.389 --> 00:07:47.269
the top one document. Okay, I think I follow.
00:07:47.569 --> 00:07:50.389
So if a query asks for a comparison of three
00:07:50.389 --> 00:07:53.449
companies and the proxy model notices the top
00:07:53.449 --> 00:07:56.649
one document only mentions one company. Oh, it
00:07:56.649 --> 00:07:58.990
flags the query as having a high potential benefit
00:07:58.990 --> 00:08:01.790
for escalation. Exactly. Because it clearly needs
00:08:01.790 --> 00:08:04.310
more info. But it doesn't actually know if the
00:08:04.310 --> 00:08:06.689
top five documents contain the missing two companies.
00:08:06.930 --> 00:08:09.350
It's just guessing that they might. Feels like
00:08:09.350 --> 00:08:11.670
a brutal limitation. You're buying completely
00:08:11.670 --> 00:08:15.470
blind. You are. But the researchers are refreshingly
00:08:15.470 --> 00:08:18.009
candid about that blindness. Estimating string
00:08:18.009 --> 00:08:20.550
length, the cost, is a solved computer science
00:08:20.550 --> 00:08:22.910
problem. Right. Predicting the latent reasoning
00:08:22.910 --> 00:08:25.810
trajectory of a frontier language model before
00:08:25.810 --> 00:08:29.170
it actually executes the prompt, that is an unsolved,
00:08:29.230 --> 00:08:32.019
profoundly difficult problem. Here's where it
00:08:32.019 --> 00:08:34.320
gets really interesting, though, because despite
00:08:34.320 --> 00:08:37.460
that incredibly weak benefit predictor, the actual
00:08:37.460 --> 00:08:39.840
impact they recorded is what makes this paper
00:08:39.840 --> 00:08:41.899
worth discussing today. Yeah, the results are
00:08:41.899 --> 00:08:43.940
pretty surprising. Let's look at the numbers.
00:08:44.159 --> 00:08:47.480
Under a strict token budget, say they only allowed
00:08:47.480 --> 00:08:50.379
10 % of the maximum possible token spend, the
00:08:50.379 --> 00:08:53.940
system still works. It works remarkably well
00:08:53.940 --> 00:08:56.139
considering the handicap. When they applied this
00:08:56.139 --> 00:08:59.100
gain per token prioritization using their 0 .60
00:08:59.100 --> 00:09:02.720
AUROC predictor, They managed to improve overall
00:09:02.720 --> 00:09:06.519
answer quality by 0 .034 points on the benchmark
00:09:06.519 --> 00:09:09.639
scale. Which, I mean, doesn't sound like a tectonic
00:09:09.639 --> 00:09:13.720
shift on paper. No, 0 .034 is small. But in the
00:09:13.720 --> 00:09:16.379
context of finance bench, squeezing out any positive
00:09:16.379 --> 00:09:18.899
delta while severely starving the system of resources
00:09:18.899 --> 00:09:21.899
is a real achievement. And the real headline
00:09:21.899 --> 00:09:25.059
is the efficiency metric. By targeting the escalation
00:09:25.059 --> 00:09:28.139
rather than applying a blanket rule. They used
00:09:28.139 --> 00:09:31.740
46 .6 % fewer tokens than a uniform top five
00:09:31.740 --> 00:09:34.480
retrieval policy. Wow. Yeah. They essentially
00:09:34.480 --> 00:09:37.500
halved their operational cost while still realizing
00:09:37.500 --> 00:09:40.299
a quality improvement. So they saved nearly half
00:09:40.299 --> 00:09:43.059
their token budget and got slightly better answers.
00:09:43.299 --> 00:09:46.360
But I have to ask, is this actually a new computer
00:09:46.360 --> 00:09:49.220
science breakthrough or just a clever repackaging
00:09:49.220 --> 00:09:52.100
of an old business idea? That's a very fair question.
00:09:52.299 --> 00:09:55.029
Because looking at the architecture... They haven't
00:09:55.029 --> 00:09:57.289
invented a new neural network, and they haven't
00:09:57.289 --> 00:09:59.649
really altered the fundamental mechanics of retrieval
00:09:59.649 --> 00:10:02.389
itself. If we evaluate the pure novelty of the
00:10:02.389 --> 00:10:04.909
work, it sits around a three out of five. The
00:10:04.909 --> 00:10:06.750
components themselves are totally off the shelf.
00:10:07.090 --> 00:10:10.389
So what's the real advance here? The true innovation
00:10:10.389 --> 00:10:13.929
is the decision under budget framing. They took
00:10:13.929 --> 00:10:16.070
a concept that has existed in data science and
00:10:16.070 --> 00:10:18.350
marketing for decades, which is uplift targeting,
00:10:18.629 --> 00:10:21.730
imported it directly into the LLM routing layer.
00:10:22.210 --> 00:10:24.289
Uplift targeting. Okay, let's map that for a
00:10:24.289 --> 00:10:26.710
second. In marketing, you divide your customer
00:10:26.710 --> 00:10:29.230
base into segments before you send out an expensive
00:10:29.230 --> 00:10:31.509
promotional discount, right? Exactly. You have
00:10:31.509 --> 00:10:33.870
your sure things, people who will buy your product
00:10:33.870 --> 00:10:35.970
at full price anyway. Right. And you have your
00:10:35.970 --> 00:10:37.950
lost causes, people who won't buy it even if
00:10:37.950 --> 00:10:40.350
it's 90 % off. You don't want to waste your marketing
00:10:40.350 --> 00:10:42.769
budget on either of those groups. No, that's
00:10:42.769 --> 00:10:44.669
just throwing money away. You only want to spend
00:10:44.669 --> 00:10:47.470
money on the persuadables, where the coupon actually
00:10:47.470 --> 00:10:50.230
changes the behavioral outcome. And the mapping
00:10:50.230 --> 00:10:52.429
to language models here is completely symmetrical.
00:10:52.830 --> 00:10:55.149
How so? Well, the sure things are the queries
00:10:55.149 --> 00:10:57.649
where the top one document is already perfectly
00:10:57.649 --> 00:11:00.429
sufficient. Spending tokens to pull top five
00:11:00.429 --> 00:11:02.909
is a total waste. Because they already have the
00:11:02.909 --> 00:11:05.549
right answer. Exactly. And the lost causes are
00:11:05.549 --> 00:11:08.490
queries so convoluted or where your database
00:11:08.490 --> 00:11:10.950
is so incomplete that even pulling 50 documents
00:11:10.950 --> 00:11:12.970
won't help the AI generate a correct answer.
00:11:13.169 --> 00:11:15.110
Expanding the context window there is also a
00:11:15.110 --> 00:11:17.769
waste. Right. You only want to spend the extra
00:11:17.769 --> 00:11:21.090
compute on the persuadable queries, the ones
00:11:21.090 --> 00:11:23.029
where the answer is currently wrong under top
00:11:23.029 --> 00:11:25.629
one, but will mathematically flip to a correct
00:11:25.629 --> 00:11:28.450
answer under top five. And that reframing is
00:11:28.450 --> 00:11:30.309
the honest lesson for the industry right now,
00:11:30.370 --> 00:11:33.190
isn't it? The paper proves that optimizing your
00:11:33.190 --> 00:11:35.250
token spend isn't necessarily about building
00:11:35.250 --> 00:11:37.669
a better search algorithm. It's about building
00:11:37.669 --> 00:11:41.509
a better triage algorithm. Precisely. The bottleneck
00:11:41.509 --> 00:11:44.409
preventing us from scaling AI efficiently isn't...
00:11:44.539 --> 00:11:46.820
generation speed anymore. It's our inability
00:11:46.820 --> 00:11:49.559
to accurately identify those persuadable queries
00:11:49.559 --> 00:11:52.000
in real time. If we connect this to the bigger
00:11:52.000 --> 00:11:55.360
picture, then if triage and budget routing are
00:11:55.360 --> 00:11:58.120
the new frontiers, Zoo's team can't be the only
00:11:58.120 --> 00:12:00.340
ones trying to solve this. How does this fit
00:12:00.340 --> 00:12:02.779
into the related work happening right now? Oh,
00:12:02.799 --> 00:12:05.200
there is a lot of movement in the agentic economic
00:12:05.200 --> 00:12:07.879
space right now. A prime example is a recent
00:12:07.879 --> 00:12:11.250
paper titled, Agentic Rag Evaluation. Budget
00:12:11.250 --> 00:12:14.110
Allocation Across Questions, Trajectories, and
00:12:14.110 --> 00:12:16.690
Reads. Based on the title alone, that sounds
00:12:16.690 --> 00:12:18.710
like it's tackling the exact same problem. They're
00:12:18.710 --> 00:12:21.669
both balancing budgets and reads. The terminology
00:12:21.669 --> 00:12:24.070
definitely overlaps, but the application is entirely
00:12:24.070 --> 00:12:27.070
different. That paper focuses heavily on the
00:12:27.070 --> 00:12:29.629
evaluation phase of system design. Meaning what
00:12:29.629 --> 00:12:32.309
exactly? Well, when you build an autonomous AI
00:12:32.309 --> 00:12:34.730
agent, you have to test it across multiple reasoning
00:12:34.730 --> 00:12:37.230
trajectories to see if it's actually working.
00:12:37.450 --> 00:12:39.570
Right. And that testing phase is incredibly expensive.
00:12:40.340 --> 00:12:42.419
Because you are paying the model to create its
00:12:42.419 --> 00:12:45.379
own complex homework, basically. Oh, I see. So
00:12:45.379 --> 00:12:47.580
their framework allocates token budgets across
00:12:47.580 --> 00:12:50.360
different grading paths to find the most efficient
00:12:50.360 --> 00:12:53.620
way to evaluate an agent offline. So it's a tool
00:12:53.620 --> 00:12:55.500
for the developers in the lab trying to figure
00:12:55.500 --> 00:12:58.100
out if the system is reliable without bankrupting
00:12:58.100 --> 00:13:01.539
their R &D budget. Exactly. Contrast that with
00:13:01.539 --> 00:13:04.659
a second notable paper we should mention. Beyond
00:13:04.659 --> 00:13:07.139
semantic similarity, performance and costs of
00:13:07.139 --> 00:13:10.019
agentic retrieval for complex tasks. Okay, what
00:13:10.019 --> 00:13:12.419
does that one do? That research takes a much
00:13:12.419 --> 00:13:14.919
wider macro -level view of how agents behave
00:13:14.919 --> 00:13:17.080
when they are allowed to autonomously browse
00:13:17.080 --> 00:13:20.320
databases or the web. It analyzes the general
00:13:20.320 --> 00:13:22.379
trade -offs between the autonomy of the agent,
00:13:22.519 --> 00:13:24.980
like how many steps it takes, how many dead ends
00:13:24.980 --> 00:13:27.799
it hits, and the resulting cloud compute bill.
00:13:28.019 --> 00:13:30.200
Which really clarifies the differentiator for
00:13:30.200 --> 00:13:32.639
Zhu's paper today. Beyond semantic similarity
00:13:32.639 --> 00:13:34.960
is looking at the messy, multi -step wandering
00:13:34.960 --> 00:13:37.690
of autonomous agents. Right. And the agentic
00:13:37.690 --> 00:13:39.970
RAG evaluation paper is about offline grading.
00:13:40.149 --> 00:13:42.669
What we are analyzing today is an in -flight
00:13:42.669 --> 00:13:45.289
router for a single real -time QA turn. Yes.
00:13:45.350 --> 00:13:47.629
It isn't about building the system or letting
00:13:47.629 --> 00:13:51.210
an agent browse the web. It is a strict moment
00:13:51.210 --> 00:13:53.570
-by -moment governor sitting inside a production
00:13:53.570 --> 00:13:56.929
environment. It's deciding in milliseconds whether
00:13:56.929 --> 00:13:58.929
a specific question gets the cheap treatment
00:13:58.929 --> 00:14:02.169
or the premium treatment. It's highly operational.
00:14:03.309 --> 00:14:05.509
It's the difference between designing a fuel
00:14:05.509 --> 00:14:08.110
-efficient engine. And actively deciding when
00:14:08.110 --> 00:14:10.610
to hit the gas pedal while you're already driving.
00:14:10.610 --> 00:14:14.129
That's a great analogy. It's totally live. So
00:14:14.129 --> 00:14:15.970
I want to carve out some space here for some
00:14:15.970 --> 00:14:19.049
critical analysis. For everyone listening, we
00:14:19.049 --> 00:14:21.690
are stepping outside the explicit claims of the
00:14:21.690 --> 00:14:24.269
paper now to offer some helpful opinions and
00:14:24.269 --> 00:14:26.250
caveats. Yeah, it's important to look at the
00:14:26.250 --> 00:14:28.470
gaps. We need to talk about the limitations,
00:14:28.649 --> 00:14:30.710
starting with the sample size. You mentioned
00:14:30.710 --> 00:14:33.809
earlier they tested this on 150 questions from
00:14:33.809 --> 00:14:37.029
FinanceBench. I did. I mean... A mid -sized hedge
00:14:37.029 --> 00:14:39.509
fund probably generates 150 questions before
00:14:39.509 --> 00:14:41.889
the morning stand -up. Yeah, the sample size
00:14:41.889 --> 00:14:44.850
is microscopic when you project it against enterprise
00:14:44.850 --> 00:14:47.990
reality. While finance bench is a rigorous, peer
00:14:47.990 --> 00:14:50.950
-reviewed standard, 150 data points simply do
00:14:50.950 --> 00:14:54.190
not provide enough statistical density to guarantee
00:14:54.190 --> 00:14:57.029
that their specific benefit prediction model
00:14:57.029 --> 00:15:00.070
will generalize. Right. If you point this routing
00:15:00.070 --> 00:15:03.519
system at a different sector, say... messy clinical
00:15:03.519 --> 00:15:06.419
trial data or dense legal contracts, there is
00:15:06.419 --> 00:15:08.600
no guarantee the proxy model will even achieve
00:15:08.600 --> 00:15:11.840
that 0 .60 AUROC. It could be even worse. And
00:15:11.840 --> 00:15:14.139
we really have to look closely at the risk profile
00:15:14.139 --> 00:15:17.840
of that 0 .60 AUROC in a live environment. Yeah,
00:15:17.840 --> 00:15:21.519
we celebrated the 46 .6 % cost savings. But if
00:15:21.519 --> 00:15:23.919
the predictor is only slightly better than chance
00:15:23.919 --> 00:15:27.159
at identifying those persuadable queries, you
00:15:27.159 --> 00:15:29.080
are inevitably going to throttle the context
00:15:29.080 --> 00:15:31.559
on questions that desperately needed escalation.
00:15:31.639 --> 00:15:33.679
Exactly. The cost savings are mathematically
00:15:33.679 --> 00:15:36.519
proven, but the risk to accuracy is not mitigated.
00:15:36.600 --> 00:15:38.480
And if you are building mission -critical software
00:15:38.480 --> 00:15:41.559
where a hallucinated number on an earnings summary
00:15:41.559 --> 00:15:44.059
could trigger a bad trade or a compliance violation,
00:15:44.480 --> 00:15:47.320
throttling contacts based on a 60 % probability
00:15:47.320 --> 00:15:49.960
metric is dangerous. Very dangerous. You are
00:15:49.960 --> 00:15:52.940
effectively accepting a certain baseline of hallucinations
00:15:52.940 --> 00:15:55.899
in exchange for keeping your AWS bill lower.
00:15:56.059 --> 00:15:58.500
So what does this all mean? It forces a pretty
00:15:58.500 --> 00:16:00.980
tough conversation for the engineering managers
00:16:00.980 --> 00:16:03.460
and product leads tasked with scaling these systems,
00:16:03.519 --> 00:16:05.940
doesn't it? It absolutely does. Brute forcing
00:16:05.940 --> 00:16:08.639
the context window works perfectly fine until
00:16:08.639 --> 00:16:12.240
the CFO actually sees the invoice. Right. But
00:16:12.240 --> 00:16:14.860
implementing an aggressive, imperfect triage
00:16:14.860 --> 00:16:17.720
system could compromise the trust of the analysts
00:16:17.720 --> 00:16:20.240
using the tool. It really comes down to understanding
00:16:20.240 --> 00:16:22.970
the nature of your specific deployment. If your
00:16:22.970 --> 00:16:26.730
system is just summarizing internal company newsletters,
00:16:26.870 --> 00:16:29.590
aggressive uplift targeting is brilliant. Oh,
00:16:29.629 --> 00:16:32.529
for sure. Go for it. But if it's performing real
00:16:32.529 --> 00:16:35.049
-time compliance checks on outbound trades, you
00:16:35.049 --> 00:16:37.110
might just have to accept the brute force token
00:16:37.110 --> 00:16:39.769
burn until the industry invents a better benefit
00:16:39.769 --> 00:16:42.169
predictor. You can't cut corners on compliance.
00:16:42.509 --> 00:16:45.330
You just have to pay the toll. Okay, let's pull
00:16:45.330 --> 00:16:47.850
all these threads together. We started by looking
00:16:47.850 --> 00:16:51.000
at the hidden token burn of RAG escalation. The
00:16:51.000 --> 00:16:53.620
danger of the AI reading too many expensive documents
00:16:53.620 --> 00:16:56.220
for every single query. Right. And we broke down
00:16:56.220 --> 00:16:59.320
Zoo and team's gain per token metric, dividing
00:16:59.320 --> 00:17:01.720
the predicted benefit by the highly predictable
00:17:01.720 --> 00:17:05.859
cost. The ROI calculation. Right. We saw that
00:17:05.859 --> 00:17:10.220
even using a flawed 0 .60 AUROC proxy model to
00:17:10.220 --> 00:17:13.059
guess that benefit, the system could slash token
00:17:13.059 --> 00:17:16.339
spend by nearly 47 % while actually slightly
00:17:16.339 --> 00:17:19.519
boosting the overall quality on a rigorous financial
00:17:19.519 --> 00:17:22.109
benchmark. And we place this inside the broader
00:17:22.109 --> 00:17:25.289
movement toward AI resource allocation, recognizing
00:17:25.289 --> 00:17:28.069
that predicting a language model's latent reasoning
00:17:28.069 --> 00:17:31.450
capability before it executes is really the next
00:17:31.450 --> 00:17:33.930
major bottleneck in computer science. The era
00:17:33.930 --> 00:17:36.349
of treating compute as infinite is definitely
00:17:36.349 --> 00:17:39.730
over. It is. And this raises an important question
00:17:39.730 --> 00:17:41.690
for everyone listening to consider as you interact
00:17:41.690 --> 00:17:44.210
with these systems going forward. Okay, let's
00:17:44.210 --> 00:17:47.470
hear it. If the computational cost of AI is highly
00:17:47.470 --> 00:17:50.200
predictable, but the actual intellectual benefit
00:17:50.200 --> 00:17:52.339
of that compute remains largely unpredictable
00:17:52.339 --> 00:17:55.579
until after the fact. How does that change the
00:17:55.579 --> 00:17:58.140
way businesses should contract for AI services?
00:17:58.559 --> 00:18:01.579
That's a fascinating angle, because the current
00:18:01.579 --> 00:18:03.980
model forces the user to absorb all the risk.
00:18:04.059 --> 00:18:06.539
You pay for the tokens the AI reads, whether
00:18:06.539 --> 00:18:09.599
the answer is brilliant or just a complete hallucination.
00:18:10.140 --> 00:18:13.480
Exactly. Will we see a shift toward paying strictly
00:18:13.480 --> 00:18:16.170
for the uplift? where the provider only gets
00:18:16.170 --> 00:18:18.890
paid if the extra compute actually delivers a
00:18:18.890 --> 00:18:20.910
mathematically verifiable improvement to the
00:18:20.910 --> 00:18:23.990
final output. Wow. It would completely upend
00:18:23.990 --> 00:18:26.509
the sauce pricing model. You'd stop paying for
00:18:26.509 --> 00:18:28.490
the hours your digital analyst spends in the
00:18:28.490 --> 00:18:31.150
library and start paying purely for the insights
00:18:31.150 --> 00:18:33.930
they deliver. A shift from renting compute to
00:18:33.930 --> 00:18:37.009
purchasing outcomes. I love that framing. Well,
00:18:37.069 --> 00:18:39.190
thank you for joining us on this deep dive, everyone.
00:18:39.700 --> 00:18:41.400
Keep questioning the architecture behind the
00:18:41.400 --> 00:18:43.920
interfaces you use. Keep an eye on your own token
00:18:43.920 --> 00:18:45.720
budgets, and we will catch you next time.
00:00:00.000 --> 00:00:02.879
Imagine hiring like a tier one financial analyst.
00:00:03.060 --> 00:00:05.240
Okay. Right. Like the best of the best. Right.
00:00:05.339 --> 00:00:07.879
They can synthesize complex market data in seconds.
00:00:07.940 --> 00:00:10.679
They never sleep and they have, you know, the
00:00:10.679 --> 00:00:13.199
entire history of the SEC's Edgar database just
00:00:13.199 --> 00:00:15.460
completely memorized. Sounds like an absolute
00:00:15.460 --> 00:00:17.760
dream hire, honestly. It does, but there's a
00:00:17.760 --> 00:00:20.359
catch. Oh. And the catch is their billing structure.
00:00:20.859 --> 00:00:24.269
Of course. Exactly. There's always a catch. So
00:00:24.269 --> 00:00:26.670
you pay this analyst for every single word their
00:00:26.670 --> 00:00:30.870
eyes scan. Oh, wow. Yeah. If they read one short
00:00:30.870 --> 00:00:32.810
summary paragraph to answer your question, the
00:00:32.810 --> 00:00:36.210
bill is negligible. But if they decide to pull
00:00:36.210 --> 00:00:38.649
five thick 10K filings off the shelf to cross
00:00:38.649 --> 00:00:41.250
-reference a single revenue line, you get hit
00:00:41.250 --> 00:00:43.310
with a massive invoice. Right, because they read
00:00:43.310 --> 00:00:45.770
so much more text. Exactly. Now scale that up
00:00:45.770 --> 00:00:48.070
to an enterprise making thousands of queries
00:00:48.070 --> 00:00:51.369
a day, and you are staring down a runaway operational
00:00:51.369 --> 00:00:54.380
expense. just to maintain basic accuracy. Yeah,
00:00:54.420 --> 00:00:56.520
and that really highlights a critical shift in
00:00:56.520 --> 00:00:58.520
the industry right now. I mean, we spent the
00:00:58.520 --> 00:01:01.520
last few years totally obsessed with the ceiling
00:01:01.520 --> 00:01:04.680
of model intelligence. Like, just how smart can
00:01:04.680 --> 00:01:08.810
we make it? Exactly. And now the hangover is
00:01:08.810 --> 00:01:11.129
sort of setting in. The conversation has completely
00:01:11.129 --> 00:01:13.829
moved from, you know, how smart is this system
00:01:13.829 --> 00:01:17.269
to how much does this system's intelligence actually
00:01:17.269 --> 00:01:20.900
cost us at scale? Because every single context
00:01:20.900 --> 00:01:23.900
window expansion has a literal quantifiable price
00:01:23.900 --> 00:01:26.200
tag attached to it. It really does. Every single
00:01:26.200 --> 00:01:29.079
token. Which brings us to the focus of today's
00:01:29.079 --> 00:01:32.040
deep dive. Welcome in, everyone. Today, we are
00:01:32.040 --> 00:01:34.319
looking at a highly focused paper from October
00:01:34.319 --> 00:01:38.459
2026. It's titled Token Budgeted Escalation for
00:01:38.459 --> 00:01:41.420
Financial Document QA. Yep. Published on Archive
00:01:41.420 --> 00:01:44.099
by Junru Zhu and their research team. That's
00:01:44.099 --> 00:01:46.120
the one. And for you listening, our mission today
00:01:46.120 --> 00:01:48.719
is to really map out the hidden economics of
00:01:48.719 --> 00:01:52.299
AI. We are bypassing all the basics to look directly
00:01:52.299 --> 00:01:55.099
at the tension between maximizing retrieval quality
00:01:55.099 --> 00:01:58.340
and, well, surviving your cloud provider's billing
00:01:58.340 --> 00:02:01.280
cycle. Which is a very real struggle. I mean,
00:02:01.280 --> 00:02:03.159
it forces us to view system architecture through
00:02:03.159 --> 00:02:05.859
this lens of microeconomics. Right. What Zhu
00:02:05.859 --> 00:02:07.959
and the team are doing here is treating the language
00:02:07.959 --> 00:02:10.919
model not just as a reasoning engine, but as
00:02:10.919 --> 00:02:13.550
a resource allocation problem. Because every
00:02:13.550 --> 00:02:15.469
additional document you feed into the prompt
00:02:15.469 --> 00:02:18.550
has a marginal cost and a marginal benefit. Exactly.
00:02:18.750 --> 00:02:21.449
You have to weigh those against each other. So
00:02:21.449 --> 00:02:22.930
let's ground this in the enterprise reality.
00:02:23.310 --> 00:02:25.990
We all know how standard RAG pipelines operate
00:02:25.990 --> 00:02:27.750
at this point, right? Right. Retrieval augmented
00:02:27.750 --> 00:02:30.250
generation. Yeah. Think of RAG like an open book
00:02:30.250 --> 00:02:32.870
test for an AI. It looks at your company's documents
00:02:32.870 --> 00:02:35.569
to find the answer. You retrieve the most semantically
00:02:35.569 --> 00:02:37.750
relevant chunk, you generate the answer, you're
00:02:37.750 --> 00:02:40.919
done. Usually, yeah. But when you hit a highly
00:02:40.919 --> 00:02:44.159
complex query, say you're asking an AI to reconcile
00:02:44.159 --> 00:02:47.159
a company's adjusted EBITDA across three turbulent
00:02:47.159 --> 00:02:50.099
quarters, that top one retrieval, just reading
00:02:50.099 --> 00:02:53.479
that one best document, it often fails. It just
00:02:53.479 --> 00:02:56.259
doesn't have enough context. Exactly. The system
00:02:56.259 --> 00:02:59.060
needs to broaden its search. It needs what we
00:02:59.060 --> 00:03:01.379
call escalation. Right. And the problem with
00:03:01.379 --> 00:03:03.979
escalation is that it's usually deployed as a
00:03:03.979 --> 00:03:06.620
blunt instrument. What do you mean by that? Well,
00:03:06.719 --> 00:03:09.520
if a company realizes their top one retrieval
00:03:09.520 --> 00:03:12.340
is failing on these hard questions, the engineering
00:03:12.340 --> 00:03:14.919
team's default reaction is to just crank the
00:03:14.919 --> 00:03:18.360
dial. They just default to top five or top ten
00:03:18.360 --> 00:03:20.979
retrieval for everything. Oh, I see. So they
00:03:20.979 --> 00:03:23.360
just expand the context window to brute force
00:03:23.360 --> 00:03:25.719
the accuracy for every single question? Exactly.
00:03:25.919 --> 00:03:29.240
Just read everything, every time. But comparing
00:03:29.240 --> 00:03:32.300
that to our analyst analogy from earlier, that's
00:03:32.300 --> 00:03:34.979
like sending an entire geographical survey team
00:03:34.979 --> 00:03:37.319
to check if it's raining instead of just sending
00:03:37.319 --> 00:03:39.860
one scout to look out the window. That's a great
00:03:39.860 --> 00:03:42.180
way to put it. I mean, sure, the survey team
00:03:42.180 --> 00:03:44.060
will give you a highly accurate answer about
00:03:44.060 --> 00:03:46.139
the rain, but you've wasted an enormous amount
00:03:46.139 --> 00:03:48.800
of resources. Yeah, when you escalate from top
00:03:48.800 --> 00:03:51.599
one to top five retrieval, you are essentially
00:03:51.599 --> 00:03:54.780
multiplying your token consumption by five for
00:03:54.780 --> 00:03:57.219
that specific query. And if you have a fixed
00:03:57.219 --> 00:03:59.520
enterprise budget, you simply cannot afford to
00:03:59.520 --> 00:04:01.580
do that across the board. You'd go bankrupt.
00:04:01.800 --> 00:04:04.280
And that constraint is really the catalyst for
00:04:04.280 --> 00:04:06.840
this entire paper. So if you have a fixed budget,
00:04:07.080 --> 00:04:10.039
how do you decide which questions actually deserve
00:04:10.039 --> 00:04:13.439
the extra spend? Right. If you only have the
00:04:13.439 --> 00:04:16.319
compute budget to escalate, say, 10 % of your
00:04:16.319 --> 00:04:19.379
daily queries to a top five retrieval, how do
00:04:19.379 --> 00:04:21.860
you mathematically identify the specific 10 %
00:04:21.860 --> 00:04:25.319
that actually need that extra context? Well,
00:04:25.379 --> 00:04:27.800
looking at their methodology, they didn't test
00:04:27.800 --> 00:04:30.160
this on general trivia, right? They went straight
00:04:30.160 --> 00:04:33.759
for the hard stuff. Oh, yeah. They used FinanceBench.
00:04:33.860 --> 00:04:36.620
Which is pretty notoriously unforgiving, from
00:04:36.620 --> 00:04:38.779
what I understand. Extremely unforgiving. It's
00:04:38.779 --> 00:04:41.180
designed specifically to stress test QA systems
00:04:41.180 --> 00:04:44.579
on dense, high -stakes financial material. We're
00:04:44.579 --> 00:04:47.600
talking earnings reports, balance sheets, complex
00:04:47.600 --> 00:04:50.579
tabular data. Not exactly light reading. No,
00:04:50.639 --> 00:04:54.470
not at all. So they pulled 150 questions from
00:04:54.470 --> 00:04:56.410
this benchmark to serve as their testing ground.
00:04:56.850 --> 00:04:59.029
Okay, let's unpack this. I want to break down
00:04:59.029 --> 00:05:01.509
the mechanics of how they actually built this
00:05:01.509 --> 00:05:03.790
trap for the token burn. They set up a two -stage
00:05:03.790 --> 00:05:06.250
process, right? Yeah, exactly. Two stages. So
00:05:06.250 --> 00:05:08.930
stage one is the baseline, top one retrieval.
00:05:08.990 --> 00:05:11.290
The model gets the single best document. Right.
00:05:11.350 --> 00:05:13.870
And stage two is the optional escalation to top
00:05:13.870 --> 00:05:16.689
five retrieval, where it reads the top five documents.
00:05:17.050 --> 00:05:19.269
But the filter between stage one and stage two
00:05:19.269 --> 00:05:20.910
is what we really care about here. How does it
00:05:20.910 --> 00:05:23.449
decide? So they rank every query using a metric
00:05:23.449 --> 00:05:27.189
they call gain per token. Gain per token. That
00:05:27.189 --> 00:05:29.930
sounds like a pure ROI calculation. It basically
00:05:29.930 --> 00:05:33.970
is. That metric is the engine of the paper. Before
00:05:33.970 --> 00:05:36.269
the system commits to the expensive top five
00:05:36.269 --> 00:05:38.930
retrieval, it runs a much smaller, lightweight
00:05:38.930 --> 00:05:41.689
prediction model. Like a proxy model. Exactly.
00:05:41.910 --> 00:05:45.209
And the goal of this proxy model is to estimate
00:05:45.209 --> 00:05:47.949
two distinct variables. First, the denominator.
00:05:48.519 --> 00:05:51.240
How many extra tokens will this specific escalation
00:05:51.240 --> 00:05:54.060
cost? Okay, the cost. And second, the numerator.
00:05:54.519 --> 00:05:57.560
What is the actual quantitative benefit to the
00:05:57.560 --> 00:06:00.839
answer quality if we spend those tokens? The
00:06:00.839 --> 00:06:03.540
logic makes total sense on paper. You divide
00:06:03.540 --> 00:06:05.379
the predicted benefit by the predicted cost,
00:06:05.639 --> 00:06:07.939
generate a ranked list based on ROI, and just
00:06:07.939 --> 00:06:10.120
buy the most profitable escalations until your
00:06:10.120 --> 00:06:12.579
token budget hits zero. That's the dream. But
00:06:12.579 --> 00:06:14.339
there's a massive friction point in the data
00:06:14.339 --> 00:06:16.560
they published. Predicting the token cost is
00:06:16.560 --> 00:06:18.920
simple math. The paper shows an R -squared of
00:06:18.920 --> 00:06:21.639
0 .93 for cost prediction. Right. They knew almost
00:06:21.639 --> 00:06:24.360
exactly what the bill will be. But the benefit
00:06:24.360 --> 00:06:26.600
prediction model, guessing whether reading those
00:06:26.600 --> 00:06:29.319
extra documents will actually fix a broken answer,
00:06:29.560 --> 00:06:35.040
that scored an AUROC of just 0 .60. Yeah, and
00:06:35.040 --> 00:06:36.699
you're touching on the core vulnerability of
00:06:36.699 --> 00:06:38.879
the entire system right there. Let's define that
00:06:38.879 --> 00:06:42.779
jargon real quick for people. AUROC. Sure. So
00:06:42.779 --> 00:06:45.829
to put that... 0 .60 AUROC into perspective.
00:06:46.250 --> 00:06:50.170
An AUROC of 0 .50 means the model's predictions
00:06:50.170 --> 00:06:52.610
are literally indistinguishable from a coin flip.
00:06:52.769 --> 00:06:55.829
Just totally random chance. Right. And an AUROC
00:06:55.829 --> 00:06:58.930
of 1 .0 is a perfect oracle. It knows with absolute
00:06:58.930 --> 00:07:03.170
certainty. Wait, so 0 .60 is barely better than
00:07:03.170 --> 00:07:05.430
a coin flip. It's very weak. Yeah. That's like
00:07:05.430 --> 00:07:08.230
knowing exactly down to the penny what a fancy
00:07:08.230 --> 00:07:09.970
five course dinner will cost, but having basically
00:07:09.970 --> 00:07:12.269
no idea if you'll actually enjoy it. That is
00:07:12.269 --> 00:07:15.110
the perfect analogy. And that friction is the
00:07:15.110 --> 00:07:17.709
central technical problem of the paper. So 0
00:07:17.709 --> 00:07:19.850
.60 means they are basically glancing at the
00:07:19.850 --> 00:07:22.029
query, glancing at the documents, and throwing
00:07:22.029 --> 00:07:25.050
a slightly weighted dart at a board. How does
00:07:25.050 --> 00:07:27.389
a lightweight proxy model even attempt to guess
00:07:27.389 --> 00:07:29.649
if a massive language model is going to experience
00:07:29.649 --> 00:07:32.769
an aha moment? Well, it has to rely on heuristic
00:07:32.769 --> 00:07:35.430
signals rather than actual comprehension. Like
00:07:35.430 --> 00:07:38.050
what? The smaller model might look at, say, the
00:07:38.050 --> 00:07:40.209
density of numerical data in the retrieved chunks.
00:07:40.689 --> 00:07:43.050
or measure the semantic distance between the
00:07:43.050 --> 00:07:45.389
phrasing of the question and the phrasing in
00:07:45.389 --> 00:07:47.269
the top one document. Okay, I think I follow.
00:07:47.569 --> 00:07:50.389
So if a query asks for a comparison of three
00:07:50.389 --> 00:07:53.449
companies and the proxy model notices the top
00:07:53.449 --> 00:07:56.649
one document only mentions one company. Oh, it
00:07:56.649 --> 00:07:58.990
flags the query as having a high potential benefit
00:07:58.990 --> 00:08:01.790
for escalation. Exactly. Because it clearly needs
00:08:01.790 --> 00:08:04.310
more info. But it doesn't actually know if the
00:08:04.310 --> 00:08:06.689
top five documents contain the missing two companies.
00:08:06.930 --> 00:08:09.350
It's just guessing that they might. Feels like
00:08:09.350 --> 00:08:11.670
a brutal limitation. You're buying completely
00:08:11.670 --> 00:08:15.470
blind. You are. But the researchers are refreshingly
00:08:15.470 --> 00:08:18.009
candid about that blindness. Estimating string
00:08:18.009 --> 00:08:20.550
length, the cost, is a solved computer science
00:08:20.550 --> 00:08:22.910
problem. Right. Predicting the latent reasoning
00:08:22.910 --> 00:08:25.810
trajectory of a frontier language model before
00:08:25.810 --> 00:08:29.170
it actually executes the prompt, that is an unsolved,
00:08:29.230 --> 00:08:32.019
profoundly difficult problem. Here's where it
00:08:32.019 --> 00:08:34.320
gets really interesting, though, because despite
00:08:34.320 --> 00:08:37.460
that incredibly weak benefit predictor, the actual
00:08:37.460 --> 00:08:39.840
impact they recorded is what makes this paper
00:08:39.840 --> 00:08:41.899
worth discussing today. Yeah, the results are
00:08:41.899 --> 00:08:43.940
pretty surprising. Let's look at the numbers.
00:08:44.159 --> 00:08:47.480
Under a strict token budget, say they only allowed
00:08:47.480 --> 00:08:50.379
10 % of the maximum possible token spend, the
00:08:50.379 --> 00:08:53.940
system still works. It works remarkably well
00:08:53.940 --> 00:08:56.139
considering the handicap. When they applied this
00:08:56.139 --> 00:08:59.100
gain per token prioritization using their 0 .60
00:08:59.100 --> 00:09:02.720
AUROC predictor, They managed to improve overall
00:09:02.720 --> 00:09:06.519
answer quality by 0 .034 points on the benchmark
00:09:06.519 --> 00:09:09.639
scale. Which, I mean, doesn't sound like a tectonic
00:09:09.639 --> 00:09:13.720
shift on paper. No, 0 .034 is small. But in the
00:09:13.720 --> 00:09:16.379
context of finance bench, squeezing out any positive
00:09:16.379 --> 00:09:18.899
delta while severely starving the system of resources
00:09:18.899 --> 00:09:21.899
is a real achievement. And the real headline
00:09:21.899 --> 00:09:25.059
is the efficiency metric. By targeting the escalation
00:09:25.059 --> 00:09:28.139
rather than applying a blanket rule. They used
00:09:28.139 --> 00:09:31.740
46 .6 % fewer tokens than a uniform top five
00:09:31.740 --> 00:09:34.480
retrieval policy. Wow. Yeah. They essentially
00:09:34.480 --> 00:09:37.500
halved their operational cost while still realizing
00:09:37.500 --> 00:09:40.299
a quality improvement. So they saved nearly half
00:09:40.299 --> 00:09:43.059
their token budget and got slightly better answers.
00:09:43.299 --> 00:09:46.360
But I have to ask, is this actually a new computer
00:09:46.360 --> 00:09:49.220
science breakthrough or just a clever repackaging
00:09:49.220 --> 00:09:52.100
of an old business idea? That's a very fair question.
00:09:52.299 --> 00:09:55.029
Because looking at the architecture... They haven't
00:09:55.029 --> 00:09:57.289
invented a new neural network, and they haven't
00:09:57.289 --> 00:09:59.649
really altered the fundamental mechanics of retrieval
00:09:59.649 --> 00:10:02.389
itself. If we evaluate the pure novelty of the
00:10:02.389 --> 00:10:04.909
work, it sits around a three out of five. The
00:10:04.909 --> 00:10:06.750
components themselves are totally off the shelf.
00:10:07.090 --> 00:10:10.389
So what's the real advance here? The true innovation
00:10:10.389 --> 00:10:13.929
is the decision under budget framing. They took
00:10:13.929 --> 00:10:16.070
a concept that has existed in data science and
00:10:16.070 --> 00:10:18.350
marketing for decades, which is uplift targeting,
00:10:18.629 --> 00:10:21.730
imported it directly into the LLM routing layer.
00:10:22.210 --> 00:10:24.289
Uplift targeting. Okay, let's map that for a
00:10:24.289 --> 00:10:26.710
second. In marketing, you divide your customer
00:10:26.710 --> 00:10:29.230
base into segments before you send out an expensive
00:10:29.230 --> 00:10:31.509
promotional discount, right? Exactly. You have
00:10:31.509 --> 00:10:33.870
your sure things, people who will buy your product
00:10:33.870 --> 00:10:35.970
at full price anyway. Right. And you have your
00:10:35.970 --> 00:10:37.950
lost causes, people who won't buy it even if
00:10:37.950 --> 00:10:40.350
it's 90 % off. You don't want to waste your marketing
00:10:40.350 --> 00:10:42.769
budget on either of those groups. No, that's
00:10:42.769 --> 00:10:44.669
just throwing money away. You only want to spend
00:10:44.669 --> 00:10:47.470
money on the persuadables, where the coupon actually
00:10:47.470 --> 00:10:50.230
changes the behavioral outcome. And the mapping
00:10:50.230 --> 00:10:52.429
to language models here is completely symmetrical.
00:10:52.830 --> 00:10:55.149
How so? Well, the sure things are the queries
00:10:55.149 --> 00:10:57.649
where the top one document is already perfectly
00:10:57.649 --> 00:11:00.429
sufficient. Spending tokens to pull top five
00:11:00.429 --> 00:11:02.909
is a total waste. Because they already have the
00:11:02.909 --> 00:11:05.549
right answer. Exactly. And the lost causes are
00:11:05.549 --> 00:11:08.490
queries so convoluted or where your database
00:11:08.490 --> 00:11:10.950
is so incomplete that even pulling 50 documents
00:11:10.950 --> 00:11:12.970
won't help the AI generate a correct answer.
00:11:13.169 --> 00:11:15.110
Expanding the context window there is also a
00:11:15.110 --> 00:11:17.769
waste. Right. You only want to spend the extra
00:11:17.769 --> 00:11:21.090
compute on the persuadable queries, the ones
00:11:21.090 --> 00:11:23.029
where the answer is currently wrong under top
00:11:23.029 --> 00:11:25.629
one, but will mathematically flip to a correct
00:11:25.629 --> 00:11:28.450
answer under top five. And that reframing is
00:11:28.450 --> 00:11:30.309
the honest lesson for the industry right now,
00:11:30.370 --> 00:11:33.190
isn't it? The paper proves that optimizing your
00:11:33.190 --> 00:11:35.250
token spend isn't necessarily about building
00:11:35.250 --> 00:11:37.669
a better search algorithm. It's about building
00:11:37.669 --> 00:11:41.509
a better triage algorithm. Precisely. The bottleneck
00:11:41.509 --> 00:11:44.409
preventing us from scaling AI efficiently isn't...
00:11:44.539 --> 00:11:46.820
generation speed anymore. It's our inability
00:11:46.820 --> 00:11:49.559
to accurately identify those persuadable queries
00:11:49.559 --> 00:11:52.000
in real time. If we connect this to the bigger
00:11:52.000 --> 00:11:55.360
picture, then if triage and budget routing are
00:11:55.360 --> 00:11:58.120
the new frontiers, Zoo's team can't be the only
00:11:58.120 --> 00:12:00.340
ones trying to solve this. How does this fit
00:12:00.340 --> 00:12:02.779
into the related work happening right now? Oh,
00:12:02.799 --> 00:12:05.200
there is a lot of movement in the agentic economic
00:12:05.200 --> 00:12:07.879
space right now. A prime example is a recent
00:12:07.879 --> 00:12:11.250
paper titled, Agentic Rag Evaluation. Budget
00:12:11.250 --> 00:12:14.110
Allocation Across Questions, Trajectories, and
00:12:14.110 --> 00:12:16.690
Reads. Based on the title alone, that sounds
00:12:16.690 --> 00:12:18.710
like it's tackling the exact same problem. They're
00:12:18.710 --> 00:12:21.669
both balancing budgets and reads. The terminology
00:12:21.669 --> 00:12:24.070
definitely overlaps, but the application is entirely
00:12:24.070 --> 00:12:27.070
different. That paper focuses heavily on the
00:12:27.070 --> 00:12:29.629
evaluation phase of system design. Meaning what
00:12:29.629 --> 00:12:32.309
exactly? Well, when you build an autonomous AI
00:12:32.309 --> 00:12:34.730
agent, you have to test it across multiple reasoning
00:12:34.730 --> 00:12:37.230
trajectories to see if it's actually working.
00:12:37.450 --> 00:12:39.570
Right. And that testing phase is incredibly expensive.
00:12:40.340 --> 00:12:42.419
Because you are paying the model to create its
00:12:42.419 --> 00:12:45.379
own complex homework, basically. Oh, I see. So
00:12:45.379 --> 00:12:47.580
their framework allocates token budgets across
00:12:47.580 --> 00:12:50.360
different grading paths to find the most efficient
00:12:50.360 --> 00:12:53.620
way to evaluate an agent offline. So it's a tool
00:12:53.620 --> 00:12:55.500
for the developers in the lab trying to figure
00:12:55.500 --> 00:12:58.100
out if the system is reliable without bankrupting
00:12:58.100 --> 00:13:01.539
their R &D budget. Exactly. Contrast that with
00:13:01.539 --> 00:13:04.659
a second notable paper we should mention. Beyond
00:13:04.659 --> 00:13:07.139
semantic similarity, performance and costs of
00:13:07.139 --> 00:13:10.019
agentic retrieval for complex tasks. Okay, what
00:13:10.019 --> 00:13:12.419
does that one do? That research takes a much
00:13:12.419 --> 00:13:14.919
wider macro -level view of how agents behave
00:13:14.919 --> 00:13:17.080
when they are allowed to autonomously browse
00:13:17.080 --> 00:13:20.320
databases or the web. It analyzes the general
00:13:20.320 --> 00:13:22.379
trade -offs between the autonomy of the agent,
00:13:22.519 --> 00:13:24.980
like how many steps it takes, how many dead ends
00:13:24.980 --> 00:13:27.799
it hits, and the resulting cloud compute bill.
00:13:28.019 --> 00:13:30.200
Which really clarifies the differentiator for
00:13:30.200 --> 00:13:32.639
Zhu's paper today. Beyond semantic similarity
00:13:32.639 --> 00:13:34.960
is looking at the messy, multi -step wandering
00:13:34.960 --> 00:13:37.690
of autonomous agents. Right. And the agentic
00:13:37.690 --> 00:13:39.970
RAG evaluation paper is about offline grading.
00:13:40.149 --> 00:13:42.669
What we are analyzing today is an in -flight
00:13:42.669 --> 00:13:45.289
router for a single real -time QA turn. Yes.
00:13:45.350 --> 00:13:47.629
It isn't about building the system or letting
00:13:47.629 --> 00:13:51.210
an agent browse the web. It is a strict moment
00:13:51.210 --> 00:13:53.570
-by -moment governor sitting inside a production
00:13:53.570 --> 00:13:56.929
environment. It's deciding in milliseconds whether
00:13:56.929 --> 00:13:58.929
a specific question gets the cheap treatment
00:13:58.929 --> 00:14:02.169
or the premium treatment. It's highly operational.
00:14:03.309 --> 00:14:05.509
It's the difference between designing a fuel
00:14:05.509 --> 00:14:08.110
-efficient engine. And actively deciding when
00:14:08.110 --> 00:14:10.610
to hit the gas pedal while you're already driving.
00:14:10.610 --> 00:14:14.129
That's a great analogy. It's totally live. So
00:14:14.129 --> 00:14:15.970
I want to carve out some space here for some
00:14:15.970 --> 00:14:19.049
critical analysis. For everyone listening, we
00:14:19.049 --> 00:14:21.690
are stepping outside the explicit claims of the
00:14:21.690 --> 00:14:24.269
paper now to offer some helpful opinions and
00:14:24.269 --> 00:14:26.250
caveats. Yeah, it's important to look at the
00:14:26.250 --> 00:14:28.470
gaps. We need to talk about the limitations,
00:14:28.649 --> 00:14:30.710
starting with the sample size. You mentioned
00:14:30.710 --> 00:14:33.809
earlier they tested this on 150 questions from
00:14:33.809 --> 00:14:37.029
FinanceBench. I did. I mean... A mid -sized hedge
00:14:37.029 --> 00:14:39.509
fund probably generates 150 questions before
00:14:39.509 --> 00:14:41.889
the morning stand -up. Yeah, the sample size
00:14:41.889 --> 00:14:44.850
is microscopic when you project it against enterprise
00:14:44.850 --> 00:14:47.990
reality. While finance bench is a rigorous, peer
00:14:47.990 --> 00:14:50.950
-reviewed standard, 150 data points simply do
00:14:50.950 --> 00:14:54.190
not provide enough statistical density to guarantee
00:14:54.190 --> 00:14:57.029
that their specific benefit prediction model
00:14:57.029 --> 00:15:00.070
will generalize. Right. If you point this routing
00:15:00.070 --> 00:15:03.519
system at a different sector, say... messy clinical
00:15:03.519 --> 00:15:06.419
trial data or dense legal contracts, there is
00:15:06.419 --> 00:15:08.600
no guarantee the proxy model will even achieve
00:15:08.600 --> 00:15:11.840
that 0 .60 AUROC. It could be even worse. And
00:15:11.840 --> 00:15:14.139
we really have to look closely at the risk profile
00:15:14.139 --> 00:15:17.840
of that 0 .60 AUROC in a live environment. Yeah,
00:15:17.840 --> 00:15:21.519
we celebrated the 46 .6 % cost savings. But if
00:15:21.519 --> 00:15:23.919
the predictor is only slightly better than chance
00:15:23.919 --> 00:15:27.159
at identifying those persuadable queries, you
00:15:27.159 --> 00:15:29.080
are inevitably going to throttle the context
00:15:29.080 --> 00:15:31.559
on questions that desperately needed escalation.
00:15:31.639 --> 00:15:33.679
Exactly. The cost savings are mathematically
00:15:33.679 --> 00:15:36.519
proven, but the risk to accuracy is not mitigated.
00:15:36.600 --> 00:15:38.480
And if you are building mission -critical software
00:15:38.480 --> 00:15:41.559
where a hallucinated number on an earnings summary
00:15:41.559 --> 00:15:44.059
could trigger a bad trade or a compliance violation,
00:15:44.480 --> 00:15:47.320
throttling contacts based on a 60 % probability
00:15:47.320 --> 00:15:49.960
metric is dangerous. Very dangerous. You are
00:15:49.960 --> 00:15:52.940
effectively accepting a certain baseline of hallucinations
00:15:52.940 --> 00:15:55.899
in exchange for keeping your AWS bill lower.
00:15:56.059 --> 00:15:58.500
So what does this all mean? It forces a pretty
00:15:58.500 --> 00:16:00.980
tough conversation for the engineering managers
00:16:00.980 --> 00:16:03.460
and product leads tasked with scaling these systems,
00:16:03.519 --> 00:16:05.940
doesn't it? It absolutely does. Brute forcing
00:16:05.940 --> 00:16:08.639
the context window works perfectly fine until
00:16:08.639 --> 00:16:12.240
the CFO actually sees the invoice. Right. But
00:16:12.240 --> 00:16:14.860
implementing an aggressive, imperfect triage
00:16:14.860 --> 00:16:17.720
system could compromise the trust of the analysts
00:16:17.720 --> 00:16:20.240
using the tool. It really comes down to understanding
00:16:20.240 --> 00:16:22.970
the nature of your specific deployment. If your
00:16:22.970 --> 00:16:26.730
system is just summarizing internal company newsletters,
00:16:26.870 --> 00:16:29.590
aggressive uplift targeting is brilliant. Oh,
00:16:29.629 --> 00:16:32.529
for sure. Go for it. But if it's performing real
00:16:32.529 --> 00:16:35.049
-time compliance checks on outbound trades, you
00:16:35.049 --> 00:16:37.110
might just have to accept the brute force token
00:16:37.110 --> 00:16:39.769
burn until the industry invents a better benefit
00:16:39.769 --> 00:16:42.169
predictor. You can't cut corners on compliance.
00:16:42.509 --> 00:16:45.330
You just have to pay the toll. Okay, let's pull
00:16:45.330 --> 00:16:47.850
all these threads together. We started by looking
00:16:47.850 --> 00:16:51.000
at the hidden token burn of RAG escalation. The
00:16:51.000 --> 00:16:53.620
danger of the AI reading too many expensive documents
00:16:53.620 --> 00:16:56.220
for every single query. Right. And we broke down
00:16:56.220 --> 00:16:59.320
Zoo and team's gain per token metric, dividing
00:16:59.320 --> 00:17:01.720
the predicted benefit by the highly predictable
00:17:01.720 --> 00:17:05.859
cost. The ROI calculation. Right. We saw that
00:17:05.859 --> 00:17:10.220
even using a flawed 0 .60 AUROC proxy model to
00:17:10.220 --> 00:17:13.059
guess that benefit, the system could slash token
00:17:13.059 --> 00:17:16.339
spend by nearly 47 % while actually slightly
00:17:16.339 --> 00:17:19.519
boosting the overall quality on a rigorous financial
00:17:19.519 --> 00:17:22.109
benchmark. And we place this inside the broader
00:17:22.109 --> 00:17:25.289
movement toward AI resource allocation, recognizing
00:17:25.289 --> 00:17:28.069
that predicting a language model's latent reasoning
00:17:28.069 --> 00:17:31.450
capability before it executes is really the next
00:17:31.450 --> 00:17:33.930
major bottleneck in computer science. The era
00:17:33.930 --> 00:17:36.349
of treating compute as infinite is definitely
00:17:36.349 --> 00:17:39.730
over. It is. And this raises an important question
00:17:39.730 --> 00:17:41.690
for everyone listening to consider as you interact
00:17:41.690 --> 00:17:44.210
with these systems going forward. Okay, let's
00:17:44.210 --> 00:17:47.470
hear it. If the computational cost of AI is highly
00:17:47.470 --> 00:17:50.200
predictable, but the actual intellectual benefit
00:17:50.200 --> 00:17:52.339
of that compute remains largely unpredictable
00:17:52.339 --> 00:17:55.579
until after the fact. How does that change the
00:17:55.579 --> 00:17:58.140
way businesses should contract for AI services?
00:17:58.559 --> 00:18:01.579
That's a fascinating angle, because the current
00:18:01.579 --> 00:18:03.980
model forces the user to absorb all the risk.
00:18:04.059 --> 00:18:06.539
You pay for the tokens the AI reads, whether
00:18:06.539 --> 00:18:09.599
the answer is brilliant or just a complete hallucination.
00:18:10.140 --> 00:18:13.480
Exactly. Will we see a shift toward paying strictly
00:18:13.480 --> 00:18:16.170
for the uplift? where the provider only gets
00:18:16.170 --> 00:18:18.890
paid if the extra compute actually delivers a
00:18:18.890 --> 00:18:20.910
mathematically verifiable improvement to the
00:18:20.910 --> 00:18:23.990
final output. Wow. It would completely upend
00:18:23.990 --> 00:18:26.509
the sauce pricing model. You'd stop paying for
00:18:26.509 --> 00:18:28.490
the hours your digital analyst spends in the
00:18:28.490 --> 00:18:31.150
library and start paying purely for the insights
00:18:31.150 --> 00:18:33.930
they deliver. A shift from renting compute to
00:18:33.930 --> 00:18:37.009
purchasing outcomes. I love that framing. Well,
00:18:37.069 --> 00:18:39.190
thank you for joining us on this deep dive, everyone.
00:18:39.700 --> 00:18:41.400
Keep questioning the architecture behind the
00:18:41.400 --> 00:18:43.920
interfaces you use. Keep an eye on your own token
00:18:43.920 --> 00:18:45.720
budgets, and we will catch you next time.
