Artificial Analysis: Independent LLM Evals as a Service — with George Cameron and Micah-Hill Smith

WEBVTT
1
00:00:00.100 --> 00:00:17.700
[ on-hold music] This is kind of a full circle moment for us in a way, um- Yeah... 'cause the, like, first time Artificial Analysis got mentioned on a podcast was you and Alessio on LaneSpace. Amazing.
2
00:00:17.960 --> 00:00:29.260
Which was January 2024. I don't even re- I, I don't even remember doing that. [laughs] But yeah. It was, it was very influential to me. Um, yeah, I'm looking at AI News for Jan 17 or Jan 16, 2024.
3
00:00:29.820 --> 00:00:40.820
Uh, I said, "This gem of a models and hosts comparison site was just launched." And, uh, and then I put in a few screenshots. And I said, um, "It's an independent third party.
4
00:00:41.000 --> 00:00:48.780
It clearly outlines the quality versus throughput trade-off." Mm-hmm. "And it breaks out by model and hosting provider." Uh, I did give you shit for missing Fireworks.
5
00:00:48.780 --> 00:00:57.920
[laughs] And, um [laughs] uh, how do you have a model benchmarking thing without Fireworks? But you had Together, you had Perplexity, and, uh, I think we just started chatting there.
6
00:00:58.400 --> 00:01:04.319
Welcome, George and Micah, uh, to LaneSpace. [on-hold music] You've-- I've been following your progress. Congrats on, uh, an amazing year.
7
00:01:04.480 --> 00:01:14.380
You guys have really come together to be the presumptive new gardener of AI, right, which is- Thanks... something that, uh- Yep. But you can't, uh, pay us for better results. Yeah, exactly. [laughs] Very important.
8
00:01:15.220 --> 00:01:25.680
Start off- Kicking off straight into it. Let's go. Start off with a spicy take. Okay. How do I pay you? [laughs] And let's get right into that. How do you make money? [laughs] Um, well, very happy to talk about that.
9
00:01:26.060 --> 00:01:35.400
So it's been a, like, big journey the last couple of years. Artificial Analysis is gonna be two years old in January 2026- Mm-hmm... which is pretty soon now. We
10
00:01:36.320 --> 00:01:48.140
first run, like, the website for free, obviously, and give away a ton of data to help developers and companies navigate AI and make decisions about models, providers, technologies across the AI stack for building stuff.
11
00:01:49.000 --> 00:02:01.760
We're very committed to doing that, intend to keep doing that. We have, along the way, built a business that is working out pretty sustainably. We've got just over 20 people now and two main customer groups.
12
00:02:02.160 --> 00:02:12.200
So we want to be who enterprise look to for data and insights on AI, so we want to help them with their decisions about models and technologies for building stuff.
13
00:02:12.820 --> 00:02:24.820
And then on the other side, we do private benchmarking for companies throughout the AI stack who build AI stuff. So no one pays to be on the website.
14
00:02:25.280 --> 00:02:32.720
We've been very clear about that from the very start because there's no use doing what we do unless it's independent AI benchmarking. Yeah.
15
00:02:32.840 --> 00:02:38.380
Um, but turns out a bunch of our stuff can be pretty useful to companies building AI stuff.
16
00:02:39.060 --> 00:02:51.200
And is it like, "I'm a Fortune 500, I need advisors on objective analysis," and I call you guys, and you pull up a custom report for me, you come into my office and give me a workshop?
17
00:02:51.240 --> 00:02:53.380
What, what, what kind of engagement is that?
18
00:02:53.780 --> 00:03:04.580
So we have a benchmarking insight subscription which looks like standardized reports that cover key topics or key challenges enterprises face when looking to understand AI and choose between all the technologies.
19
00:03:04.980 --> 00:03:25.660
And so, for instance, one of the report is a model deployment report, how to think about choosing between serverless inference, managed deployment solutions, or leasing chips and running inference yourself is, is, is an example kind of decision that big enterprises, uh, face and it's hard to, hard to reason through.
20
00:03:25.700 --> 00:03:34.940
Like, this AI stuff is, is really new to, to everybody. And so we try and help with our reports and insight subscription companies navigate that.
21
00:03:35.560 --> 00:03:45.520
We also do custom private benchmarking and, um, so that's very different from the public benchmarking, um, that we publicize and there's no commercial model around that.
22
00:03:45.560 --> 00:03:52.700
For b- for private benchmarking, we'll at times create benchmarks, run benchmarks to specs that enterprises want.
23
00:03:53.200 --> 00:04:09.140
And we'll also do that sometimes for AI companies who have built things, and we help them understand what they've built with private benchmarking, um, you know, through the expertise mainly that we've developed through trying to support everybody, uh, publicly, uh, with our public benchmarks.
24
00:04:09.460 --> 00:04:18.060
Yeah. Uh, let's talk about tech stack, uh, behind that. But okay, I'm gonna rewind, uh, all the way to when you guys started this project. Uh, you were all, all the way in Sydney?
25
00:04:18.420 --> 00:04:25.420
Where- Yeah, Sy- well, Sydney, Australia for me. George was in SF, but- Yeah... he's Australian, but had moved here already. Yeah. And, um, I remember I had the, the Zoom call with you.
26
00:04:25.520 --> 00:04:33.300
Um, what was the impetus for starting Artificial Analysis in the first place? You know, you started with, uh, public benchmarks, and so let's, let's start there, and we'll go to the private stuff. Yeah.
27
00:04:33.340 --> 00:04:44.860
Why don't we even go back a little bit to, like, why we, you know, thought that it was, that it was needed? Yeah. The story kind of begins, like, in 2022, 2023.
28
00:04:44.900 --> 00:04:59.500
Like, both George and I have been into AI stuff for quite a while. In 2023 specifically, I was trying to build a legal AI research assistant. So it actually worked pretty well for, for, for its era, I would say.
29
00:05:00.100 --> 00:05:09.100
But was finding that the more you go into building something using LLMs, the more each bit of what you're doing ends up being a benchmarking problem.
30
00:05:09.580 --> 00:05:18.220
So had, like, this multi-stage algorithm thing, trying to figure out what the minimum viable model for each bit was, trying to optimize every bit of it.
31
00:05:18.640 --> 00:05:24.560
As you build that out, right, like you're trying to think about accuracy, a bunch of other metrics, and performance and cost.
32
00:05:25.120 --> 00:05:33.680
And mostly just no one was doing anything to independently evaluate all the models and certainly not to look at the trade-offs for speed and cost.
33
00:05:34.120 --> 00:05:45.860
So we basically set out just to build a thing that developers could look at to see the trade-offs between all of those things measured independently across all the models and providers.
34
00:05:46.660 --> 00:05:54.040
Honestly, it was probably meant to be a side project when we first started doing it. [laughs] Like, we didn't, like, get together and say, like, "Hey," like, "we're gonna, um, stop working- Yeah...
35
00:05:54.060 --> 00:05:59.840
on other stuff," and, like, "this is gonna be our main thing." When I first called you, I think, uh, you, you hadn't decided on starting a company yet. That's actually true. No.
36
00:05:59.900 --> 00:06:05.980
I don't even think we'd paused, like, like George hadn't quit his job. I hadn't quit working on my legal AI thing. Like, it, it was genuinely a side project. Yeah. Yeah.
37
00:06:06.200 --> 00:06:17.920
Um- We built it because we needed it as people building in the space And thought, oh, other people might find it useful too, so we'll buy a domain and link it to the Vercel deployment that, that, that we had.
38
00:06:18.020 --> 00:06:24.780
[laughs] Yeah. We, we, we... and, and, and, and, and, and tweet about it, and but very quickly it started getting attention.
39
00:06:24.840 --> 00:06:31.320
Thank you, swyx, for, I think, doing an initial retweet and spotlighting it there, this project that, that we released.
40
00:06:31.740 --> 00:06:45.660
And then very quickly, though, it, it was useful to others, but very quickly it became more useful as the number of models, uh, released accelerated. Uh, we had Mixtral 8 times 7B, and it was a key- Oh, that's a fun one.
41
00:06:45.880 --> 00:07:01.660
Yeah, like a, a, a open source model that really changed the landscape and opened up people's eyes to other serverless inference providers and thinking about speed, thinking about cost, and so it became more useful, um, quite quickly.
42
00:07:01.940 --> 00:07:11.960
Yeah. What I love talking to, to people like you who sit across the ecosystem is, well, I have theories about what people want, but you have data, and that's, uh, obviously more, more relevant.
43
00:07:12.320 --> 00:07:15.020
Uh, but I wanna stay on the origin story a little bit more.
44
00:07:15.440 --> 00:07:24.920
Um, when you started out, I would say I think the, the status quo at the time was every paper would come out, and they would report their numbers versus competitor numbers, and that's basically it.
45
00:07:25.000 --> 00:07:27.740
And, uh, I remember I did the legwork.
46
00:07:27.820 --> 00:07:37.500
I think, I think everyone has some version of Excel sheet or s- uh, Google sheet where you just, like, copy and paste the numbers from every paper and just post it up there, and then sometimes they don't line up because they're independently run.
47
00:07:37.880 --> 00:07:42.020
And so [laughs] your numbers are gonna look, uh, better, uh, than the...
48
00:07:42.200 --> 00:07:49.260
or, or your reproductions of other people's numbers are gonna look worse 'cause you don't hold their n- models correctly or whatever, whatever the, the excuse is.
49
00:07:49.720 --> 00:07:58.180
I think then Stanford Helm, Percy Liang's project, would also have some, some of these numbers, and I don't know if there's any other source that you can cite.
50
00:07:58.660 --> 00:08:07.540
The way that if I were to start artificial news at the same time you guys started, I would have used EleutherAI's, um, eval framework, uh, harness. Yep. Yep. That was some cool stuff.
51
00:08:07.880 --> 00:08:20.380
Um, at the end of the day, right, running these evals, it's like if it's a simple Q&A eval, all you're doing is asking a list of questions and checking if the answers are right, which shouldn't be that crazy.
52
00:08:20.380 --> 00:08:24.900
[laughs] But it turns out there are an enormous number of things that you've got to control for.
53
00:08:25.260 --> 00:08:38.380
And I mean, back when we started the website, like, one of the reasons why we were-- we realized that we had to run the evals ourselves and couldn't just take, um, results from the labs, was just that they would all prompt the models differently.
54
00:08:38.760 --> 00:08:46.520
And when you're competing over a few points, then you can, um, pretty easily get- You can put the answer into the model [laughs] and into the prompts. Yeah. I, I mean that, that in the extreme.
55
00:08:46.520 --> 00:09:00.740
[laughs] And, like, you get crazy cases like back when, um, Google did Gemini 1.0 Ultra and needed a number that would say it was better than GPT-4 and, um, like, constructed, um, I think never published, like, chain of thought examples, 32 of them in every topic in MMLU to- Right...
56
00:09:00.800 --> 00:09:08.240
run it to get the score. Like, there are so many things that you, that you could do. They never shipped Ultra, right? Um. That's the, that's the one that never made it out. Not, not widely publicly, no. Yeah, yeah, yeah.
57
00:09:08.280 --> 00:09:16.200
I mean, it, it, I'm sure it existed. But yeah, so we were pretty sure that we needed to run them ourselves and just run them in the same way across all the models. Yeah.
58
00:09:16.260 --> 00:09:25.780
And we were, we were also dead certain from the start that you couldn't look at those in isolation. You needed to look at them alongside the cost and performance stuff. Yeah. Okay, a couple c- technical questions.
59
00:09:26.300 --> 00:09:34.980
I mean, so obviously I also thought about this, and I didn't do it because cost. Yep. Did you, did you not worry about cost? Were you funded already? Clearly not, but, you know. No, we, we...
60
00:09:35.020 --> 00:09:41.700
well, we definitely weren't at the start. So, like, um, I mean, we were paying for it personally at the start. There's, there's not money [laughs]. So, well, the numbers weren't- Yeah...
61
00:09:41.760 --> 00:09:47.820
nearly as bad a couple of years ago. So we, like, certainly, um, like, incurred
62
00:09:48.900 --> 00:09:58.460
s-some costs, but we were probably in the order of, like, hundreds of dollars of spend across all the benchmarking that we were doing- Okay, so nothing... at the start. Yeah, yeah. It was, like, kind of fine. Yeah, yeah.
63
00:09:58.960 --> 00:10:06.540
These days, that's gone up an enormous amount for a bunch of reasons that we can talk about. Um, but yeah, it wa- it wasn't that bad 'cause if...
64
00:10:06.700 --> 00:10:15.820
you can also remember that, like, the number of models that we were dealing with was hardly any, and the complexity of the stuff that we wanted to do to evaluate them was a lot less.
65
00:10:15.860 --> 00:10:27.980
Like, we were just asking some Q&A-type questions. And then one specific thing was, for a lot of evals initially, we were just, like, sampling an answer directly without letting the models think.
66
00:10:28.020 --> 00:10:34.000
We weren't even doing chain of thought stuff initially, and that was the most useful way to get some results initially. Yeah.
67
00:10:34.040 --> 00:10:39.160
And so for, if, for people who haven't done this work, literally parsing the responses is a whole thing, right? Yeah.
68
00:10:39.180 --> 00:10:54.420
Like, because sometimes the models, the models can answer any way they f-feel fit, and sometimes they actually do have the right answer, but they just return the wrong format, and they will get a zero for that unless you f- work it into your parser, and that- Yeah, depending how you grade it...
69
00:10:54.420 --> 00:11:02.020
involves more work. Yeah. And so there, I mean, but there's an open question whether you should give it points for not f-following your instructions on the format, so. It depends what you're looking at, right? Yeah.
70
00:11:02.020 --> 00:11:18.200
[laughs] 'Cause you can, if you're trying to see whether or not it can solve a particular type of reasoning problem, and you don't want to test it on its ability to do answer formatting at the same time, then you might want to use an LLM as answer extractor approach to make sure that you get the answer out no matter how it answered.
71
00:11:18.560 --> 00:11:21.400
But these days, it's mostly less of a problem.
72
00:11:21.440 --> 00:11:29.420
Like, if you instruct a model and give it examples of what the answer should look like, it can get the answers, um, in your format, and then you can do, like, a simple regex. Yeah, yeah.
73
00:11:29.640 --> 00:11:38.220
And then, uh, there's other questions around, I guess sometimes if you have a multiple choice question, sometimes there's a bias towards the first answer, so you have to randomize the responses. Yep.
74
00:11:38.260 --> 00:11:46.960
All these nuances, you al- like- Totally... once you dig into benchmarks, you're like, "I don't know how anyone believes the numbers on all these things." [laughs] 'Cause it's, it's so, it's so dark magic.
75
00:11:47.120 --> 00:11:52.140
You've also, yeah, got, like, the, um, uh, different degrees of variance in different benchmarks, right? Yeah.
76
00:11:52.180 --> 00:12:06.860
So if, if you, if you run four-question multi-choice on a modern reasoning model at the temperature suggested by the labs for their own models, the variance that you can see on a four-question multi-choice eval is pretty enormous if you only do a single run of it and it has a small number of questions es-especially.
77
00:12:07.380 --> 00:12:32.280
So, like, one of the things that we do is run an enormous number of all of our evals when we're developing new ones and doing upgrades to our intelligence index to bring in new things, so that we can dial in the right number of repeats, so that we can get to the 95% confident- confidence intervals that we're comfortable with, so that when we pull that together, we can be confident in intelligence index to at least as tight as, like, a plus or minus one at a 95% confidence.
78
00:12:32.824 --> 00:12:38.984
Yeah. And again, that just adds a straight multiple to the cost and [laughs] Oh, just, yeah. Yeah, right? So that's one of many reasons- [laughs]...
79
00:12:39.324 --> 00:12:49.224
that cost has gone up, uh, a lot more than linearly over the last couple of years. [laughs] We, um, we report a cost to run the artificial analysis intelligence index on our website. Mm.
80
00:12:49.304 --> 00:12:57.704
And currently that's assuming one repeat- Okay. Yeah... um, in terms of how we re- report it because we wanna reflect a, a bit about the weighting of the index.
81
00:12:57.804 --> 00:13:04.743
But, um, but our cost is actually a lot higher than what we, uh, report there because of the repeats. [laughs] Yeah, yeah, yeah.
82
00:13:04.784 --> 00:13:13.984
And probably this is true, but just checking, the, you don't have any special deals with the labs, they don't, they don't discount it, you j- you just pay out of pocket or out of your, your sort of customer funds?
83
00:13:14.864 --> 00:13:23.044
Uh, th- th- there is, there is a mix. Okay. So we- So, so the, the, the issue is that sometimes they may give you a special endpoint, which- Ah, 100%. Yeah, yeah, yeah. Okay. Ex- exactly.
84
00:13:23.344 --> 00:13:31.544
So we, uh, um, laser focus, like on everything we do on having the best independent metrics and making sure that no one can manipulate them in any way.
85
00:13:32.044 --> 00:13:50.884
There are quite a lot of processes we've developed over the last couple of years to make that true for like, with the one you bring up, like right here, of the fact that if we're working with a lab, if they're giving us a private endpoint to evaluate a model, that it is totally possible that what's sitting behind that black box is not the same as they serve on a public endpoint.
86
00:13:51.264 --> 00:13:52.344
We're very aware of that.
87
00:13:52.724 --> 00:14:07.844
We have what we call a mystery shopper policy, and so, and we're totally transparent with all the labs we work with about this, that we will register accounts not on our own domain and run both intelligence evals and performance benchmarks- Yeah, that's the job...
88
00:14:07.864 --> 00:14:10.004
without them being unidentified.
89
00:14:10.384 --> 00:14:23.064
And no one's ever had a problem with that, 'cause like a thing that turns out to actually be quite a good factor in the industry is that they all want to believe that none of their competitors could manipulate what we're doing either.
90
00:14:23.584 --> 00:14:31.484
That's true. I never thought about that. I've been in a database data industry prior, and there's a lot of shenanigans around benchmarkings, right? Yeah. So I'm just kind of going through the mental laundry list.
91
00:14:31.704 --> 00:14:35.984
Did I miss anything else in, in that, in this category of shenanigans? Oh, potential shenanigans.
92
00:14:35.984 --> 00:14:43.864
[laughs] I mean, okay, the, the, the, the biggest one like that I'll bring up like is more of a conceptual one actually than like direct shenanigans.
93
00:14:44.224 --> 00:14:57.644
It's that the things that get measured become things that get targeted by labs- Good workflow... what they're trying to build, right? Exactly. So that doesn't mean anything that we should really call shenanigans.
94
00:14:57.684 --> 00:15:14.464
Like I'm not talking about training on test set, but if you know that you're gonna be great in another particular thing, if you're a researcher, there are a whole bunch of things that you can do to try to get better at that thing that preferably are going to be helpful for a wider range of how actual users wanna use the thing that you're building, but will not necessarily do that.
95
00:15:14.784 --> 00:15:20.444
So for instance, the models are exceptional now at answering competition maths problems.
96
00:15:20.664 --> 00:15:31.204
[laughs] There is some, uh, relevance of that type of reasoning and that type of work, um, to like how we might use modern coding agents and stuff. Um, but it's clearly not one-to-one.
97
00:15:31.524 --> 00:15:44.564
So the thing that we have to be aware of is that once an eval becomes the thing that everyone's looking at, the scores can get better on it without there being a reflection of overall generalized intelligence of these models getting better.
98
00:15:44.944 --> 00:15:48.784
That has been true for the last couple of years, it'll be true for the next couple of years.
99
00:15:49.084 --> 00:15:59.064
There's no silver bullet to defeat that other than building new stuff to stay relevant and measure the capabilities that matter most to real users. Yeah.
100
00:15:59.084 --> 00:16:02.424
And we'll, we'll cover, we'll cover some of the new stuff that you guys are building as well.
101
00:16:02.884 --> 00:16:14.424
Uh, which is cool, like you, you used to just run other people's evals, but now you're coming up with your own, and I think obviously that is a, a necessary path once you're at the frontier, you've exhausted all the existing well-known ones.
102
00:16:14.904 --> 00:16:24.544
I think the next point in history that I have for you is AI Grant, that you guys decided to, to join- Yeah. Yeah, that's right... and, and move here. What's, what was it like? I think you're, you were in like batch two?
103
00:16:25.144 --> 00:16:36.884
Batch four. Batch four? Okay. I mean, it was great. Nat and Daniel are obviously great, and it's a really cool group of companies that we were in AI Grant alongside. It was really great to get Nat and Daniel on board.
104
00:16:37.144 --> 00:16:46.904
Obviously, they've done a whole lot of great work in the space with a lot of leading companies, and we're extremely aligned with the mission of what we were trying to do.
105
00:16:47.244 --> 00:16:55.664
Like, we're not quite typical of like a lot of the other AI startups that they've invested in, and they were very much here for the mission of what we wanna do.
106
00:16:56.224 --> 00:17:03.924
Did they s- say any advice that really affected you in some way? Or like, were one of the events very impactful? That's an interesting question.
107
00:17:04.204 --> 00:17:13.264
I, I mean, I remember fondly a bunch of the, um, speakers who came and did fireside chats at AI Grant. Which is also like a crazy list. Yeah. Oh, totally. Yeah. Yeah, yeah. Yeah.
108
00:17:13.324 --> 00:17:29.324
I, I, I-- there was something about, um, you know, speaking to Nat and Daniel about the challenges of, of, of working through a startup and just working through the questions that don't have like clear answers and how to, to work through those kind of methodically, um, and just like work through the hard decisions.
109
00:17:29.824 --> 00:17:39.484
Uh, and they've been great mentors- Yeah... to, to us as we, as we've built Artificial Analysis. Another benefit for us was that other companies in the batch and other companies in AI Grant- [laughs]...
110
00:17:39.844 --> 00:17:44.384
are pushing the capabilities of, of, of what AI can do at this time.
111
00:17:44.764 --> 00:17:59.564
And so being in contact with them, making sure that Artificial Analysis is, is useful to them, has been fantastic for, for supporting us in working out how, how should we build out Artificial Analysis to continue to being, uh, useful to those like, you know, building on AI.
112
00:17:59.964 --> 00:18:10.384
I think to some extent I'm mixed opinion on that one because to some extent, your target audience is not people in AI Grants who are obviously at the frontier. Yeah. You disagree? To, to some, to some extent.
113
00:18:10.424 --> 00:18:29.684
But then- [laughs] So a lot of what the AI Grant companies are doing is, um, taking capabilities coming out of the labs and trying to push the limits of what they can do across the entire stack for building great applications, which actually makes some of them pretty archetypical power users of Artificial Analysis.
114
00:18:30.044 --> 00:18:36.304
Some of the people with the strongest opinions about what we're doing well and what we're not doing well and what they, uh, wanna see next from us.
115
00:18:36.744 --> 00:18:43.364
'Cause when you're building any kind of AI application now, chances are you're using a whole bunch of different models.
116
00:18:43.764 --> 00:18:50.564
You're maybe switching reasonably frequently for different models in different parts of your application to optimize what...
117
00:18:50.904 --> 00:18:56.804
You're able to, to do with them at an accuracy level and to get better speed and cost characteristics.
118
00:18:57.184 --> 00:19:07.404
So for many of them, you know, they're, um, like, not commercial customers of ours, like we don't charge for all our data on the website, but they are absolutely some of our power users.
119
00:19:08.024 --> 00:19:15.444
So let's talk about just the, the, the evals as well, right? Like, you see, you start out from the g- the general, like, MMLU and, and GPQA stuff. Um, what's next?
120
00:19:15.484 --> 00:19:22.444
How do you, how do you sort of build up to, um, the overall index, what was in V1, and how did you evolve it? Okay.
121
00:19:22.804 --> 00:19:39.584
So first just, like, background, like we're talking about the artificial analysis intelligence index, which is our synthesis metric that we pull together currently from 10 different eval data sets to give what we're pretty confident is the best single number to look at for how smart the models are.
122
00:19:40.244 --> 00:19:48.484
Obviously doesn't tell the whole story. That's why we publish the whole website of all the charts to dive into every part of it and look at the trade-offs, but best single number.
123
00:19:49.104 --> 00:19:59.204
So right now it's gotten a, a bunch of Q&A type data sets that have been very important to the industry, like a couple that you just mentioned. It's also got a couple of agentic data sets.
124
00:19:59.464 --> 00:20:03.324
It's got our own long context reasoning data set and some other use case focused stuff.
125
00:20:03.804 --> 00:20:17.104
As time goes on, the things that we're most interested in that are gonna be important to the capabilities that are becoming more important for AI, what developers are caring about, are gonna be first around agentic capabilities.
126
00:20:17.404 --> 00:20:26.744
So surprise, surprise, we're all loving our coding agents, and how the model's gonna perform like that and then do similar things for different types of work are really important to us.
127
00:20:27.084 --> 00:20:31.924
The linking to use cases, to economically valuable use cases are extremely important to us.
128
00:20:32.624 --> 00:20:46.224
And then we've got some of these, um, things that the models still struggle with, like working really well over long contexts that are not gonna go away as specific capabilities and use cases that we need to keep evaluating.
129
00:20:46.624 --> 00:20:55.844
Mm-hmm. And but I guess one thing I was driving was, like, the V1 versus the V2 and how- Oh, yeah... that evolves over time. Like our, like how we've changed the index to- Yes... where we are.
130
00:20:56.004 --> 00:21:05.084
Yeah, I think that reflects on, well, the change in the industry, right? Yep. So that's a nice way to tell that story. Well, V1 would be completely saturated right now- [laughs]...
131
00:21:05.124 --> 00:21:19.684
by almost every model coming out because doing things like writing the Python functions in HumanEval is now pretty trivial. It's easy to forget actually, I think, how much progress has been made in the last two years.
132
00:21:19.944 --> 00:21:32.444
Like, we, we obviously play the game constantly of, like, the today's version versus last week's version and the week before and all of the small changes and the horse race between the current frontier and the who has the best, like, smaller than 10 B model, like, right now this week, right?
133
00:21:32.844 --> 00:21:38.484
And that's very important to a lot of developers and people in, especially in this particular city of San Francisco.
134
00:21:39.024 --> 00:21:55.484
But when you zoom out, a couple of years ago, literally most of what we were doing to evaluate the models then would all be 100% solved by even pretty small models today, and that's been one of the key things, by the way, that's driven down the cost of, um, intelligence at every tier of intelligence we can talk about more in a bit.
135
00:21:56.044 --> 00:22:12.524
So V1, V2, V3, we made things harder. We covered a wider range of use cases, and we tried to get closer to things developers care about as opposed to, like, just the Q&A type stuff that MMLU and GPQA represented. Yeah.
136
00:22:12.584 --> 00:22:22.804
I don't know if you have anything to add there. Uh, or we could just go right into showing people the benchmark and, like, clicking around and ask- asking questions about it. [laughs] Yeah. Let's do it. Okay.
137
00:22:23.104 --> 00:22:31.804
This will be a pretty good way to chat about a few of the new things we've launched recently. Yeah. And I think a little bit about the direction that we wanna take it, and we wanna push benchmarking.
138
00:22:32.764 --> 00:22:58.144
Currently, the intelligence index and, and, and evals focus a lot on kind of raw intelligence, but we kind of want to diversify how we think about intelligence and, uh, we can, we can talk about it, but kind of new evals that we've kind of built, um, and partnered on focus on, uh, topics like hallucination, and we've got a lot of topics that I think are not covered by the current eval set that should be.
139
00:22:58.484 --> 00:23:03.324
Mm-hmm. Uh, and, and so we want to bring that forth. But before we get into that, um- And so, so for listeners- Mm-hmm...
140
00:23:03.644 --> 00:23:15.784
just as a timestamp, right now on number one is Gemini 3 Pro High, then followed by Claude Opus at 70, uh, just 5.1 High, you don't have 5.2 yet, uh, and Kimi k2 Thinking. Wow, still hanging in there.
141
00:23:16.284 --> 00:23:23.404
So those, those are the top four. That will date this podcast quickly. [laughs] Yeah, yeah. I mean, I, I like, I love it. I love it. No, no, 100%. It's the moment in time.
142
00:23:23.404 --> 00:23:32.664
Look back this time next year and go, "How cute." Yep. Like [laughs] Totally. A quick view of that is, okay, there's a lot. I love this chart. This is, this is such a favorite, right? Yeah.
143
00:23:32.764 --> 00:23:41.804
Um, and almost every talk that George or I give at conferences and stuff, like, we always put this one up first to just talk about situating where we are in this moment in history.
144
00:23:42.244 --> 00:23:49.884
This, I think, is the, the visual version of what I was saying before about the zooming out and remembering how much progress there's been.
145
00:23:50.204 --> 00:24:02.024
If we go back to just over a year ago, before o1, before Claude Sonnet 3.5, we didn't have reasoning models or coding agents as a thing, and the game was very, very different.
146
00:24:02.224 --> 00:24:10.504
If we go back even a little bit before then, we're in the era where when you look at this chart, like, OpenAI was un- untouchable for well over a year.
147
00:24:10.924 --> 00:24:20.064
And I mean, you would remember that time period well of, like, there being very open questions about whether or not AI was going to be competitive. Mm-hmm. Like, full stop.
148
00:24:20.384 --> 00:24:29.204
Whether or not OpenAI would just run away with it, whether we would have the, a few frontier labs and no one else would really be able to do anything other than consume their APIs.
149
00:24:29.764 --> 00:24:42.524
I am quite happy overall that the world that we have ended up in is one where- Multi-model. Yeah... absolutely, and strictly more competitive every quarter over the last two years. Yeah. This year has been insane. Yeah.
150
00:24:42.764 --> 00:24:48.524
You can see it. Uh, this chart with everything added is hard, hard to read currently. Um- Yeah...
151
00:24:48.584 --> 00:24:57.464
there's so many dots on it, but I think [laughs] it reflects a little bit, you know, what, what we felt like, how crazy it's been. Is this a-- Why 14, uh, as the default? Is that a manual choice?
152
00:24:58.192 --> 00:25:07.752
'Cause you got ServiceNow in there- Mm-hmm... that are, you know, less, less traditional names. Yeah, it's models that we're kind of highlighting by default in our charts, in our intelligence index- Okay...
153
00:25:07.792 --> 00:25:10.772
is where this- You just have a manually curated list of stuff. Yeah, that's right.
154
00:25:10.812 --> 00:25:20.572
But, um, something that I actually don't think every, uh, artificial [chuckles] analysis user knows is that you can customize our charts and, and choose what, uh, what models- Oh, yeah... are, are, are highlighted.
155
00:25:20.612 --> 00:25:27.232
Super important. Yeah, yeah. Um, and so if we, you know, take off a few names, it gets a little easier to- Yeah, yeah... a little easier to read. Yeah, totally.
156
00:25:27.472 --> 00:25:38.872
Yeah, but you, you can-- Uh, I love that you can see the o1 jump. Look at that. Mm. September 2024. And the DeepSeek jump. That is... [laughs] Yeah. Which got close to- Ah... OpenAI's leadership. They were so close.
157
00:25:39.732 --> 00:25:49.692
I think, yeah, we, we remember that moment. Ar-around this time, uh, last year, actually. Yeah, yeah, yeah. Agree. Yeah, well, couple of weeks. It was, it was Boxing Day in New Zealand- [laughs]...
158
00:25:49.712 --> 00:25:52.452
uh, when, when DeepSeek V3 came out.
159
00:25:52.892 --> 00:26:02.851
And I-- Like, we'd been tracking DeepSeek and a bunch of the other global players that were less known over, like, the second half of 2024 and had run evals on the earlier ones and stuff.
160
00:26:03.192 --> 00:26:13.032
I, I very distinctly remember Boxing Day in New Zealand, I, 'cause I was with family for Christmas and stuff, running evals and getting back result by result on DeepSeek V3.
161
00:26:13.472 --> 00:26:22.732
Um, so this was, like, the, the first of their V3 architecture, the six seven oneb MoE. Um, and we were very, very impressed.
162
00:26:22.972 --> 00:26:32.592
Like, that was the moment where we were sure that DeepSeek was no longer just one of many players that had jumped up to be a thing.
163
00:26:33.012 --> 00:26:39.232
Um, the world's really noticed when they followed that up with the RL working on top of V3 and R1 succeeding, like, a few weeks later.
164
00:26:39.572 --> 00:26:48.892
Um, but the groundwork for that absolutely was laid with a, like, just extremely strong base model, completely open weights, that we had as the best open weights model on Boxing Day last year. Yep.
165
00:26:48.932 --> 00:26:59.752
Boxing Day is the, the day after Christmas for, for those not- [laughs] True. No familiar- I mean, I'm, I'm from Singapore. A lot of us remember Boxing Day, uh, for, for a different reason, for the tsunami that happened.
166
00:27:00.172 --> 00:27:12.512
Oh, of course. Yeah. Yeah. So I was... Yeah, yeah. But that was a long time ago. So yeah, so this is the, the rough, uh, pitch of AQI. Uh, or is it AQI or A-AII? II. Okay. So good, good memory, though. So- I don't know.
167
00:27:12.592 --> 00:27:20.432
I'm not used to it... once, once upon a time, we did call it quality index. Okay. Um, and we would talk about, um, quality, performance, and price, but, um, we changed it to intelligence. Yeah.
168
00:27:20.452 --> 00:27:28.012
There's been a few naming changes. We added hardware benchmarking to the site- Right... and, um, so benchmarks at a, at a kind of system level.
169
00:27:28.312 --> 00:27:36.392
And so then we changed our throughput metric to we now call it output speed, and then [chuckles] throughput makes sense at a system level, so [laughs] Got it. Got it... took that name.
170
00:27:36.472 --> 00:27:43.732
Uh, take me through more charts today. What should, what should people know? You know, obviously, the way you look at the site is probably different than how a beginner may look at it. Yeah, it's-- That, that's fair.
171
00:27:43.772 --> 00:27:51.472
We can... There's, there's a lot of fun stuff to dive into. Um, maybe so we can hit past all the, like, we've, like, we have lots and lots of evals and stuff.
172
00:27:51.892 --> 00:28:00.852
Um, the interesting ones to talk about today that'd be great to bring up are, like, a few of our recent things, I think, um, that probably not many people will be familiar with yet.
173
00:28:01.292 --> 00:28:11.412
So first one of those is our omniscience index. So this one is a little bit different to most of the intelligence evals that we run.
174
00:28:11.892 --> 00:28:28.512
We built it specifically to look at the embedded knowledge in the models and to test, um, hallucination by looking at when the model doesn't know the answer, it's not able to get it correct, what's its probability of saying, "I don't know," or giving an incorrect answer.
175
00:28:29.132 --> 00:28:41.012
So the metric that we use for omniscience goes from negative 100 to positive 100 because we're simply taking off a point if you give an incorrect answer to the question.
176
00:28:41.772 --> 00:28:52.152
We're pretty convinced that this is an example of where it makes most sense to do that because it's strictly more helpful to say, "I don't know," instead of giving a wrong answer to a factual knowledge question.
177
00:28:52.952 --> 00:29:06.072
And one of our goals is to shift the incentive that evals create for, um, models and the labs creating them to get higher scores.
178
00:29:06.512 --> 00:29:18.152
And almost every eval across all of AI up until this point, it's been graded by simple percentage correct as the main metric, the main thing that gets hyped. And so you should take a shot at everything.
179
00:29:18.192 --> 00:29:30.632
There's no incentive to say, "I don't know." So we did that for this one here. I think there's a general f-uh, field of calibration as well, like the confidence in your answer versus the- Yeah... rightness of the answer.
180
00:29:30.812 --> 00:29:32.012
Yeah. Completely agree. Yep. Yeah.
181
00:29:32.412 --> 00:29:43.872
On that, and one reason that we didn't do that is be-- uh, or put that into this, uh, index, is that we think that the, the way to do that is not to ask the models how confident they are. I don't know. Maybe.
182
00:29:43.912 --> 00:29:49.532
I mean, it might be, though, right? You put a- Like-... you put a JSON field, say, say confidence, and maybe it spits out something. Yeah.
183
00:29:49.852 --> 00:29:57.512
You know, and we have done a few evals podcasts over the, over the years, and we did one with Clementine of Hugging Face. Yeah, yeah, yeah. Clementine's the open source leaderboard.
184
00:29:57.852 --> 00:30:05.312
And this was one of her top requests, which is some kind of hallucination/lack of confidence calibration thing. And so, hey, this is one of them.
185
00:30:05.592 --> 00:30:18.672
And I mean, like anything that we do, it's not a perfect metric or the whole story of everything that you think about as hallucination. Mm-hmm. Um, but yeah, it's pretty useful and has some interesting results.
186
00:30:18.712 --> 00:30:30.572
Like, one of the things that we saw in the hallucination rate is that Anthropic's Claude models at the, the, the very left-hand side here were the lowest hallucination rates out of the models that we've evaluated omniscience on.
187
00:30:31.192 --> 00:30:43.292
That is an interesting fact. I think it probably correlates with a lot of the previously not really measured vibe stuff that people like about some of the Claude models. Is the dataset public? Or what's...
188
00:30:43.372 --> 00:30:51.532
Is it, is there a held-out set? There's a held-out set for this one. Okay. Um, so it-- We, we have published a public test set- Mm... but we, we've only published 10% of it.
189
00:30:51.952 --> 00:31:02.232
The reason is that for this one here specifically, it would be very, very easy to, um, like, have data contamination- Yeah... because it is just factual knowledge questions.
190
00:31:02.552 --> 00:31:10.884
Um, we will update it over time to also, um, prevent that, but- We've, yeah, kept most of it held out so that we can keep it reliable for a long time.
191
00:31:11.384 --> 00:31:23.444
It leads us to a bunch of really cool things, including breakdown quite granularly by topic, and so we've got some of that disclosed on the website publicly right now, and there's lots more coming in terms of our ability to break out very specific topics.
192
00:31:23.844 --> 00:31:32.624
Yeah. I would be interested. Let's, let's dwell a little bit on this hallucination one. I noticed that Haiku hallucin- hallucinates less than Sonnet hallucinates less than Opus. And- Yeah...
193
00:31:32.684 --> 00:31:37.204
would that be the other way around in a normal capability environment? I don't know. What's... What do you make of that?
194
00:31:37.564 --> 00:31:46.224
One interesting aspect is that we've found that there's not really a, not a strong correlation between intelligence and hallucination rate.
195
00:31:46.264 --> 00:31:55.624
That's to say that the smarter the, the models are in a general sense isn't correlated with their ability to, when they don't know something, say that they don't know. Mm.
196
00:31:55.744 --> 00:32:10.284
It's interesting that Gemini 3 Pro Preview was a big leap over here, Gemini 2.5 Flash and, and, and 2.5 Pro, but, and if I add Pro quickly here. I bet Pro's really good. Uh, actually, no. It's similar.
197
00:32:10.284 --> 00:32:20.384
So I me- I meant, uh, the GPT Pros. Oh, yeah. 'Cause GPT Pros are rumored, we don't know for a fact, that it's like eight runs, and then with the LM judge on top. Yeah.
198
00:32:20.664 --> 00:32:25.444
So we saw a big jump in, this is accuracy, so this is just percent that they get, uh, correct.
199
00:32:25.844 --> 00:32:38.904
And Gemini 3 Pro knew a lot more than the other models, and so big jump in accuracy, but relatively no change between the Google Gemini models between releases. And the hallucination rate. Exactly. Metric, yeah.
200
00:32:38.924 --> 00:32:47.524
And so it's likely due to just kind of different post-training recipe between the, the Claude models- Yeah... um, that- Yeah... that's driven this. Yeah.
201
00:32:47.804 --> 00:32:59.004
You can, uh, you, you can partially blame us in how we define intelligence, having until now not defined hallucination as, uh, negative in the way that we think about intelligence, and so that's what we're changing.
202
00:32:59.444 --> 00:33:07.764
Uh, I know many smart people who are confidently incorrect. [laughs] Look- Look, that, that, that is- Very human... very true. [laughs] And there's times and a place for that.
203
00:33:07.884 --> 00:33:18.184
I think our view is that hallucination rate makes sense in this context where it's around knowledge, but in many, uh, cases people want the models to hallucinate, to have a go.
204
00:33:18.464 --> 00:33:32.304
Often that's the case in coding or when you're trying to generate newer ideas. One eval that we added to artificial analysis is, is, is Critical Point, and it's really hard, uh, physics problems. Okay.
205
00:33:32.404 --> 00:33:40.224
And- Is, is it sort of like a human eval type or something different? Or like a frontier math type? It's not dissimilar to frontier, frontier math.
206
00:33:40.284 --> 00:33:51.684
So these are kind of research questions that kind of academics in the physics, physics world would be able to answer. But models really struggle to answer, so the top score here is n- nine percent.
207
00:33:52.104 --> 00:34:01.504
And when the people that, that created this, like Min Hui and, and, and actually O- Ophir, who was kind of behind Sweet Bench- What organization is this? Oh, is this, it's Princeton?
208
00:34:02.044 --> 00:34:10.164
Kind of range of academics from- Okay... from, uh, different academic institutions. Really smart people. They talked about how they turn the models up in terms of the temperature.
209
00:34:10.184 --> 00:34:19.344
[laughs] As high a temperature as they can when they're trying to explore kind of new ideas in physics as a, as a thought partner just because they, they want the models to hallucinate. Mm-hmm. Um- Mm-hmm.
210
00:34:19.624 --> 00:34:22.064
Yeah, sometimes it's a feature... and maybe get to something new. Yeah. Yeah, exactly. It's a feature.
211
00:34:22.164 --> 00:34:28.644
Um, so not right in every situation, but, um, think it makes sense, you know, to test hallucination in scenarios- Yeah... where it makes sense.
212
00:34:29.104 --> 00:34:41.144
Also, the obvious question is, uh, this is one of many that there is, there, every lab has a system card that shows some kind of hallucination number, and you've chosen to not, uh, endorse that, and you've made your own.
213
00:34:41.524 --> 00:34:50.124
And I think that's a, that's a choice. Um- Totally... i- in some sense, the rest of artificial analysis is public benchmarks that other people can independently rerun. You provide it as a service.
214
00:34:50.524 --> 00:34:58.284
Here you have to fight the, "Well, who are we to, to, like, do this?" And- Yeah... your, your answer is that we have a lot of customers and, you know.
215
00:34:58.584 --> 00:35:08.724
But, like, I guess, how do you converge the industry on one number that actually everyone agrees is, is the rate, right? 'Cause you have your numbers. They have their numbers. Never the, the two shall meet.
216
00:35:09.484 --> 00:35:16.424
I mean, I think, I think for hallucinations specifically, there are a bunch of different things that you might care about reasonably and that you'd measure quite differently.
217
00:35:16.724 --> 00:35:23.363
Like, we've called this AA amnesia and hallucination rate, not trying to declare like it's the only one- Humanity is less than the hallucination.
218
00:35:23.384 --> 00:35:27.964
[laughs] [laughs] You could, uh, you could have some interesting naming conventions and all this stuff.
219
00:35:28.303 --> 00:35:37.024
Um, the biggest picture answer to that, and something that I actually wanted to mention just as George was explaining Critical Point as well, is so as we go forward, we are building evals internally.
220
00:35:37.364 --> 00:35:41.544
We're partnering with academia and partnering with AI companies to build great evals.
221
00:35:41.904 --> 00:35:54.704
We have pretty strong views on, in various ways for different parts of the AI stack, where there are things that are not being measured well or things that developers care about that should be measured more and better, and we intend to be doing that.
222
00:35:55.084 --> 00:36:05.264
We're not obsessed necessarily with the everything we do we have to do entirely within our own team. Critical Point is a cool example of where we were a launch partner for it working with academia.
223
00:36:05.564 --> 00:36:09.364
We've got some partnerships coming up with a couple of leading companies.
224
00:36:09.764 --> 00:36:15.524
Those ones obviously we have to be careful with on some of the independent stuff, but with the right disclosure, like, we're completely comfortable with that.
225
00:36:15.584 --> 00:36:26.264
A lot of the labs have released great datasets in the past that we've used to great success independently. And so it's between all of those techniques we're gonna be releasing more stuff in the future. Cool.
226
00:36:26.624 --> 00:36:33.504
Let's cover the, the last couple, and then we'll, I want to talk about your trends analysis stuff, you know? Totally. Before that, actually, I have one, like, little factoid on- Okay... amnesia.
227
00:36:33.584 --> 00:36:48.544
If you go back up to accuracy on amnesia, an interesting thing about this accuracy metric is that it tracks more closely than anything else that we measure the total parameter count of models. Ooh.
228
00:36:48.584 --> 00:36:53.504
Makes a lot of sense intuitively, right? Because this is a knowledge eval. This is the pure knowledge metric.
229
00:36:53.544 --> 00:37:04.584
We're not looking at the index and the hallucination rate stuff that we think is much more about how the models are trained. This is just what facts did they recall, and yeah, it tracks parameter count extremely closely.
230
00:37:05.104 --> 00:37:14.824
Okay. What's the rumored size of GPT 3 Pro? [laughs] And to be clear, not confirmed for any official source. Yeah, no, not, not at all. Just, just rumors. But rumors do fly around. Rumors, yeah.
231
00:37:14.884 --> 00:37:22.904
I get, I hear all sorts of numbers. I don't know what to trust. So if you, if, if you draw the line on amnesia's accuracy versus total parameters, we've got all the open weights models.
232
00:37:23.364 --> 00:37:35.008
You can squint and see that- Likely, the leading frontier models right now are quite a lot bigger than the one trillion parameters that the open-weights models cap out at and the ones that we're looking at here.
233
00:37:35.588 --> 00:37:47.648
There's an interesting extra, um, data point that Elon Musk revealed recently about xAI that Grok three- Four... four, three trillion parameters for Grok three and four, six trillion for Grok five.
234
00:37:47.868 --> 00:37:53.648
Um, but that's not out yet. Take those together, have a look. You might reasonably form a view- Yeah...
235
00:37:53.688 --> 00:38:05.068
that there's a pretty good chance that Gemini 3 Pro is bigger than that, that it could be in the five to 10 trillion parameter range. To be clear, I have absolutely no idea, but just based on this chart- Yeah...
236
00:38:05.088 --> 00:38:12.248
like that's where you would, you would land if you have a look at it. Yeah. A- and to some extent, I actually kind of discourage people from guessing too much because- Yeah... what does it really matter?
237
00:38:12.288 --> 00:38:17.188
Like, as long as they can serve it as a sustainable cost, that's about it. Like- Yeah, totally.
238
00:38:17.188 --> 00:38:23.528
[laughs] They've also got different incentives in play compared to like open-weights models who are thinking to supporting others in self-deployment.
239
00:38:23.708 --> 00:38:34.028
For the labs who are doing inference at scale, it's, I think, less about total parameters in many cases when thinking about inference costs and, and more around number of active parameters.
240
00:38:34.068 --> 00:38:43.448
And so there's a bit of an incentive- Totally... towards larger, sparser models. Agreed. Understood. Yeah. Great. I mean, obviously, if you're a developer or company using these things, none of it...
241
00:38:43.748 --> 00:38:47.808
E- exactly as you say, it doesn't matter. You should be looking at all the different ways that we measure intelligence.
242
00:38:47.848 --> 00:38:55.828
You should be looking at our cost to run index number and the different ways of thinking about token efficiency and cost efficiency based on the list prices, 'cause that's all that matters.
243
00:38:56.468 --> 00:39:07.148
It's not as good for the content creator rumor mill, where I can say, "Well, GPT-4 is this small circle. Look at GPT-5, this big circle." And that, and that used to be a thing for a while. Um- Yeah.
244
00:39:07.148 --> 00:39:12.388
[laughs] I mean, but that, that, that is like a, on, on its own, actually a very interesting one, right? That- Is it?
245
00:39:12.388 --> 00:39:20.948
[laughs] Well, no, j- just purely that chances are the last couple of years haven't seen a dramatic scaling up in the total size of these models. Yeah. Yeah, we hit a wall.
246
00:39:20.948 --> 00:39:33.688
[laughs] And so there's a lot of room to go up probably in total size of the models, especially with the upcoming hardware generations. Yes. So, um, you know, taking off my shitposting face for a minute. [laughs] Okay.
247
00:39:33.708 --> 00:39:38.828
Uh, yes, yes. At the same time, I, I do feel like, you know, especially coming back from Europe, people do feel like
248
00:39:39.768 --> 00:39:47.968
Ilya is probably right that the paradigm is, doesn't have many more orders of magnitude to scale out more. Mm-hmm. And therefore we need to start exploring at least a different path.
249
00:39:48.368 --> 00:39:57.528
GDP Eval, I think it's like only like a month or so old. Um, I was also very positive when it first came out. I actually talked to Tejo, uh, who was the, the lead researcher on that. Oh, cool.
250
00:39:57.588 --> 00:40:04.228
And, uh, you have your own version. It's a fantastic data set. Yeah. And- Maybe I will recap for people who are still out of it.
251
00:40:04.268 --> 00:40:12.468
It's like 44 tasks that based on some kind of GDP cutoff that's like re- meant to represent broad white collar work that is not just coding. Yeah. Yeah.
252
00:40:12.908 --> 00:40:17.828
Each of the tasks have a whole bunch of detailed instructions, some input files for a lot of them. It's... I...
253
00:40:18.028 --> 00:40:27.488
Within the 44, it's divided into like 225 maybe, um, sub-tasks that are the, the level of that we run through the agentic harness. And yeah, they're really interesting.
254
00:40:27.808 --> 00:40:34.828
I will say that it doesn't necessarily capture like all the stuff that people do at work. No eval is perfect.
255
00:40:34.848 --> 00:40:46.928
There's always gonna be more things to look at, largely because in order to make the tasks well enough to find that you can run them, they need to only have a handful of input files and very specific instructions for that task.
256
00:40:47.328 --> 00:40:59.228
And so I think the easiest way to think about them are that they're like quite hard take-home exam tasks that you might do in an interview process. Yeah, for listeners, it is not, no longer like a long prompt.
257
00:40:59.588 --> 00:41:06.188
It is like, "Well, here's a zip file with like a spreadsheet or a PowerPoint deck or a PDF, and go, go nuts and answer this question." Yeah.
258
00:41:06.488 --> 00:41:16.468
OpenAI released a great data set, and they released a good paper which looks at performance across the different, you know, web chatbots on the data set. It's a great paper. Encourage people to read it.
259
00:41:16.768 --> 00:41:23.568
What we've done is taken that data set and turned it into an eval that can be run on any model.
260
00:41:24.228 --> 00:41:35.848
So we created a reference agentic harness that can run the models on the data set, and then we developed evaluator approach to compare outputs.
261
00:41:36.248 --> 00:41:50.048
That's kind of A- A- AI enabled, so it uses, uh, Gemini 3 Pro Preview to compare results, which we tested pretty comprehensively to ensure that it's aligned to, to human, human preferences.
262
00:41:50.508 --> 00:42:01.648
One data point there is that, um, even as the, as, as an evaluator, Gemini 3 Pro interestingly, uh, doesn't do, uh, actually that well in GDP Eval, uh, AA. Yeah.
263
00:42:01.688 --> 00:42:08.868
The, the thing that you have to watch out for with LLM Judge is self-preference, that models usually prefer their own output. Uh, and in this case, it was not. Totally.
264
00:42:09.268 --> 00:42:19.288
I think the, the, the way that we're, that we're thinking about the places where it makes sense to use an LLM as Judge approach now are like quite different to some of the early LLM as Judge stuff a couple of years ago.
265
00:42:19.548 --> 00:42:28.628
Okay. 'Cause some of that, and MT-Bench was a great project that was a good example of some of this a while ago, was about judging conversations and like a lot of style type stuff.
266
00:42:29.188 --> 00:42:38.708
Here, we've got the task that the grader in grading model is doing is quite different to the task of taking the test. When you're taking the test, you've got all of the agentic tools.
267
00:42:38.768 --> 00:42:44.788
You're working with the code interpreter on web search, the file system, to go through many, many turns to try to create the documents.
268
00:42:45.188 --> 00:43:01.148
Then on the other side, when we're grading it, we're running it through a pipeline to extract visual and text versions of the files and be able to provide that to Gemini, and we're providing the criteria for the task and getting it to pick which one more effectively meets the criteria of the task out of two potential outcomes.
269
00:43:01.508 --> 00:43:08.788
It turns out that we proved that it's just very, very good at getting that right, matched with human preference a lot of the time because it's...
270
00:43:09.088 --> 00:43:26.408
I think it's got the raw intelligence, but it's combined with the correct representation of the outputs, the fact that the outputs were created with an agentic task that is quite different to the way the grading model works, and we're comparing it against criteria, not just kind of zero shot trying to ask the model to pick which one is better.
271
00:43:27.136 --> 00:43:42.356
Got it. Why is this an ELO and not a percentage, like GDPVal? So the outputs look like documents, and there's video outputs or audio outputs from some of the tasks. And so- It has to make a video? Yeah.
272
00:43:42.396 --> 00:43:51.056
For some of the tasks, yeah. Some of the tasks. What task is that? [laughs] I mean, it's in, it's in the data set. Maybe a YouTuber or like- It's a marketing video. Oh, wow. What?
273
00:43:51.436 --> 00:43:58.276
Like, model has to go find clips on the internet and try to put it together. The models are not that good at doing that one for now, to be clear. [laughs] It's pretty, it's pretty hard to do that with the code as of now.
274
00:43:58.276 --> 00:44:09.776
[laughs] Um, and the computer stuff doesn't work quite well enough, and so on and so on. But, um, yeah. And so there's no kind of ground truth necessarily to compare against to work out percentage correct.
275
00:44:10.196 --> 00:44:23.976
It's ha- hard to come up with correct or in-in-incorrect there. And so it's on a relative basis, and so we use an E- ELO approach to compare outputs from, uh, each of the models, um, between, between the task.
276
00:44:24.136 --> 00:44:32.375
You know what you should do? You should, you should pay a contractor human to do the same task and then give it an ELO. [laughs] And then, so you have- Yeah... you have human there.
277
00:44:32.456 --> 00:44:42.776
So it's, I think what, what's helpful about GDPVal, the OpenAI one, is that 50% is meant to be normal human- Yes... and, and, and maybe domain expert is higher than that.
278
00:44:42.816 --> 00:44:52.536
But 50% was the, the bar for, like, well, if you've crossed 50, you are superhuman. Yeah. So we, like, haven't grounded this score in that exactly.
279
00:44:52.856 --> 00:44:57.796
I agree that it can be helpful, but we wanted to generalize this to a very large number of models.
280
00:44:57.836 --> 00:45:03.836
That's one of the reasons that presenting it as ELO is quite helpful and allows us to add models, and it'll stay relevant for quite a long time.
281
00:45:04.356 --> 00:45:14.996
I also think it, it, it can be tricky looking at these exact tasks compared to the human performance, 'cause the way that you would go about it as a human is quite different to how the models would go about it. Yeah.
282
00:45:15.376 --> 00:45:26.236
Uh, I also like that you included Llama 4 Maverick in there. Is that, like, just one last, like [laughs] Well- Like- No, no, no, no, no, no. [laughs] It is the, it is the best model released by Meta.
283
00:45:26.296 --> 00:45:40.736
[laughs] And so it makes it into the homepage default set still for now. Other inclusion that's quite interesting is we also ran it across the latest versions of the web chatbots. Yeah. And so we have- Oh, that's right.
284
00:45:40.816 --> 00:45:49.496
Oh, sorry, I, yeah, I completely missed that. Okay. No, not at all. So the, that which has a checkered pattern there- So, so that is their harness not yours, is what you're saying? Exactly.
285
00:45:49.896 --> 00:46:02.516
And what's really interesting is that if you compare, for instance, Claude 4.5 Opus using the Claude web chatbot, it performs worse than the model in our agentic harness. Mm-hmm.
286
00:46:02.916 --> 00:46:13.016
And so in every case, the model performs, uh, better in our agentic harness than its web chatbot counterpart, the harness that they created.
287
00:46:13.656 --> 00:46:20.896
Well, my backwards explanation for that would be that, well, it's meant for consumer use cases, and here- Totally... you're pushing it for something- The constraints are different- Yeah...
288
00:46:20.976 --> 00:46:27.536
and the amount of freedom that you can give the model is different. So- Also, you, like, have a cost goal. We- Yeah... yeah, let the models work as long as they want, basically. Yeah.
289
00:46:28.036 --> 00:46:33.456
Do you copy/paste manually into the chatbot? Yeah. Yeah. That's amazing. That was how we got the chatbot reference once, yeah. Yeah.
290
00:46:33.456 --> 00:46:40.496
[laughs] We, we're not gonna be keeping those updated at, like, quite the same scale as- Yeah. [laughs]... running it on the hundreds of models on the other harness. Well, so, and, I don't know, talk to BrowserBase.
291
00:46:40.536 --> 00:46:43.196
They'll, they'll automate it for you. You know, like- True. Yeah. Yeah. We should.
292
00:46:43.736 --> 00:46:53.456
Uh, yeah, I, I have thought about, like, well, we should turn these chatbot versions into an API because they are legitimately different agents in themselves. Yes. Right? Yeah.
293
00:46:53.636 --> 00:46:56.456
So- And that's grown a huge amount over the last year, right? Mm-hmm.
294
00:46:56.476 --> 00:47:08.956
Like, the tools that are available have actually diverged, in my opinion, a fair bit across the major chatbot apps, and the amount of data sources that you can connect them to have gone up a lot, meaning that your experience and the way you're using the model is- Yeah...
295
00:47:09.276 --> 00:47:16.896
more different than ever. What tools and what data connections come to mind when you say... What's interesting? What, what, what, what's notable work that people have done? Oh, okay.
296
00:47:17.156 --> 00:47:26.736
So my favorite example on this is that until very recently, I would argue that it was basically impossible to get an LLM to draft an email for me- Mm... in any useful way. Mm.
297
00:47:26.756 --> 00:47:37.416
Because most times that you're sending an email, you're not just writing something for the sake of writing it. Chances are context required is a whole bunch of historical emails. Maybe it's notes that you've made.
298
00:47:37.616 --> 00:47:45.616
Maybe it's meeting notes. Maybe it's, um, pulling something from your, um, any of, like, wherever you at work store stuff.
299
00:47:45.676 --> 00:47:51.476
So for me, like, Google Drive, OneDrive, um, and our Supabase databases if we need to do some analysis on some data or something.
300
00:47:52.116 --> 00:47:57.936
Preferably, model can be plugged into all of those things and can go do some useful work based on it.
301
00:47:58.416 --> 00:48:11.816
The things that, like, I find most impressive currently that I am somewhat surprised work really well in late 2025 are that I can have models use Supabase MCP to- Query... read-only, of course. Yeah.
302
00:48:11.836 --> 00:48:23.396
Run a whole bunch of SQL queries to do pretty significant data analysis and make charts and stuff, and can read my Gmail and my Notion. And okay, you actually use that. That's good. That's, that's, that's good.
303
00:48:23.576 --> 00:48:32.556
Is that a Claude thing? To various degrees supported on both ChatGPT and Claude- Yeah... right now. I would say that the stuff like barely works in fairness right now. [laughs] Okay.
304
00:48:32.816 --> 00:48:39.176
[laughs] Like- 'Cause people are actually gonna try this after they hear it. If you get an email from Micah, odds are it wasn't written by a chatbot. No.
305
00:48:39.176 --> 00:48:49.496
[laughs] So yeah, I, I think it is true that I have never actually sent anyone an email drafted by a chatbot yet. Um, [laughs] and so- But you can, you can feel it, right? And- Yeah. Yeah...
306
00:48:49.536 --> 00:48:56.036
this time, this time next year we'll come back and see where it's going. Totally. Um, Supabase shout-out, another famous Kiwi. Uh- Yeah.
307
00:48:56.216 --> 00:49:04.996
[laughs] I don't know if you've, you've any conversations with him about anything in particular on, on AI building and AI infra? We have had- Is that right?...
308
00:49:05.036 --> 00:49:16.656
uh, Twitter DMs, um, with, with him because we're quite big, uh, Supabase users and p- and power users, and we probably do some things more manually than we should in, in Supabase.
309
00:49:16.676 --> 00:49:22.736
[laughs] So he's just the support line because you're, you're Kiwis. A, a little bit, yeah. [laughs] Be- being, being super friendly.
310
00:49:22.756 --> 00:49:41.176
[laughs] One extra, um, point regarding, um, GDPVal AI is that on the basis of the overperformance of the models compared to the, the chatbots, turns out we realized that, oh, like, our reference harness that we built actually w- works quite well on, like, gen- generalist agentic tasks.
311
00:49:41.276 --> 00:49:46.236
Mm-hmm. This proves it in a sense. Mm. And so the agent harness is very minimalist.
312
00:49:46.376 --> 00:50:02.212
I think it follows some of the ideas that are in Claude Code, and we- All that we give it is context management capabilities, a web search, web browsing, uh, tool, a code execution, uh, environment. Anything else?
313
00:50:02.652 --> 00:50:10.392
I mean, we can equip it with more tools, but, like, by default, yeah, it's... that's it. We, we, we give it the GDP-valid tool to, uh, view an image specifically- Mm...
314
00:50:10.452 --> 00:50:19.912
um, because the models, you know, can just use a terminal to pull stuff in text form into context. But to pull visual stuff into context, we had to give them a custom tool. Yeah. Um, but yeah, exactly. Um- Yeah...
315
00:50:20.012 --> 00:50:29.432
you, you can de- explain it next, but. No, so it's, it, it, w- we turned out that we created a good generalist agentic harness, and so we, um, released that on, on GitHub yesterday.
316
00:50:29.672 --> 00:50:39.552
It's called Stirrup, so if people want to check it out. Yeah. And, and it's a great, um, you know, base for, you know, generalist, uh, building a generalist agent- It is kind of cool... for, for more specific tasks.
317
00:50:39.952 --> 00:50:51.352
I'd say the best way to use it is Git clone and then have your favorite coding agent make changes to it to do whatever you want because it's not that many lines of code, and the coding agents can work with it super well.
318
00:50:51.812 --> 00:51:03.572
Well, that's nice for the, the community to explore and share and hack on it. I think maybe in, in, in other similar environments, the Terminal Bench guys have done, uh, started Harbor. Yeah.
319
00:51:03.572 --> 00:51:14.092
Uh, and so it's, it, it's a, it's a bundle of, well, we need our minimal harness, which for them is Terminus. Yep. And we also need the RL environment, so our Docker deployment thing to, to run independently.
320
00:51:14.152 --> 00:51:18.792
So, uh, I don't know if you've looked into Harbor at all. Is that, is that, like, a, a standard that people want to adopt?
321
00:51:19.712 --> 00:51:28.832
Yeah, we've looked at it from a evals, um, perspective, and we love Terminal Bench and, and host benchmarks of, of, of Terminal Bench on, on artificial analysis.
322
00:51:28.932 --> 00:51:36.572
Um, we've looked at it from a, from a coding agent, uh, perspective- Mm-hmm... but could see it being a great, um, basis for a- any kind of agents.
323
00:51:36.812 --> 00:51:55.832
I think where we're getting to is that these models have gotten smart enough, they've gotten better, better tools, that they can perform better when just given a minimalist set of tools and, and let them run, let the model control the, the agentic workflow rather than using another framework that's a bit more built out that tries to dictate the, dictate the flow.
324
00:51:56.112 --> 00:52:06.632
Awesome. Let's cover the openness index, and then let's go into the report stuff. Uh, so that's the, that's the last of the proprietary art numbers, I guess. I don't know how you sort of classify all these.
325
00:52:06.632 --> 00:52:13.272
[laughs] Yeah, or call, call it, let's call it the last of, like, the, the three new things that we're talking about- The three new things... from, like, the last few weeks. Yes. Um, 'cause there, I mean, there's a...
326
00:52:13.772 --> 00:52:24.192
W- we do a mix of stuff that where we're using open source, where we open source some of what we do, and, um, proprietary stuff that we don't always open source, like long context reasoning data set last year we did open source.
327
00:52:24.392 --> 00:52:33.772
Um, and then all of the work on performance benchmarks across the site, some of them we're looking to open source, but some of them, like, we're constantly iterating on and so on and so on and so on. Yes.
328
00:52:33.792 --> 00:52:42.732
So there, there's a huge mix, I would say, just of, like, stuff that is open source and not across the site. S- so that's AALCR for people- Yeah. Yeah, yeah, yeah... But let's talk, let's, let's talk about open.
329
00:52:43.072 --> 00:52:50.212
Let's talk about openness index. This here is, call it, like, a new way to think about how open models are.
330
00:52:50.892 --> 00:53:01.012
We, for a long time, have tracked where the models are open weights and what the licenses on them are, and that's, like, pretty useful. That tells you what you're allowed to do with the weights of a model.
331
00:53:01.572 --> 00:53:11.592
But there is this whole other dimension to how open models are that is pretty important that we haven't tracked until now, and that's how much is disclosed about how it was made.
332
00:53:11.672 --> 00:53:20.512
So transparency about data, pre-training data and post-training data, and whether you're allowed to use that data. Mm-hmm. And transparency about methodology and training code. Mm-hmm.
333
00:53:20.792 --> 00:53:32.552
So basically, those are the components. We bring them together to score an openness index for models so that you can, in one place, get this full picture of how open different models are.
334
00:53:33.192 --> 00:53:44.192
I feel like I've seen a couple other people try to do this, but it, uh, they're not maintained. I, I do think this does matter. I don't know what the numbers mean apart from... Is there a max number? Is this out of 20?
335
00:53:44.712 --> 00:53:51.752
It's out of 18 currently. Oh, 18, okay. And so we've got an openness index, um, page. But essentially, these are points.
336
00:53:51.832 --> 00:54:04.832
You get points for being more open across these different categories, and the, the maximum you can achieve is 18. So AI2 with their extremely open Olmo332B Think model is- It's with Hugging Face... the leader in a sense.
337
00:54:04.932 --> 00:54:12.932
With Hugging Face. Oh, with their, with their small model? Yeah. Um, it's coming soon. I think we need to run, we need to get the intelligence benchmarks run- You're right... to get it on the site.
338
00:54:12.992 --> 00:54:17.812
You can't have an open index and not include Hugging Face. Like it's- We love Hugging Face. We'll have that, we'll have that up very soon. I mean f- Yeah...
339
00:54:18.172 --> 00:54:25.152
uh, you know, the Refined Web and all, all that stuff, it's, uh, it's amazing. Or is it called FineWeb? FineWeb. FineWeb, yeah, yeah. No, totally. Yeah. Yep.
340
00:54:25.192 --> 00:54:34.972
One of the reasons this is cool, right, is that if you're trying to understand the holistic picture of the models and what you can do with all the stuff the company's contributing, this gives you that picture.
341
00:54:35.272 --> 00:54:46.372
And so we are gonna keep it up to date alongside all the models that we do intelligence index on on the site, and it's just an extra view to understand. Can you scroll down to the, the, the trade-offs chart? The, the...
342
00:54:46.452 --> 00:54:54.012
Yeah, yeah, that one. Oh. Yeah, this, this really matters, right? [laughs] Obviously- Yeah... 'cause you can be super open but dumb. I mean- And [laughs]... the, the slot obviously goes the wrong way here, right?
343
00:54:54.052 --> 00:55:04.092
[laughs] Um, and- A lot of people would like to see, uh, Labs hill climb on the end target- This is the-... the, uh, openness index. [laughs] Yeah, this is the access to, to hill climb. Yeah.
344
00:55:04.152 --> 00:55:11.892
Unfortunately, it might be fundamentally true that the, the slump will always go this direction because once- Uh, okay... you open something up, then everyone else can get to the level of what you have done.
345
00:55:11.932 --> 00:55:18.512
Well, so let me, let me tweak your point system, right? Like, you have these, like, numbers on the point system, and they go up to 18.
346
00:55:19.112 --> 00:55:28.732
You know, but, like, just because I have a little bit of open data doesn't mean I'm necessarily that much better as in someone who put a lot of effort into their open weights that is, that is smarter. Yeah.
347
00:55:28.772 --> 00:55:36.372
So I might, I might just mess with the point system to make sure that, like, I'm accurately representing the, the contribution to the open sys- openness.
348
00:55:36.872 --> 00:55:42.452
It is hard to wait for the materiality of the contribution to, to open source.
349
00:55:42.492 --> 00:55:49.412
Like it, it's, we, we, we tried to make it so that it is quite well-defined, and no one can disagree about, like, which, um, category things should be in.
350
00:55:49.452 --> 00:55:58.532
So we're not saying, like, this was a big contribution or a small contribution in terms of, um, impact on the industry or anything. It's just, like, how much of your data did you release?
351
00:55:58.852 --> 00:56:18.392
I would say that it is still valid to say that we trained a model that's not that smart, maybe even not at the frontier for a particular size category, but we chose to open up all- All the data, all the training code, that is a very useful exercise for the industry, and we want to recognize that even if the smallest model in the category.
352
00:56:18.752 --> 00:56:26.712
Yeah. Yeah. A-and also a special shout-out to NVIDIA NeMo Tron, which doesn't get enough credit for the amount of stuff that they do. Yeah. And honestly, it's a sales enablement for NVIDIA as well.
353
00:56:26.732 --> 00:56:29.752
[chuckles] Like, the fact that they can do this as a side project. [chuckles] Totally.
354
00:56:29.772 --> 00:56:36.492
But I mean, but it, it is true that NVIDIA have actually put an enormous amount of effort over the last year especially into the NeMo Tron models. Yeah. And it really went well.
355
00:56:36.492 --> 00:56:44.872
And so many people actually use it for, like, synthetic data and stuff. Like, it's, it's, uh, it's a pretty interesting secret of the industry that, uh, NVIDIA holds up all these guys.
356
00:56:44.872 --> 00:56:53.332
[laughs] I mean, it's, it's in their interest for there to be more AI, right? [chuckles] Uh, so obviously I think you want to push openness as, as having an index.
357
00:56:53.932 --> 00:57:05.152
Every index that you push, like, has encoded some kind of opinion or value. Yes. I think one of the openness questions so-from this year was people messing with the in- the, the license. And so- Yeah...
358
00:57:05.172 --> 00:57:11.892
Llama had this like if you have seven hundred million daily active users, y-you're not allowed to use our model or you have to talk to us, something like that.
359
00:57:12.212 --> 00:57:20.612
So basically, like what are your customers telling you about the kind of licensing worries that they have, right? Because obviously most people will never hit seven hundred million users.
360
00:57:20.952 --> 00:57:28.072
We have like a detailed breakdown of that in the Openness Index, and that was actually one of the initial questions that like took us down the route of wanting to do this.
361
00:57:28.132 --> 00:57:40.112
[laughs] Um, 'cause yeah, the simplest thing that like our opinion is, is that there is a lot of advantage to having, like a official OSI license like MIT or Apache 2 because then the box is just checked. Checked.
362
00:57:40.172 --> 00:57:44.532
You don't even need to read it because it's just Apache 2 and you can do whatever you want and it's fine.
363
00:57:44.852 --> 00:57:53.912
There are often very good reasons that companies don't want to release language models with those completely open licenses. The index tells you. So if you- Cool...
364
00:57:53.932 --> 00:58:03.272
hit the top category, that's one of those licenses, you're totally good. And then we've got, um, some lower categories for when attribution is required, um, and then when commercial use is not allowed.
365
00:58:03.632 --> 00:58:07.992
Yeah, they're there. So that's the open-openness index. Thank you for doing all those, all those works.
366
00:58:08.132 --> 00:58:14.432
Let's talk a little bit or at least end the pod on just the, the trend reports that you guys do, which is kind of a bit of the bread and butter, how you make money.
367
00:58:14.932 --> 00:58:24.732
Highly encourage everyone to see George's talk at World's Fair, which gives a little bit of a preview. And you, you were very excited about talking about the smiling curve or I don't know what you call it. Um- Yeah.
368
00:58:24.772 --> 00:58:31.812
Yeah. Yeah. Yeah, let's talk about that one. Let's explain it for people and, and I might, I might actually put, put it up, um, because I don't have it, uh, due to- Oh, great. Yeah. Oh, great. Yeah. Me too.
369
00:58:31.812 --> 00:58:33.652
If you've got a, got a copy of the slide, that'd be, that'd be excellent.
370
00:58:33.692 --> 00:58:42.532
Um- It-it's, it's important for people to have in their head because yeah, people only get the marketing message from the labs that, "Oh, we're cutting costs all the time." Yep. Yep. But it's, it's true.
371
00:58:42.532 --> 00:58:43.992
[laughs] It's just that it's not the whole picture.
372
00:58:44.272 --> 00:58:53.252
So okay, a couple of like the big trends that we track at Artificial Analysis over time and that like we're always showing charts of on the trends page in these reports and stuff.
373
00:58:54.052 --> 00:59:04.512
One, that the cost of intelligence has been falling dramatically over the last couple of years. The best way to think about that is that the cost for each tera of intelligence- Sure, yeah... has been dropping.
374
00:59:04.932 --> 00:59:15.072
The, like one fact on that is that you can get intelligence at the level of GPT-4 for over a hundred times cheaper than GPT-4 was at launch right now. I think my number's a thousand actually.
375
00:59:15.852 --> 00:59:25.032
If you look at, um, the Amazon Nova models which are very, very cheap. Yeah. Like, uh, my, my conservative, uh, statement is normally like 100X. 100 is a conservative, yeah.
376
00:59:25.072 --> 00:59:33.992
But in fairness, this slide like I-- we were actually saying before the podcast, right, is like maybe six months old now and it's conceptually still correct but like could actually probably do- Yeah, it should be around...
377
00:59:34.012 --> 00:59:38.952
a tweak on the exact numbers because like the market's moving so quickly. Yeah. Feel free to kick it off. I, I, I mean we'll have this chart.
378
00:59:39.232 --> 00:59:45.892
I told people to watch the World's Fair talk, but let's, let's introduce what context makes you make something like this.
379
00:59:46.692 --> 00:59:56.172
There are two trends that seem to not make sense together, both of which we talk a lot about at Artificial Analysis and are very important to developers building stuff in AI.
380
00:59:56.912 --> 01:00:04.372
The first is that the cost of intelligence for each level of intelligence has been dropping dramatically over the last couple of years.
381
01:00:04.952 --> 01:00:21.772
We track the cost to run Artificial Analysis intelligence index for each bucket of intelligence index scores and each bucket you just see the line go down really, really quickly and actually go down more quickly for each new level of intelligence that's been achieved over the last couple of years.
382
01:00:21.992 --> 01:00:33.232
So the rate of that cost decline has actually been going up. So we've got that being true and yet it is clearly possible to spend quite a lot more on AI inference now than it was a couple of years ago.
383
01:00:33.792 --> 01:00:46.352
NVIDIA stock go up. It's going, it's going really up. Uh, I just heard from a friend's startup that just went to the Shift 0, they're spending $5,000 per employee on coding agents spend alone.
384
01:00:46.832 --> 01:00:51.772
That's, uh- That's ridiculous amount of coding... that's an impressive number. We need to get our numbers up. We're, uh, we're not, we're not quite hitting, hitting 5K.
385
01:00:51.832 --> 01:00:55.312
Well, I was like it's so high that I'm like are you doing something wrong? [laughs] Yeah.
386
01:00:55.352 --> 01:01:06.492
It's 'cause there are some efficiency questions along the way but like you can make AI inference useful to that level in a bunch of ways that I can imagine, right? Yeah. Um, I, I don't think that's that nuts.
387
01:01:06.832 --> 01:01:15.012
Um, but basically the, the reason we made this slide to ask the question, right, is to show that the crazy thing is that it is actually true.
388
01:01:15.052 --> 01:01:32.032
We've had this 100X to 1000X decline in the cost of GPT-4 level intelligence on the left-hand side and yet on the right-hand side because the multipliers are so big for the fact that even though small models can do GPT-4 level now, we still want to use big models and probably bigger than ever models to, um, do frontier level intelligence.
389
01:01:32.312 --> 01:01:43.552
We've got reasoning models using tokens and then we're throwing them in these, them in these agentic workflows where they're consuming enormous numbers of input tokens and making enormous numbers of output tokens working for a really long time.
390
01:01:43.852 --> 01:01:50.992
Those two things taken together get you back to we can spend enormously more today than we could a couple of years ago. Yep. I think that's right.
391
01:01:51.032 --> 01:01:59.272
There's a number of drivers at play and we kind of outline kind of six key ones here. Um, but you know, as complex it's changing quickly.
392
01:01:59.352 --> 01:02:05.432
All of these have changed very dramatically in the last, uh, in the last 12 months. Le-let's speak on hardware efficiency since- Mm...
393
01:02:05.632 --> 01:02:20.252
you also have, you also track hardware stuff and I think the general assertion or the, the message is that the efficiency from next gen NVIDIA chips is actually not 4X or you have what 3X or 4X? You have 3X in here.
394
01:02:20.664 --> 01:02:31.304
And it's, it's like two X maybe, or it's more of like a power story rather than like a, a sheer sort of compute tokens efficiency story. But yeah, what, what's going on in, in hardware? Okay.
395
01:02:31.564 --> 01:02:42.004
So the, the, the, the answer unfortunately, uh, is it depends, and it just depends massively on like so many things across a bunch of different types of workloads and ways to think about it.
396
01:02:42.404 --> 01:03:00.204
So one of the simplest ways to think about this is to take single relevant model, to think about serving it at speeds that are realistic for what you actually might want to hit and can afford to hit, and then think about the throughput per GPU that you can achieve serving the model at those speeds.
397
01:03:00.744 --> 01:03:07.404
Reason-- One of the reasons that's important is that there's a trade-off between the throughput per GPU that you can achieve and the per user speed that you can achieve.
398
01:03:07.704 --> 01:03:19.724
And listen, it costs more to serve stuff fast to users. When you run all of that for especially big sparse models, you can get a lot better than two or three X gain going from Hopper to Blackwell generation Nvidia.
399
01:03:20.584 --> 01:03:21.204
I am...
400
01:03:22.244 --> 01:03:49.104
This shouldn't be too controversial a statement, like I'm pretty confident that Blackwell has delivered pretty enormous gains, and that the next couple of years of Nvidia's roadmap are going to continue to deliver quite enormous gains, and that those will actually come through as lower total cost per token to the companies that are running models on them, and will allow bigger models, will allow way more tokens to be made for lower cost, and that that's gonna continue.
401
01:03:49.504 --> 01:03:54.424
These things also stack on all of the software and model improvements being made.
402
01:03:54.784 --> 01:04:07.904
So basically, like my prediction across like both sides of that like smile chart, uh, that we're gonna see the left-hand side continue to be true and probably like for another order of magnitude, and the right-hand side continue to be true for another order of magnitude.
403
01:04:08.344 --> 01:04:18.924
And that's gonna enable a whole lot of things. Okay. Well, I'll push on. Let's go back to the, the, the smile chart. I'll push back on sparsity, right? W- Uh, we've gone a long way on sparsity.
404
01:04:19.264 --> 01:04:29.123
DeepSeek was a major pusher of fine-grain experts, let's call it. Yep. Right? I m- have a mental number of sparsity in terms of, let's say, active params versus total params. Yep.
405
01:04:29.144 --> 01:04:39.544
And that number went from twenty-five percent- Yep... let's say, down to like fifteen, right? You obviously can't really go below, I don't know, five. Is that obvious?
406
01:04:39.564 --> 01:04:47.804
So, so there's a lower limit to, to sparsity is what I'm saying. I don't know that that's that obvious, actually. All right. [laughs] Um, there, there must be a lot somewhere, right? Yeah, exactly.
407
01:04:48.064 --> 01:05:03.624
But we've got numbers in the wild that are quite a lot lower than that right now. So the GPD OSS models, like the big ones at about five percent, um, active. Kimi 2, is it like three percent active? Really? Oh, ooh.
408
01:05:03.664 --> 01:05:12.844
Okay. I think. Pretty sure. I, I've looked at those numbers. I've calculated them. I don't remember. Yeah, but I'll-- I, I remember thinking like- Yeah... this must be it. Y- y- your five percent is- [laughs]...
409
01:05:12.864 --> 01:05:17.544
i- i, yeah, is exactly, is like around the ballpark for the- [laughs]... for the open weights models of, of what's released today.
410
01:05:17.844 --> 01:05:37.964
I think one interesting that gives me kind of pause when thinking that it won't go, the sparsity won't go higher or the number of percentage of active parameters lower, is that we, in our benchmark, see a lot of performance to- correlated more with, uh, total parameters than active, and not that correlated with how sparse like the models are.
411
01:05:38.064 --> 01:05:50.364
Our accuracy benchmark as part of AA omniscience, it's very correlated with total. It's not correlated with, with active, uh, parameters. Yeah. Which I think is very-- at all, which is very, very interesting.
412
01:05:50.404 --> 01:05:54.864
And so I think, yeah, there could, there could be quite a bit, um, to go here. Awesome.
413
01:05:55.024 --> 01:06:01.724
Well, we don't have that much time, but I w- I did wanna leave some room to cover reasoning and non-reasoning models and token efficiency. Let's do that one.
414
01:06:01.744 --> 01:06:11.724
So at a, at a super high level, people have to classify this binary thing of reasoning versus non-reasoning. People who are insider have some discomfort with that because basically- Totally...
415
01:06:11.724 --> 01:06:22.964
you just have the think tag or no think tag. How have you guys decided to approach this? And also how does that s- laid out in over, over the course of the year where we have things like GPT 5, which is a model router?
416
01:06:23.584 --> 01:06:32.764
Let's say GPT 5 and ChatGPT, the consumer experience is a model router. When you're hitting the API, like we can-- you can pick the different versions and you can pick reasoning strength of the different versions.
417
01:06:33.064 --> 01:06:35.844
But that, that goes to why this is now such a complex thing.
418
01:06:36.503 --> 01:06:52.644
So earlier this year, and probably when you and George last spoke for the AI Engineers World's Fair, we had this great slide that was super easy, where we would show that the average reasoning model is using ten times the number of tokens per query in our intelligence index as the average non-reasoning model.
419
01:06:53.284 --> 01:06:59.064
And there was this moment where that was a pretty clear distinction and extremely useful to look at it just like that.
420
01:06:59.564 --> 01:07:13.084
Definitely no longer the case, not least because you can think about reasoning strength for a bunch of these different models, but particularly because different models have wildly different token efficiency now, more than an order of magnitude in difference.
421
01:07:13.664 --> 01:07:30.464
That means that the way that you probably need to think about cost for any application is to use something like our cost to run intelligence index metric as the starting point for what it's gonna look like for these different models, these different reasoning strengths, and this continuous spectrum from non-reasoning to reasoning.
422
01:07:31.064 --> 01:07:32.624
That's basically like where we're at.
423
01:07:32.684 --> 01:07:41.224
So we will still show reasoning and non-reasoning and define reasoning as when there is that separated chain of thought that you're getting at a different parameter in an API normally.
424
01:07:41.604 --> 01:07:51.524
But it doesn't necessarily anymore mean that that model is actually going to have longer end-to-end latency, that it's going to use more tokens than something that is branded a n- non-reasoning model- Mm...
425
01:07:51.544 --> 01:07:53.344
for the same task. Mm. That's true.
426
01:07:53.764 --> 01:08:10.184
I think 5.1 was it, and then, uh, 5.1 Codex had these, these chart, which was super nice of this like, let's say, bottom ten percentile query being faster but l- top ten percentile being longer, and that's a kind of the efficiency chart you want to see, right?
427
01:08:10.564 --> 01:08:17.864
Yep. That-- So, so that is a, an extra thing. [laughs] Let's say, let's say, let's say that we've got-- That, that's a really important extra thing though, right?
428
01:08:17.924 --> 01:08:29.944
That you've got not just the average number of tokens being used by the model, which we cover really well right now, but the b- behavior that you want in the model is it to use more tokens when it needs more tokens and not to use more tokens when it doesn't need more tokens.
429
01:08:30.284 --> 01:08:34.104
So that's what OpenAI were basically claiming that 5.1 Codex is better at.
430
01:08:34.504 --> 01:08:52.688
We- Don't actually publish anything on this right now, but have tracked it a bunch internally in our internal analytics on evals across all the models that we run, where we look at the difficulty of the questions and the correlation between token usage and difficulty, and net-net, surprise, surprise, like, models have got better at doing that over the course of this year.
431
01:08:53.008 --> 01:09:01.308
I think going into next year that's gonna be really important, especially as you multiply it by the number of steps in an agentic workflow that a model has to take to get to an answer.
432
01:09:01.628 --> 01:09:11.688
We are going to care a lot about token efficiency and number of turns efficiency for getting to what we want. Which would you rather have, token efficiency or number of turns efficiency?
433
01:09:12.428 --> 01:09:19.768
Um- Or, like, which is more important to work on? It's like it depends on the application, and both are gonna be really important. Ugh. Yeah.
434
01:09:19.768 --> 01:09:22.628
'Cause your total, your total cost is just- Like Taubitch retail, Taubitch airline. Yeah.
435
01:09:23.148 --> 01:09:51.308
Interestingly, in Tau, um, Taubench Telecom, it's cheaper to run, m- you know, on a per token basis more expensive models like a GPT-5 compared to some smaller open source models because the, um, some of the GPT-5, for instance, uh, got to the answer faster and so was able to resolve the customer's query faster in fewer turns, and maybe it used more tokens per turn, but it certainly cost more per token.
436
01:09:51.728 --> 01:09:56.128
So you would always rather use GPT-5 in, in, in that scenario. And so I think that's what, that's where we're getting to.
437
01:09:56.188 --> 01:10:05.788
I think number of turns is, is gonna be a metric that we're gonna be talking about a lot more, and, uh, I think it'll be something that people want to, uh, really start to think about, uh- Yeah... a lot more.
438
01:10:06.088 --> 01:10:18.208
There's a trade-off in benchmarking here where most benchmarks needs to be one turn to be autonomous, to be parallelized and all that, but most, a lot of your real life use cases need to be multi-turn and, and especially, like, quick multi-turns so you can align.
439
01:10:18.388 --> 01:10:25.528
Yeah. Yeah. I mean, I, I would say that historically benchmarks have been single turn, but I wouldn't say they need to be at all into the future, right?
440
01:10:25.788 --> 01:10:29.968
Like, we have a couple of agentic benchmarks in the index right now and GDP, all that we were talking about.
441
01:10:30.388 --> 01:10:37.668
We let the models do up to 100 turns and, um, our stirrup agent accounts to do that eval, and we're gonna build similar stuff like that in the future.
442
01:10:37.908 --> 01:10:48.268
It definitely is hard, and you've got whole kinds of infrastructure problems to run that and exactly as you say, parallelize it because we need to run that on hundreds of models, and we want to- Hmm...
443
01:10:48.308 --> 01:10:58.248
do that really fast when new models come out and when labs want us to run it on their models. But you can do it. We're putting in the work to build that stuff, and it's gonna be great. Okay. So we've covered...
444
01:10:58.348 --> 01:11:07.248
I mean, there's a lot more to cover. I- and we haven't even touched on multimodal, which [laughs] that's also huge. [laughs] We also do speech benchmarking- Speech, image... image benchmarking- Video...
445
01:11:07.488 --> 01:11:15.268
uh, video benchmarking- Video. Yeah, yeah... hardware. I like the way that you've done it, uh, 'cause it's very smart- Mm... which is, uh, video takes a long time, so you pre-generate, right?
446
01:11:15.408 --> 01:11:26.868
So then people just pick their preferences, and you can see the, the overall arena results, and you also avoid, like, any sensitivity issues ar-ar-around, like, you know, unsafe content that, that is being generated.
447
01:11:27.188 --> 01:11:27.308
Yeah.
448
01:11:27.908 --> 01:11:49.568
And you can see it as a, a good, good thing or bad thing depending on what your view is, but it means that we have a quite active creative direction approach to trying to understand what creative professionals and users want to do with those image and video models and so that we can be directing the arenas in our categories toward gathering votes on what people care about.
449
01:11:49.928 --> 01:11:58.428
One call out actually to listeners, like, if you are using our arenas is that you can submit requests to us for things that we should cover and- I didn't know that. Yeah...
450
01:11:58.488 --> 01:12:09.268
understudied categories, areas that you think the models are bad at and the labs don't focus on enough. Like, if you want something solved, one of the levers that you have is send us a couple of prompts on it.
451
01:12:09.488 --> 01:12:17.288
We might be able to get a category going on it. And this thing we were talking about earlier, right, that once things get measured, they can get targeted. You can make that work for you.
452
01:12:17.748 --> 01:12:24.408
For me as a content creator, infographics. Mm. Very needed. I took the latest DeepSeek paper and, uh, I...
453
01:12:24.468 --> 01:12:29.928
You know, they had some descriptions of their search agents and their coding agents, and I put it in there, and I, I created an infographic.
454
01:12:29.988 --> 01:12:45.648
And, um, I, I just think, like, that's an industrial use case that doesn't require a lot of, I guess, design taste, but just requires some, like, you, you need to conform to some preset references which is something that, uh, that is increasingly important especially in, like, the Nano Banana series.
455
01:12:46.008 --> 01:12:50.968
But, um, yeah. And I think, like, OpenAI is releasing Image two soon which, um, is gonna have that.
456
01:12:51.308 --> 01:13:01.188
So I, I, I think, like, it's, it's all, like, of a kind where people need to incentivize, like, workhorse use cases and not just art. I don't know. Totally. Yeah. And what are we gonna be talking about next year?
457
01:13:01.388 --> 01:13:05.688
Like, what's, what's, like, what's emerging that you're seeing and, like, maybe not in the discussion?
458
01:13:06.188 --> 01:13:17.248
The first answer that I'll give to that is the, the boring answer is that on most of our charts the lines go in a particular direction, and our overall prediction is the lines are gonna keep going in that direction.
459
01:13:17.288 --> 01:13:24.568
Keep going. [laughs] We're, we're gonna do a lot and do a lot to be as useful as possible to developers and companies to measure what's important on every one of those and along those lines.
460
01:13:24.828 --> 01:13:33.588
But I think we're gonna talk about similar stuff. It's just that we're gonna have continued on this trajectory for another year, and things are gonna feel pretty different because of that happening.
461
01:13:34.128 --> 01:13:41.968
I know this is the boring answer to that question. [laughs] Um, so like- No, no. I mean, you know, I, I'm a fan of things, uh, that truths that don't change because you can build and plan for that.
462
01:13:42.628 --> 01:13:51.708
And I think in media in general, in the podcast business, newsletter business, Twitter business, people are addicted to change. Like, oh, everything's breaking, everything's... No.
463
01:13:51.908 --> 01:14:04.728
Like, there are some truths that are just constants that you can plan on and build and yeah. I, I think one of the truths is that the demand for AI intelligence and smarter AI intelligence is going to be insatiable.
464
01:14:05.328 --> 01:14:11.688
Um, uh, and some people disagree that, okay, once we reach certain thresholds, then you don't need more intelligence.
465
01:14:11.928 --> 01:14:25.808
I think to that I ask people have they ever worked with or managed someone in a work environment and wouldn't press the button that they were smarter [laughs] to make them smarter or better at their job or, or would they never press that for themselves?
466
01:14:25.848 --> 01:14:28.448
And I'm not sure that that's, that's the case. Yeah.
467
01:14:28.468 --> 01:14:37.888
But I think for artificial analysis we'll keep benchmarking, uh, raw intelligence, but we also want to think about it and explore models more deeply across other axes as well.
468
01:14:37.968 --> 01:14:49.548
I think hallucination's the start of that, but we're getting into wanting to support people and understanding, okay, the behavior, the pers- personalities, uh, of the models to help people make more nuanced, uh, decisions.
469
01:14:49.708 --> 01:15:01.128
You're gonna have a personality bench. Maybe. [laughs] That is a direction that ChatGPT, OpenAI is leaning into a lot. Uh, so if you manage to solve that you should definitely talk to, uh, Fiji and Rudin. Oh, okay.
470
01:15:01.228 --> 01:15:12.512
Um, yeah. So what is gonna be included in, let's say, like, a V3 of the intelligence index because obviously you're gonna saturate In, in March Why don't we break it now? How soon is the podcast [laughs] gonna come out?
471
01:15:12.712 --> 01:15:18.952
Whenever you want. Okay. So we- we're, so we're, we're at V3 right now. So the, so the, the version that we, that's going inside is, is, is V3.
472
01:15:19.332 --> 01:15:28.912
V4 is what we're gonna call the next, you know, major update, surprise, surprise. We're gonna be adding several of the things that we've actually talked about today that we've launched over the last few weeks. Got it.
473
01:15:28.952 --> 01:15:39.392
So it's not, that's not gonna be wildly shocking, but some of the things that are most exciting is that adding GDPVal is gonna give us this general agentic performance in a really strong way in intelligence index.
474
01:15:39.772 --> 01:15:49.372
Adding critical point, the, um, physics eval George was talking about, similar to frontier math, that gives us completely new view with a brand-new data set of very, very hard research problems.
475
01:15:49.912 --> 01:15:59.132
We are gonna be using omniscience, and we are gonna be using hallucination rate. The exact ways that all those are gonna come together, um- The weighting's gonna be hard 'cause the numbers are different. Yeah. Yeah.
476
01:15:59.152 --> 01:16:07.832
We're g- we, we're gonna make sure that we don't do anything to- [laughs]... cause odd distortions and stuff that could be misleading. But- But every time you version it, you have a one-time reset of the- Exactly...
477
01:16:07.872 --> 01:16:17.632
you know? Yep. Yeah. Yep, that's exactly how we think about it. We will make sure that within each version number, that there's no, um, drift in any of the scores so that people can rely on them and reference them. Yeah.
478
01:16:17.692 --> 01:16:27.312
Um, you just have to watch out for that version number. Once it's V4.1, those numbers won't be compatible with V4. Of course. Uh, there, there is a little bit of debate over the, the accuracy of Tau Bench.
479
01:16:27.452 --> 01:16:37.372
I don't know if you s- you're clued in to what's going on. Yeah. Apparently, like, a very high number of Tau, Tau Bench tests are impossible. Potentially for the earlier versions. Tau, Tau Bench Telecom- Yeah...
480
01:16:37.452 --> 01:16:51.012
we're pretty convinced- Good. Okay... is pretty good. If anything, the only issue there is that models have got very good at doing it. And so like anything- Tau3. Yeah. Yeah. On we go. Yeah, on we go. All right.
481
01:16:51.052 --> 01:16:58.912
Well, thank you so much for, uh, providing such a great service to the industry. I, I, I'm glad to at least know you guys as before you got famous, and now, now you are famous. [laughs] So.
482
01:16:58.952 --> 01:17:01.752
Oh, look, our pleasure, and we really appreciate your support along the way. Yeah, yeah.
483
01:17:01.772 --> 01:17:08.932
Like, I, I wasn't kidding at the start, right, that it was a, um, quite material moment for us, like, when Artificial Analysis was covered on Latest Space.
484
01:17:08.952 --> 01:17:16.252
Some, some random guy in San Francisco mentions you, and [laughs]... Well, no. So I, I, I was a fan of Latest Space for, like, a year- Oh, really? Okay... before you mentioned us. All right.
485
01:17:16.272 --> 01:17:27.572
So I'd been, I'd been, I'd been listening. Um, I don't think I was, um, like, familiar with, um, like, you personally yet at that point, but, like, I was... I listened to your voice probably for many, many hours.
486
01:17:27.632 --> 01:17:34.012
Oh, okay. And so once, like, you mentioned it- It's nice to hear... and then, like, got to get to know you and, like, meet you for the first time nearly a couple of years ago, like, it was really cool, honestly. Yeah.
487
01:17:34.052 --> 01:17:35.652
So yeah. Yeah. It's great to be here.
488
01:17:35.712 --> 01:17:43.712
And thanks for, you know, being such a great member of the community and kind of spotlighting, you know, projects which don't, don't have attention and, and bringing them to your, to your audience. Yeah.
489
01:17:43.992 --> 01:17:51.872
Uh, well, actually, so it wasn't me, right? Uh, someone in the Discord dropped it in our, in our Discord, and that's... I, I rely on our community, and it kind of feeds itself, right? Nice.
490
01:17:51.892 --> 01:17:59.812
So, uh, so someone brought it to my attention. I don't know who. We should probably go back and check. [laughs] But once I saw it, I was like, "This is, this looks good. This is something I always wanted.
491
01:17:59.872 --> 01:18:09.832
I, I, I wanted to build it. I, I was too shy or dumb or lazy to build it," and you guys did, and, um, now, now it's a whole thing. So thank you for doing it. You built some really cool other stuff, like, like this pod.
492
01:18:09.832 --> 01:18:23.892
[laughs] Yeah, yeah. Totally. So thank you. That's it. Great. Cool. Thank you. Thanks. [outro music]