Episode 303: Open Models, GPU Kernels & autoresearch with Mark Saroufim: täielik transkriptsioon

the open models, it can be served in many different ways. 00:04:29.040 --> 00:04:42.480 And the there can be, and I I did not realize the degree to which this can have an like who's serving your inference for your model can have an impact on the performance, especially around things like tool calls, etc. 00:04:42.800 --> 00:04:44.319 I mean, I didn't realize that. 00:04:44.560 --> 00:05:03.680 And I I've actually heard that some labs have changed their licensing a little bit, not to make them more closed, but to prevent them from being served in a way that would create this perception of, oh, this model's so bad because I tried it on this one provider that did not have a proper setup. 00:05:03.920 --> 00:05:05.040 I mean, it's a good question. 00:05:05.279 --> 00:05:20.319 I mean, like, I like as I sort of think through this, I think there's like a bit of an like iceberg here, which is like if you're like, you know, one way you could be an inference provider is like you just have GPUs and then you just like you know, VLANSG Line and then you charge people for tokens. 00:05:20.560 --> 00:05:27.360 You know, like slightly you know deeper would be you're like, okay, I'll call a compiler and I'll like make sure CUDA graphs are applied. 00:05:27.519 --> 00:05:36.720 You know, slightly deeper is you know, maybe you write a few custom kernels here and there, you know, a bit deeper is you provide the fine-tuning service for your customers and then let them redeploy those. 00:05:36.800 --> 00:05:40.720 And so you have to write some backwards kernels for them, which typically the Chinese labs do not provide. 00:05:40.959 --> 00:06:01.839 And so, like, and then sort of the but but but what I sort of like and then and then what's interesting, even at this sort of and then you have quantization, I think it's just very controversial because like people maybe don't mind being quantized, but they'd like to know when they're being quantized, which is very bad because if you have some sort of like let's say priority cue thing going on, like okay, so so let's do not match the points. 00:06:01.920 --> 00:06:09.680 Let's say you're just like a normal user and you just hit a request to an inference uh user an endpoint, you may not even know if you're running on NVIDIA or AMD GPU, for instance. 00:06:09.920 --> 00:06:22.319 And then the numerics might diverge slightly, and then so it's not like the you're being nerfed, it's just like the numerics diverge slightly, and maybe they were only like thoroughly tested on Nvidia, and then you get like an AMD inference and you're like upset, basically. 00:06:22.560 --> 00:06:37.040 Another one is like, you know, like AMD has a bit more VRAM, and so now like the the batch size might be like a bit bigger if you're serving multiple requests, and the kernels might not be batch invariant, and so now again your numerics change, and you're like, oh, why am I getting nerfed? 00:06:37.199 --> 00:06:39.199 And then finally, you could actually be getting quantized. 00:06:39.279 --> 00:06:43.199 Like, like this is also like very problematic that like it's not sort of crystal clear. 00:06:43.279 --> 00:06:46.800 And I think why there's sort of sometimes on Twitter, like some conspiracy theories around it. 00:06:46.959 --> 00:06:48.480 People sort of say, like, oh, like at 1 a.m. 00:06:48.639 --> 00:06:50.079 Claude is quantizing us. 00:06:50.160 --> 00:06:50.639 Like, who knows? 00:06:50.720 --> 00:06:52.720 Like, it could be like, is it running on training? 00:06:52.800 --> 00:06:54.240 Is it running on like an H100? 00:06:54.319 --> 00:06:55.759 Is it running on a GB300? 00:06:55.839 --> 00:06:58.879 Like, is it like running, you know, on a CPU? 00:06:59.040 --> 00:07:04.240 Is it running with CPU offloading because it's running on an older generation A100 that happens to be the data center? 00:07:04.399 --> 00:07:07.439 And so some of the layers are offloaded and computed on CPU. 00:07:07.519 --> 00:07:12.800 So there's sort of like a lot of unknowns, but I think a lot of it in my mind is like numerics suck. 00:07:13.040 --> 00:07:22.319 And as a vendor tries to make these straight-offs to save costs, they might be creating numerics problems, and as a result, results won't be perfectly reproducible, and people might believe that they're they're getting nerfed. 00:07:22.480 --> 00:07:33.839 But yeah, I mean it's it's fascinating to me that well, and one of the things that I think is interesting here is the non-deterministic nature of LLMs means that it's very hard to tell whether something changes. 00:07:34.000 --> 00:07:44.480 And it's fascinating to me how you know, like we call it like Opus 4.8, but like the Opus 4.8 that you got at one point in time might be different from what you get at a different point in time. 00:07:44.720 --> 00:07:52.000 And you know, like in any other part of software of the software industry, like a version would have like a concrete meaning. 00:07:52.160 --> 00:07:57.519 And if you just suddenly changed the version that you were serving to your customers, like that wouldn't be cool. 00:07:57.839 --> 00:08:01.199 But in this, in this industry, it happens all the time. 00:08:01.439 --> 00:08:01.759 Yeah. 00:08:02.000 --> 00:08:20.800 And I like I I've noticed I've noticed like all sorts of just even switching like same model between like different like in for in like endpoints and providers with an Nvidia's like like a hub, like sometimes there's like a weird behavior, and like, could this be could this be in like how I'm getting routed to this model? 00:08:20.959 --> 00:08:31.040 And yeah, I I I didn't realize before I got more into this, like the degree of complexity that goes into serving inference for a particular model and how it can impact the quality that you get out of it. 00:08:31.360 --> 00:08:42.480 Yeah, I mean what I will say though, like I am sort of seeing some light at the end of the tunnel these days because so we recently had like

James Demel, like the linear algebra legend from like Lapac, come give us a talk. 00:08:42.639 --> 00:08:51.919 And like one of the things he mentioned, like I was like asking him a question in private, like about like formal verification, like, can't we just like have better guarantees of like ways of testing these kernels? 00:08:52.080 --> 00:09:11.360 Because just as a baseline for your viewers, the way a lot of kernels get tested today is basically using like absolute and relative tolerances, which basically means like let's say your output is a tensor, you iterate over all the elements of a tensor, and then you sort of compare them to like a reference to like your let's say your faster implementation, and then you're like, are they far apart from each other? 00:09:11.440 --> 00:09:16.399 And typically, if like I don't know, they're within one to the e minus three of each other, people go like ah it looks fine. 00:09:16.480 --> 00:09:18.000 But like, what do you mean it looks fine? 00:09:18.080 --> 00:09:24.960 Like basically one to e minus three is like very big, and if it's like for float 64, like that's a more major issue than if it's it's for int4. 00:09:25.279 --> 00:09:28.960 And so, like, what ends up happening is that like these values only ever increase. 00:09:29.039 --> 00:09:37.440 Like, if you look at PyTorch, for example, or most open source repos, a tolls and r tolls only ever increase because you usually get flakes to avoid the flakes, people just increase the tolerances. 00:09:37.679 --> 00:09:43.440 And we just don't have like a good way of knowing how to fix these problems because we go, like, oh, like is it a CUDA driver thing? 00:09:43.519 --> 00:09:44.639 Is it like a library thing? 00:09:44.799 --> 00:09:46.559 Is it like a new bug in the kernel? 00:09:46.639 --> 00:09:55.759 And then, like, maybe the bug is okay because the performance speed up is okay, but maybe some customers are not okay with an 8% speed up if it changes the numeric slightly, and maybe some are. 00:09:56.000 --> 00:10:05.919 Then, you know, there's like a long list of PyTorch issues that get opened up of like, well, we're trying to like I have this CPU program that doesn't match my GPU program, and then they're like, How come it's a bug? 00:10:06.080 --> 00:10:16.399 And you're like, Well, it's not a bug because turns out like it's not like Nvidia follows the entire IEEE format for like tensor cores, and like that's just like something like we have to like understand and be okay with. 00:10:16.480 --> 00:10:20.399 But I think it's very counterintuitive, like when you what when you first come across it. 00:10:20.559 --> 00:10:30.639 So so so just to bring it back to what stuff James was saying, like what James's point was that like you cannot have element-wise comparisons, like you you it's much superior to have like property-based testing. 00:10:30.799 --> 00:10:36.799 So, for instance, we're doing a QR kernel, like we just look at how orthogonal is

Q and how triangular is R. 00:10:37.039 --> 00:10:39.360 This is a much better way of doing like testing. 00:10:39.440 --> 00:10:42.799 And I think Lapac, it's like again, it's like a shoulder of giants sort of thing. 00:10:42.879 --> 00:10:44.879 They've been thinking about correctness for decades. 00:10:45.039 --> 00:10:48.960 Whereas in deep learning, maybe we've been thinking about it for like nine years and we're not very good at it. 00:10:49.039 --> 00:11:01.200 And so, this is like to me one area where we kind of need to look at what our predecessors did and then like understand, like, you know, in some sense, we know how far apart the floats are from each other, and so so so there's no like there's no secret there. 00:11:01.279 --> 00:11:07.679 And so, like, we should be able to just more rigorously think of how to set tolerances and what kinds of properties we expect our programs to have. 00:11:07.840 --> 00:11:09.519 And I just don't think we're we're there yet. 00:11:09.759 --> 00:11:10.559 Yeah, it's interesting. 00:11:10.639 --> 00:11:18.639 When I I used to work on these astrophysics codes simulating uh like binary white dwarfs that'd be rotating around each other. 00:11:18.799 --> 00:11:27.519 And one of the ways that we checked for correctness was because this code was a code where it was rotating, it was very important to conserve angular momentum. 00:11:27.679 --> 00:11:35.600 So we would just like one of the checks was you sum up with very high precision the angular momentum across the whole grid. 00:11:35.759 --> 00:11:45.600 And like if you lose any angular momentum, then like your simulation isn't respecting the most important like law of physics that you're trying to simulate here, which is like conservation. 00:11:45.759 --> 00:11:51.279 And that was just like a simple, like just from the physics like test, the property of the system that you wanted to enforce. 00:11:51.440 --> 00:11:56.240 And like, you know, yeah, like we looked at like residuals and like other errors and tolerances, but like that was like the key thing. 00:11:56.399 --> 00:12:01.120 Like as long as we're not, as long as we don't have outflows of this one property. 00:12:01.200 --> 00:12:12.480 And the interesting thing too was when you were designing that simulation, there's a bunch of different quantities that were evolving throughout the simulation, like radiation and energy. 00:12:12.799 --> 00:12:21.120 And it was very challenging, like you have to kind of pick and choose, like, which of these are we going to preserve quality for? 00:12:21.279 --> 00:12:24.879 And like we knew that this one, this was the variable that like was most important. 00:12:24.960 --> 00:12:30.720 That like if if we didn't conserve angular momentum, then the whole simulation would become numerically unstable. 00:12:30.799 --> 00:12:36.080 But there were other things where it was like, eh, you know, if a little bit of this leaks off the grid, it's okay. 00:12:36.320 --> 00:12:36.720 Yeah. 00:12:37.039 --> 00:12:46.639 I mean, like, I like to compare this to how I remember us doing in PyTorch in the Torch AO project, how we do correctness testing for diffusion model quantization. 00:12:46.799 --> 00:12:54.000 And it would literally be, okay, well, you know, before the model gave a cat, okay, we quantize, okay, still a cat. 00:12:54.240 --> 00:12:58.080 You know, like this was sort of like so this this was the way correctness testing was done. 00:12:58.159 --> 00:13:06.000 And then you're like, well, is it like it's a bit yeah, it's it's it's a bit question mark, but but like I think that was sort of the state of the maybe it maybe it's better now. 00:13:06.080 --> 00:13:10.159 I don't want my diffusion friends to flame me, but like, but at least a year ago, this is what it was like. 00:13:10.480 --> 00:13:12.879 So there was another tough question I was gonna have for you. 00:13:12.960 --> 00:13:14.480 But I'll I'll ask I'll ask a different question. 00:13:14.559 --> 00:13:30.320 So, okay, you you um you said you've never worked at a frontier lab, but you're now working at Core Auto, which I don't know a lot about, but I've like very I'm from what I do know, your mission is like so appealing, so appealing to me right now as I am in search of automating all the things. 00:13:30.399 --> 00:13:38.879 So like are are you not working you say you haven't worked at a frontier lab, but do you hope that the lab that you're at will eventually become a frontier lab? 00:13:38.960 --> 00:13:49.120 Like, what's the is the mandate of Core Auto to like develop core technology that other labs would use, or like is your goal like we're gonna build a frontier model? 00:13:49.440 --> 00:14:01.679 Yeah, I I mean I I I think these things evolve quite a bit, but yeah, I mean like I I I I I think the the the lab's like mission is to build the world's most automated lab and then figure out how to automate the rest of the world. 00:14:01.759 --> 00:14:10.960 Yeah, and I think like basically there's a few good proxies for this, which is like, of course, like you know, how quickly can we speed run towards like building like a frontier model? 00:14:11.120 --> 00:14:13.039 I think is just like a really key component. 00:14:13.200 --> 00:14:25.519 And because we're making a lot of bets on architecture research, it was based on like when you look at like the talent we have on the team, it's just really critical that this set of researchers feels productive with the kinds of work they can run. 00:14:25.679 --> 00:14:30.080 And this is kind of where I come in, which is like on kernel alum generation. 00:14:30.240 --> 00:14:43.600 And so this I think is like a very key and underrated like pre-pre-req, basically, I think of a lot of like auto research work, which is this like, hey, if you want to iterate on whether a research idea is good, but it's like four times slower than what it should be, like that's bad. 00:14:43.840 --> 00:14:48.480 You

know, one notable example that you know we worked on, Rice, was like you know, the the QR kernel. 00:14:48.639 --> 00:14:53.360 Like the the the context there is that, like, actually, maybe maybe maybe maybe I should take a step back. 00:14:53.519 --> 00:14:54.399 Let's take a step back. 00:14:54.559 --> 00:15:02.080 So, the step back is when I first started working on GPU mode, my impression was that like the bottleneck for progress was like an educational bottleneck. 00:15:02.159 --> 00:15:04.320 Like basically people just don't know how GPUs work. 00:15:04.480 --> 00:15:14.559 And if you we taught people how GPUs worked, and then we gave them sort of good problems along the way, and then made them more and more ambitious over time that people would figure out like how to use GPUs. 00:15:14.720 --> 00:15:33.679 And the interesting thing is that like basically near the end of our like and and and halfway through like GPU mode's journey, basically was about a year ago, AI systems weren't very good at writing kernels, like they're they actually they sucked, and the code they would generate would be incorrect, it would be slow, it would be sort of this hallucination between like PyTorch and CUDA and Triton. 00:15:33.759 --> 00:15:35.679 It just like was wasn't like super coherent. 00:15:35.919 --> 00:15:42.799 And then, like, earlier this year, basically, sort of the the the great, you know, the like the great lock-in of Christmas. 00:15:42.960 --> 00:15:57.519 We came back from this kernel competition on NVFP4, which was like one of our most popular competition yet that we worked on with Nvidia, and like the fourth place winner was like, Oh hey, like I've never written a kernel day in my life, but I have the fourth fastest kernel in the world for gems, and I'm like, What? 00:15:57.679 --> 00:16:06.240 And then like a few days later, someone's like, Oh, like I'm a high school teacher, I don't even know what a kernel is, but like I have the fifth fastest kernel for like a dual gem problem. 00:16:06.320 --> 00:16:07.039 Like, thanks, Mark. 00:16:07.120 --> 00:16:12.960 And I'm like, okay, like this is bizarre to me because it and generally it felt like systems was this very gatekeepy thing. 00:16:13.039 --> 00:16:14.480 People are like, No, you're incorrect. 00:16:14.559 --> 00:16:18.559 People are just like very grumpy, you know, the sort of traditional vibe of the C programmer. 00:16:18.720 --> 00:16:27.759 Sorry for the sort of casual insult, but but but you know, it was like not the most like friendly bunch because like there's like sort of a lot of rigor and a lot of ways you can shoot yourself in the foot. 00:16:27.840 --> 00:16:32.159 And so it makes sense that the community would be sort of more inaccessible and gate, like a lot of things. 00:16:32.320 --> 00:16:40.879 So here we are, we're now in a world where like turns out like AIs are better at teaching how GPUs work than I was as an educator over two years in GPU mode. 00:16:40.960 --> 00:16:46.000 Yeah, and so like to me, it was like, okay, well, what like what what do we do now, right? 00:16:46.159 --> 00:16:52.159 And so looking at it, I'm like, okay, well, it seems that people just maybe aren't sure which problems are worth accelerating. 00:16:52.320 --> 00:16:58.879 And so this is kind of like now, like my duty, I feel as an educator, has become like telling people what are important problems to accelerate. 00:16:58.960 --> 00:17:10.000 And so the linear algebra kernels came about because we needed those for our research on like optimizers, specifically like a shampoo optimizer that like Ron, one of our one of our researchers, like very passionate about. 00:17:10.160 --> 00:17:20.720 And so when I look at this, I'm like, okay, well, as long as we can sort of like very clearly formulate the problem and have like a lot of facilities for reward hacking, like we can make fairly consistent breakthroughs and systems. 00:17:20.880 --> 00:17:34.960 And once we do that, now like my research buddies can sort of like come up with all sorts of like cursed ideas, and instead of having to like recruit me, which traditionally they would need to do, like, I would need to go look at their code, understand the math, like write the kernel. 00:17:35.119 --> 00:17:43.839 Now, like I can just basically give them tools for anti-word hacking, and then they can just like go cook, and then they're very productive and they never need to talk to me. 00:17:44.000 --> 00:17:46.880 And this is great because then I can think of like more general problems. 00:17:46.960 --> 00:17:49.519 And I and I'm very passionate about this space for a few reasons. 00:17:49.599 --> 00:17:56.960 Like, I think it applies to like not just kernels, but like you know, inference engines, systems code, you know, what will repos like PyTorch look like in the future. 00:17:57.680 --> 00:17:58.880 There's so many applications. 00:17:59.119 --> 00:18:02.319 Like, you know, you can use auto research to optimize for performance. 00:18:02.480 --> 00:18:13.920 You can also use like if you have the same setup, the same guardrails, the same report reward hack preventions, you can use it to simplify code, you know, like minimize, minimize the diff without regressing performance. 00:18:14.079 --> 00:18:16.160 You can use it to increase code coverage. 00:18:16.319 --> 00:18:21.359 Like anything that's a metric that you can like measure and quantify, you can then like automate. 00:18:21.440 --> 00:18:25.119 And all you need is like to have the guardrails and the correctness checks in place. 00:18:25.279 --> 00:18:32.240 And yeah, I think like when I talk to people about auto research, I think they think I'm solely talking about just like performance optimization. 00:18:32.319 --> 00:18:37.519 They're not realizing just like how many anything that can be an iterative, measurable process, you can automate. 00:18:37.680 --> 00:18:44.160 And then like you can build, you know, these processes,

you can chain these different loops together and and build these workflows. 00:18:44.240 --> 00:18:46.799 And yeah, it's so it feels so incredibly powerful. 00:18:47.279 --> 00:19:14.880 I mean, yeah, I mean, at least like my observation here has been like I I I just find the competitive leaderboard format to be a very powerful framing because like so, first off, like you are like the way I think of it, like very broadly speaking, is that like you set an eval, and the eval is still expensive because the eval ultimately needs to run on real GPUs, so it's like a real cost, but it's actually negligible relative to the token cost that people end up paying to like do really well in these things. 00:19:15.279 --> 00:19:20.400 And and so actually, yeah, Brad, I don't know if you're comfortable sharing like your most recent token expenses. 00:19:20.880 --> 00:19:27.359 Well, I I I've I've spent 300 billion tokens, I think, since the start of the year, and uh there's various ways of accounting for the cost. 00:19:27.519 --> 00:19:42.000 But the the thing that I when I first got into this, I too was worried about the cost of running on the GPUs, of renting the GPUs, and I d definitely it is the case that I no longer worry about the cost of running on the GPUs because relative to the token cost, it is it is trivial. 00:19:42.160 --> 00:19:59.680 But I if I had to guess, I would say that I think today for for most problems, if you want to write like a speed of light kernel, you it's you know, in the thousands of dollars of inference at least, I think to do it to really like if you just want like a decent one, you know, maybe you can get away with less. 00:19:59.759 --> 00:20:11.519 But but the thing that's the thing that I'm starting to see changing, and one thing that was an interesting thing from this weekend, is I'm starting to see people getting results from the open models that are much cheaper. 00:20:11.680 --> 00:20:21.039 Like one of the the the person from our group at the EuroPython Sprint who has the best score on the Chelesky problem, it was actually one of the the competitors. 00:20:21.119 --> 00:20:24.240 Uh there were two me and one other person who I think had the most experience. 00:20:24.319 --> 00:20:31.119 Um the other fellow I think didn't have as much experience with CUDA kernels, but he was using a variety of models, including some open models. 00:20:31.200 --> 00:20:44.640 And I haven't checked which of which of the models he's been using is the uh is is the the winner, but I know I've started to see been able to get meaningful results with open models, which could maybe lower the cost of this a lot. 00:20:44.799 --> 00:21:00.160 I think one of the problems is you know, like how do we how do we if it if you need to have a ridiculous amount of inference to do well in these competitions, how do we like seed this out to researchers and people so that they can learn, so that we can build up this skill set? 00:21:00.319 --> 00:21:02.480 It

seems tough with the current costs. 00:21:02.880 --> 00:21:05.839 Yeah, I mean, I'm a bit more optimistic than you are. 00:21:06.000 --> 00:21:07.680 Like here's how I would think through this. 00:21:07.839 --> 00:21:11.759 Like basically, like let's say you're trying to serve like uh an MOE on a node. 00:21:12.000 --> 00:21:20.720 Basically, if you're doing like very small batch size inference, then like only a few of your experts are activated, and so the inference ends up being like quite quite fast. 00:21:20.799 --> 00:21:34.400 But if you're like medium batch size, then like probably most of your experts will be activated, and so but then if you have a lot of like users and a lot, like a lot of requests, it might be as efficient to do that than to do the medium request scale, because anyway all your experts are activated. 00:21:34.559 --> 00:21:51.759 And so it's like one of those things where like it seems that like these things do benefit from like economies of scale, but economies of scale not in terms of like I have like a 10,000 GPU cluster and I'm intelligently writing things to them because that's just like not what I think a good chunk of inference providers are actually doing. 00:21:51.839 --> 00:21:59.839 Like, I think they're doing much more managed deployments, I think it's like simpler to price and finance, like you just pay them some margin on the price of a B200. 00:22:00.400 --> 00:22:02.000 And then you get like tokens back. 00:22:02.240 --> 00:22:15.279 So again, like when I when I look at this to me, it's sort of it screams like, well, as long as the emphasis is on one node, and if there's like a few tricks that matter, like basically coupographs, a few kernels, you know, quantization. 00:22:15.359 --> 00:22:24.880 Like if if that's it and there's no more secret sauce, then like why not just have like a node per small pod or team? 00:22:25.119 --> 00:22:27.759 Like to me, that seems like pretty economically feasible. 00:22:28.000 --> 00:22:37.839 If the models get to three trillion and stuff, and this is what we're running, like maybe not, then you're basically because you know, one thing I like to mention to people is like, you know, distributed computing never became popular at home. 00:22:38.000 --> 00:22:45.200 You know, like we have very powerful like laptops and phones, but we don't have like it's very rare for people to have a cluster at home that communicates. 00:22:45.440 --> 00:22:48.880 So that to me feels like it'll always be like a niche thing. 00:22:49.039 --> 00:22:59.680 But yeah, I mean, I'm more optimistic, and and I think like there's this nice virtuous loop where like if the task is make the inference engine faster, you could presumably imagine having something really fast and custom made. 00:22:59.839 --> 00:23:02.960 It's like one thing I think would be pretty cool as a GPU mode working group. 00:23:03.039 --> 00:23:06.720 So if anyone's interested in tag teaming on that, like I'll probably I'll probably work with them on it. 00:23:06.960 --> 00:23:08.960 Uh a working group on what specifically? 00:23:09.359 --> 00:23:19.519 Basically, having a very like basically like a starter kit for a very cheap inference service that you can deploy at a company with like auto research that profiles the requests. 00:23:19.599 --> 00:23:22.079 So like something like that I think would be quite interesting to look at. 00:23:22.480 --> 00:23:40.880 My my dream is my my dream actually is a world in which when you get your next NVIDIA system, like and you install like you know, CUDA, there's like a little tool there that's just like, hey, like give me your existing CUDA code and I'll just like port it to this GPU and make it fast. 00:23:41.039 --> 00:23:46.319 So I I I won't say the numbers on air, but but but I can show you guys. 00:23:46.559 --> 00:23:48.799 Here's my uh here's my spend. 00:23:48.880 --> 00:23:53.440 I thought it was gonna go up a bit, but it's not it it's in the six figures. 00:23:53.759 --> 00:23:54.880 Can say that at least. 00:23:55.039 --> 00:24:15.279 But so this is the amount, and then direct from Claude is direct from the anthropic subscription is 50 billion tokens, and then direct from the open air subscription is 63 billion tokens, and then from our internal inference hub is 162 billion tokens. 00:24:15.440 --> 00:24:27.359 So that's like maybe it's not 300 billion yet, but I I don't know that this is entirely uh let's see, that's like one one one three and okay, yeah, it's close to it's close to 300 billion. 00:24:27.440 --> 00:24:28.960 That's the that's through the start of the year. 00:24:29.119 --> 00:24:44.400 But also I think I think one of the other things is that the the token cost has gone down dramatically over time, and that's just been as I've learned the things that are wasteful that are being done during the process. 00:24:44.559 --> 00:24:50.559 And I I've I've started to see like increases in token efficiency. 00:24:50.799 --> 00:24:51.200 Yeah. 00:24:51.359 --> 00:24:51.839 I don't know. 00:24:51.920 --> 00:24:53.039 Connor, what what are you at? 00:24:53.200 --> 00:24:59.759 Yeah, Connor at one point was number seven on the internal like uh uh codex leaderboard. 00:25:00.720 --> 00:25:01.279 I don't know. 00:25:01.519 --> 00:25:07.599 They don't they don't show us the leaderboards anymore, and uh I don't really I don't really track. 00:25:07.759 --> 00:25:27.359 I mean I do a lot of auto research stuff too, but um you should tell Mark about you should tell Mark about the I I I think most people now have like stopped talking about the limitations of models, but like you you hear like oh it's not good for X, it's not good for Y, and it's like no, like it's good for everything now. 00:25:27.519 --> 00:25:33.920 It just you haven't set up the correct, you know, I I I don't know if you want to call it harness or agentic loop system. 00:25:34.240 --> 00:25:35.119 Well, yeah. 00:25:35.359 --> 00:25:35.920 Mark

left. 00:25:36.160 --> 00:25:38.799 I don't I assume he's listening, he's gotta say. 00:25:39.200 --> 00:25:40.319 Well, you know, it's interesting. 00:25:40.400 --> 00:25:50.079 I so I I I think most people have seeded that for these sorts of tasks, for a lot of tasks that it that, you know, yes, AI can do it. 00:25:50.240 --> 00:26:04.160 The thing that I still hear is like, yeah, but design or style, or like somebody give me an example of like, well, you know, you know, a model, an agent wrote a new function instead of using an existing function. 00:26:04.240 --> 00:26:09.440 Like it should, there were, it should have like tried to leverage some existing function and like lift it. 00:26:09.519 --> 00:26:12.799 And and it's like an and like how do we prevent that without human review? 00:26:12.960 --> 00:26:25.599 And and I was talking with some other people about it, and somebody was like, well, you know, you could like it's not easy to build like a machine checkable way to check for style for that, but for that particular case, you know, we probably could. 00:26:25.759 --> 00:26:35.519 You could build a tool that looks at like that like parses the AST of the language and then like compares the similarity of different like entities in the code. 00:26:35.599 --> 00:26:44.720 And if your diff introduces a new thing that's very similar to some other thing, then you could build a deterministic tool that says, like, hey, you should have used this other tool. 00:26:44.880 --> 00:26:54.400 Like, maybe we just haven't been trying hard enough to turn all of the like design sense and code style things into machine checkable rules. 00:26:54.480 --> 00:26:55.039 Maybe we can. 00:26:55.279 --> 00:27:10.640 This is kind of what makes working at a like a a Frontier lab interesting, is that like indeed, I think as a user of these models, like probably your best bet is coming up with like linters on top, like basically, and those linters could be other AI systems or they could just be like actual code. 00:27:10.799 --> 00:27:33.440 But like to me, when I when I hear the problem of like, well, the AI is used too many trick subblocks, or they like reinvent the wheel, a lot of those are RL problems, for instance, like which is basically if the reward is just solved the task, yeah, you know, like code reuse doesn't really matter, but like and defensibility like matters for RL because otherwise you get like syntax errors and those are bad, uh, and they like shoot down your reward. 00:27:33.599 --> 00:27:39.599 And so, like, I think there's quite a bit of interesting research to do, like, with reward shaping, with like adversarial training. 00:27:39.759 --> 00:27:52.880 I think that make these things really interesting, but but it's like one of those things where I expect, like, in the same way, like hallucination was like a big problem until like just seems like it's not right now, like post-thinking models and like web search. 00:27:53.039 --> 00:27:58.720 I think

a lot of these problems people are complaining about around code reuse will will be solved by like AI systems even without a harness. 00:27:58.960 --> 00:28:00.720 Like, they'll just swallow their harness over time. 00:28:00.880 --> 00:28:07.039 It's just that the harness lets us explore their current capabilities without waiting for like a new model release. 00:28:07.200 --> 00:28:21.680 Yeah, and similarly, like I mean, you could like come up with a kernel with like thousands of dollars, but eventually, you know, like if you're like a smart human, like presumably you're not going through 30 billion tokens of like KUDA examples before you come up with like a good kernel. 00:28:21.839 --> 00:28:23.359 Yeah, but you do this much more efficiently. 00:28:23.440 --> 00:28:26.240 So I think there's also interesting stuff to be done on the search frontier. 00:28:26.400 --> 00:28:29.759 And it's all like research I'm quite excited about and stuff, you know, I'm working on. 00:28:30.079 --> 00:28:30.720 Yeah, yeah. 00:28:30.880 --> 00:29:13.839 I I do think there's ways like to to I I don't feel personally like I have fully cracked the search process, like how to structure the the you know the orchestration and like how to guide a model to actually in particular the thing that I think remains a challenge is like what do you do, how do you prevent a stall, and and how do you you you both want to have depth in a search, but you want to have breadth too, because you maybe you've reached a local maximum, you know, and and the thing that I still run into and and I've now heard lots of other people run into is just sort of sometimes you reach this point where you're on the search. 00:29:14.000 --> 00:29:20.559 This is really just for things where you're trying to really write the speed of light thing, where you your automated process stalls out eventually. 00:29:20.640 --> 00:29:30.319 And I think part of it is if you don't shape the the reward and the metric right, then like it can end up writing like spaghetti code. 00:29:30.400 --> 00:29:38.079 And like once it's written like 10,000 lines a slop and it's got lots of code that's like dead code in there, then it becomes hard for it to like evolve the code. 00:29:38.319 --> 00:29:41.759 And uh, you know, at some point, like maybe it makes sense. 00:29:41.839 --> 00:29:46.160 Like if I was a human, at some point I might go and start fresh. 00:29:46.319 --> 00:29:49.759 And it's like, how do you properly guard guide the model? 00:29:49.839 --> 00:29:52.720 Like that it's time to go try a new approach. 00:29:53.039 --> 00:29:59.519 So so the the analogy I think of is like it's almost like it's gonna be a weird analogy, so bear with me. 00:29:59.599 --> 00:30:05.519 It's almost like let's say you had a light bulb and you're like it's a lit light bulb, and then you're throwing it in like a dark part of a room. 00:30:05.759 --> 00:30:12.960 To me, this is how I feel like auto research helps me, which is basically I

have like an idea, and I'd like to explore the limits of that idea, whether it's good. 00:30:13.119 --> 00:30:25.839 But in terms of like idea generation, like what's been more helpful for me honestly has been like reading old books, which is just like you know, I'm sure these are probably already in the corpus, they've been scanned or what have you. 00:30:26.000 --> 00:30:40.160 But but but but it's like one of those things where like you know, despite so many people doing kernel LM testing, for instance, I wasn't aware of any prior work that does property-based testing in the way like Lapac does. 00:30:40.240 --> 00:30:44.720 So like there seems to be like a basic read the literature aspect here. 00:30:44.799 --> 00:30:54.960 Now, yes, you could tell the AI read all the literature and do this, like yes, but but but but but but but I think like it's still quite powerful to have a tool to explore the limits of how good an idea is. 00:30:55.119 --> 00:31:02.319 And then you know, as you sort of get into reward shape shaping, like it's basically a conversation between you and the AI to figure out like what you actually want and if it's desirable. 00:31:02.640 --> 00:31:12.079 I guess I I may be I may be like to an extreme here where I like almost all of my work is like completely unsteered. 00:31:12.160 --> 00:31:16.319 Like I don't want to give the model, I don't want to be the one that comes up with the idea. 00:31:16.400 --> 00:31:18.720 I want the model to be the idea factory. 00:31:18.960 --> 00:31:36.319 And I I have both noticed and fear giving it specific specific steering or specific ideas or instructions because if I tell it, if I give it an example of like five things, like if I if you if I put in a skill an example of five things, it's gonna pay a lot of attention to those five things. 00:31:36.480 --> 00:31:42.720 It's not gonna try five things like those things, it's not gonna necessarily use this as an example, it's really gonna focus on those five things. 00:31:42.960 --> 00:31:56.319 And if I, you know, if I'm truly searching for like a frontier kernel for something, uh I have a bunch of biases and and I don't have the capacity to generate ideas at the rate that it does. 00:31:56.480 --> 00:32:02.160 And so I feel like it's very interesting to just let it unprompted search on its own. 00:32:02.400 --> 00:32:02.960 Let it cook. 00:32:03.119 --> 00:32:03.519 Yeah. 00:32:03.759 --> 00:32:07.039 So I I will say that like I do have some data for this. 00:32:07.200 --> 00:32:08.319 It's a bit out of date now. 00:32:08.400 --> 00:32:10.079 This is like February-based data. 00:32:10.240 --> 00:32:16.480 But at the time I remember when we were working on the Nvidia problems for NVFP4, the we were releasing solutions per problem. 00:32:16.640 --> 00:32:19.839 Like whenever a problem concludes, we released a solution, and they're all NVFP4 problems. 00:32:19.920 --> 00:32:24.240 And before our competition, there was just almost no NVFP4 data

on the public internet. 00:32:24.559 --> 00:32:33.759 But then when Gowners released his NVFP4 Gem V, which is a very high quality gem v, in that problem, almost no AI system was in the top 10. 00:32:33.839 --> 00:32:35.599 Like no AI system in the top 10. 00:32:35.680 --> 00:32:48.480 But then as soon as they had Gowners' example, which is oh, here's the minimal way of how you can use NVFP4 correctly, then all of a sudden everyone reused that trick and figured out how to already auto-research it for gem for gem, dual gem, group gem. 00:32:48.799 --> 00:32:51.759 And then the AIs were doing way, way better basically after that. 00:32:51.920 --> 00:32:59.119 But they needed the like if the token is truly rare in some sense, like a new hardware feature, it hasn't done this work. 00:32:59.200 --> 00:32:59.920 But but but you're right. 00:33:00.000 --> 00:33:05.359 Like, I mean, you mentioned earlier, like, why can't you just plop an agent and then have it port all your code? 00:33:05.519 --> 00:33:16.720 Well, you know, like one extreme version of this is you just like plop the agent, and I don't know, it figures out like the ISA and it figures out that it should append the cutlass and the string of your kernels, and it should figure out all these things. 00:33:16.799 --> 00:33:20.720 But like I think we all know aspirationally that this is where we'd like to get to. 00:33:20.799 --> 00:33:29.119 It's basically somewhere in between a performance engineer slash reverse engineer slash like it's sort of like these archetypes are becoming like one and the same. 00:33:29.200 --> 00:33:32.559 And if we could get like AIs to do that, I feel like that's it. 00:33:32.640 --> 00:33:34.559 Like basically kernel writing would be solved. 00:33:34.720 --> 00:33:34.960 Yeah. 00:33:35.200 --> 00:33:44.880 Be sure to check these show notes either in your podcast app or at adspthepodcast.com for links to anything we mentioned in today's episode, as well as a link to a GitHub discussion where you can leave thoughts, comments, and questions. 00:33:45.039 --> 00:33:45.599 Thanks for listening. 00:33:45.680 --> 00:33:47.440 We hope you enjoyed and have a great day. 00:33:47.759 --> 00:33:52.799 Low quality, high quality, that is the tagline of our podcast. 00:33:52.960 --> 00:33:53.920 That's not the tagline. 00:33:54.079 --> 00:33:56.799 Our tagline is chaos with sprinkles of information. 00:33:57.759 --> 00:34:04.160 And I actually know while I'm taking a little digression, I am baffled on what's going on with my internet right now. 00:34:04.319 --> 00:34:06.720 Supposedly, like my Wi-Fi is down. 00:34:07.200 --> 00:34:15.039 Technically, the there's a green light on the deco, but like I've been trying to like Google stuff while we've been chatting and like the internet doesn't work. 00:34:15.199 --> 00:34:21.039 And I had a banner at one point on Teams that said like Teams is not able to verify your identity. 00:34:21.280 --> 00:34:23.119 But like so, I don't actually know. 00:34:23.199 --> 00:34:30.800 Like somehow I'm getting Wi Fi, but there m there may be there's some like authentication thing that's like failing, which is actually what's causing the Wi Fi to fail. 00:34:30.960 --> 00:34:36.159 Anyways, I'm here somehow seeing and hearing you guys, so we'll keep going.