The Inference Engineering Masterclass — Philip Kiely & Ali Taha, Baseten

WEBVTT
1
00:00:00.000 --> 00:00:14.700
[on hold music] Okay, we're here in the studio with, uh, Philip, uh, old friend from, from, uh, Inference Engineering, the book, as well as Base10 and, uh, everything that you've done-- you and I have done before, as well as Ali.
2
00:00:14.740 --> 00:00:22.840
Welcome. Pleasure to meet. Waterloo intern. Waterloo intern, always. [chuckles] When did you get Waterloo intern as a- As a handle? Oh. Handle. I think the rebranding happened, like, mid-March.
3
00:00:23.080 --> 00:00:30.080
When I saw it was open, I was like, "I have to take it. Up for grabs." The problem is that Ali is really good at his job and is not gonna be an intern much longer.
4
00:00:30.080 --> 00:00:37.780
[chuckles] So we have to figure out, you know, who's gonna get the handle. Well, I'll pass the torch over to the next- Ohhh. Okay. It c- can be... Like, you just pass it to another Waterloo grad.
5
00:00:37.820 --> 00:00:44.080
To another Waterloo intern. No, bruh. Yeah. Intern. Intern, yeah. And, and no, I- You've gotta get an intern from Waterloo... I'm just- Right, right, right, right. Yeah, I've gotta get an intern from Waterloo. Yeah.
6
00:00:44.120 --> 00:00:52.240
Right, right. But they have to follow the path- Oh, it could- but it could come from Base10, so it's, like, whoever Base10 gets from Waterloo- Right... has the title of Waterloo- It stays in the ecosystem. Exactly.
7
00:00:52.240 --> 00:01:00.000
[laughs] You know, halfway through the internship, you either get it or you're out. [laughs] You should also do, like, a big graduation ceremony where you change the handle- Right. Just say- For everybody.
8
00:01:00.120 --> 00:01:10.180
[laughs] I mean- Yeah... you guys are good at ceremonies, clearly. You know, we had a nice launch of the book, very successful. But before we get into all that, I wanna a- start off with a fun question for you. Okay.
9
00:01:10.640 --> 00:01:23.440
You're expert inference engineer. What happens when I send a long query, say, 200,000 tokens into Base10's inference? What's the process of query through GPU model, routing, balancing, all that?
10
00:01:23.840 --> 00:01:32.660
What is all the stuff that we don't think about? With a long query specifically, the first thing that I'm gonna ask is, "Have you sent me this query before, or at least part of it?"
11
00:01:33.320 --> 00:01:37.560
Um, and I really hope you have, because it's gonna be a lot easier for me and a lot cheaper for you.
12
00:01:38.180 --> 00:01:49.340
So the first thing that we're gonna look at is some kind of cache-aware routing, where we're going to see we probably have a number of instances, a number of replicas up serving whatever model you're hitting.
13
00:01:49.720 --> 00:02:02.740
Uh, we want to send this one to something with, number one, available prefill workers, and number two, ideally some cached input already there so that we can skip prefill on at least part of these 200,000 tokens.
14
00:02:03.360 --> 00:02:14.320
Um, if you're doing 200,000 tokens, it's probably coding or a multi-tone agent or, or something where you would expect to have that cached. Um, if you don't, we're gonna have to send it to a prefill worker.
15
00:02:14.540 --> 00:02:32.060
Um, we've, at least on certain models, disaggregated prefill and decode, um, so you're going to have one set of GPUs that's solely going to process the input, create that KV cache, and get you your first token, and then that's going to be passed over to a separate set of GPUs, which is going to run decode.
16
00:02:32.140 --> 00:02:39.160
We're going to iteratively make those tokens. Um, we're probably going to have some kind of speculator model in front of that.
17
00:02:39.440 --> 00:02:47.960
I'm going to assume that you're doing coding, and because of that, our speculator model, which assumes you're doing coding, is gonna have a high, uh, draft token acceptance rate.
18
00:02:48.520 --> 00:02:52.660
If I'm wrong and you're asking me to summarize every Harry Potter book, uh, it's gonna be slower.
19
00:02:53.560 --> 00:03:07.320
And then we stream that output to you and, uh, account for it, um, charge you, you know, some number of couple of pennies and say, uh, "Hey, would you like to send another one?" Except Base10 doesn't charge by pennies.
20
00:03:07.560 --> 00:03:18.840
Well, yeah. We, we charge- [chuckles] We-- I'm assuming that we're talking about the public model APIs. If you are, you know, s- setting up a, a dedicated deployment, then yeah, it's not pennies. [chuckles] Yeah.
21
00:03:19.040 --> 00:03:30.920
I mean, uh, one of the key differentiators when I was talking with, uh, Base10 initially was that, uh, actually, people who want very, very high volume just need to rent by the box, 'cause then it's up to you to figure out how to saturate the box.
22
00:03:31.260 --> 00:03:38.660
And more often than not, it's, like, way cheaper if you're pushing, like, m- millions of tokens per hour, if you just pay per hour instead of pay per token. Yeah. They, they do.
23
00:03:38.720 --> 00:03:51.240
I think that we've increasingly seen a lot of demand for the sort of pay-per-token APIs, just because everyone wants to try open models, and then once they find a use case that's really sticky, uh, then they, they move over to dedicated.
24
00:03:51.280 --> 00:04:01.240
Is there a best practice on when it's time to swap over? Couple reasons. Yeah, reliability, that's a big one, right? Like, if they have a very specific use case, they want you to train something specifically for them.
25
00:04:01.280 --> 00:04:08.340
Like, they want their own SpecDeck, for instance, for their own traffic. SpecDeck is speculative decoding. Speculative decoding, yeah, yeah, yeah. Um- You have to say. [laughs] Sorry.
26
00:04:08.340 --> 00:04:16.240
Like, speculative-- the way Spe- like, basically, if you have a, a huge model, right? And so the model is going to be generating one token at a time every single turn, every single forward pass.
27
00:04:16.779 --> 00:04:22.900
So we attach, like, this little, like, kind of parasite, like, this, this layer that goes on top of the model, and this model just has to predict.
28
00:04:23.120 --> 00:04:35.960
It does three very fast autoregressive forward passes, and it will dr- predict, like, you know, three certain tokens, and then you do one forward stage over the entire original model in order to see if those predictions were correct or not, and then you accept them or you reject them.
29
00:04:36.460 --> 00:04:47.740
Now, this draft model is, is traffic specific, so if you, like, uh, like, um, you know, Philip said, if you're summarizing Harry Potter books, I can train exclusively that draft model on Harry Potter books, and I can guarantee you that I'm gonna accept the three tokens every single time.
30
00:04:48.460 --> 00:04:53.100
And so with that case, I increase your decode speed. I wouldn't be able to provide this to you if you're a shared endpoint- Yeah...
31
00:04:53.100 --> 00:04:57.100
'cause I have no idea if you're doing Harry Potter, if you're doing coding, if you're doing English. Like, we don't know.
32
00:04:57.820 --> 00:05:06.540
Um, also, there was a thing in the book that mentioned that if they really cared about a specific threshold, chapter four, I think. Do you remember that? Yeah.
33
00:05:06.600 --> 00:05:17.200
The, the things that you can do is you, you can set, you know, a specific, like, batch sizing, a specific, like, parallelism strategy if you're trying to optimize for, like, throughput versus, uh, latency.
34
00:05:17.760 --> 00:05:26.120
Um, you can, you know... Maybe, maybe a NVF full quant doesn't pass your benchmarks and you wanna run a model at higher precision, you could do that.
35
00:05:26.500 --> 00:05:40.260
Um, there's just a bunch of reasons why you might wanna have your own endpoint and, and the, the biggest one, of course, just being, like, you don't have to deal with someone else doing 100 million of tokens of benchmarking traffic at the endpoint when you happen to be trying to serve your users.
36
00:05:40.700 --> 00:05:50.580
Yeah. I think one thing that is... That, that is a classic journey. You know, like, uh, it's basically people is asking the, what happens when you type Google into the browser.
37
00:05:51.320 --> 00:05:57.680
Um, tool calling, is that just, you know, you're str- you're generating JSON or is there more complication beyond that?
38
00:05:58.100 --> 00:06:08.084
Certain customers that we have, they have their own post-trained models, and so they, they demand a tool calling that's not just, like, you know, parse a file or, you know, go find The weather.
39
00:06:08.224 --> 00:06:11.424
It's, it's something that's very specific and you have to do post-training on this.
40
00:06:11.544 --> 00:06:21.984
And if the post-training on the model is not good or if the quantization after the post-training to get the inference to be fast, the model will struggle reading the JSON file and reading the tool calling.
41
00:06:22.484 --> 00:06:30.124
But it doesn't re- it doesn't require its own like sandbox. It's not like it's going to use that tool calling to like, like escape a sandbox or like it doesn't have to be contained.
42
00:06:30.164 --> 00:06:39.304
It can just be a normal dedicated deployment. The challenge with tool calling more and more seems to be that the companies want certain tool calling which, which is a very sensitive thing to train.
43
00:06:39.904 --> 00:06:47.384
And because you're dealing with all of the JSON outputs, if it doesn't like close the end of the request in a very certain manner, you end up with a model that did the tool calling and like the thinking.
44
00:06:47.584 --> 00:06:57.164
And so as a result of that, it didn't see the result and just hallucinated the result as it decoded. That seems to be the, the most challenging thing with tool calling, not really the sandboxes model. Yeah.
45
00:06:57.304 --> 00:07:05.864
That's a challenge on the training side and then on the influence side, there's work that you can do to scope the possible output.
46
00:07:06.484 --> 00:07:23.384
So we published this actually at this point close to two years ago, um, the, the solution to this problem which is you basically make a state machine and you use that to constrain the output to a specific format.
47
00:07:23.444 --> 00:07:31.504
So this is the structured output problem. If you remember back in- Yeah... back in the day- The specific grammar is, uh- Yeah, exactly... GML had this thing. Yeah.
48
00:07:32.084 --> 00:07:40.304
So it's, it's like the, the old school like make sure this is only JSON, return only JSON or my- Yeah... grandma's gonna die type of prompt. Is it BNF grammar?
49
00:07:40.984 --> 00:07:52.564
At some point OpenAI had released a thing that was like, yeah, if you want to constrain your output, write BNF grammar, uh, back as NOR. In our influence system, it's just a, yeah, specified output format.
50
00:07:52.904 --> 00:08:02.664
Um, and, and you get the, the guarantee that your output's gonna be structured along that format. Um, and so applying that to, to tool calls can like help cut down on...
51
00:08:02.784 --> 00:08:11.844
Obviously you can still call the long tool or call no tool. It doesn't solve the, the certainty problem- Mm-hmm... but it at least solves the output structuring problem within- Yeah... tool calls.
52
00:08:12.344 --> 00:08:22.744
And MCP is just another form of tool, right? I guess- Yeah... as far as- Exactly... there's no special thing there. The thing I'm always like explaining to people is the LLM is actually not capable of doing anything.
53
00:08:22.844 --> 00:08:32.644
It's only capable of making suggestions of what to do and then tho- if those suggestions are formatted in a certain way and applied to a system that knows what to do with them, then an action occurs. Yeah.
54
00:08:33.064 --> 00:08:38.024
Part of the fun stuff is, you know, this is solved outside of tool calling too. Like in an agent loop- Mm-hmm...
55
00:08:38.404 --> 00:08:45.684
if the output is not correct or you're right, like reasoning, uh, tool calling was done in the reasoning sale just be like, "Oh, I don't know what to do. Let me just try again."
56
00:08:45.964 --> 00:08:57.124
And you know, it, it might get there after a few tries. Um, and on your point of training, sometimes this is harder in smaller models, so you don't have the same exact quality output- Right...
57
00:08:57.144 --> 00:09:03.864
when you just swap from a big model, right? Yeah. I, I will say that, um, before we-- I, I think we need to go back to inference engineering [chuckles] proper.
58
00:09:03.884 --> 00:09:14.364
[laughs] But, um, I had expected that something would replace JSON because it's hard to stream JSON 'cause JSON must be complete and you must have open and close brackets and everything.
59
00:09:14.464 --> 00:09:18.304
So it's hard to parse something or validate something while it's being streamed.
60
00:09:18.804 --> 00:09:30.144
Uh, so people invented all sorts of things that are like, uh, I, I forget the, the name of some of these alternatives, but it's basically something like TOML, something like YAML. Um, but JSON seems to be dominant still.
61
00:09:30.664 --> 00:09:40.904
The JSON outputs aren't that long, right? Like I guess you could have a long... 'cause tool calls also contain the arguments in them and, and perhaps for a certain tool you might pass like a very long argument.
62
00:09:41.564 --> 00:09:53.904
But my impression of the sort of median tool call is that it's a relatively small number of tokens, right? So I would expect that speculators are generally fairly good at something as formatted as JSON.
63
00:09:54.604 --> 00:10:01.724
Um, and so you would have like a pretty fast decode step there and, and that the streaming wouldn't be as valuable, but may- maybe I'm wrong about that.
64
00:10:01.764 --> 00:10:08.384
I think you're also bounded by the software or that the model is gonna integrate with if the software is built with JSON for the tool calls or if the company that you're...
65
00:10:08.404 --> 00:10:18.204
You know, if your customer says that this is how our software works and the- our tools are interfaced with JSON, you can ask them to like, you know, change their, their software and, and say like, "Yeah, this is gonna be better for the model."
66
00:10:18.944 --> 00:10:28.104
Y- but like with the wet training shouldn't be that much of a, a difference. Also more profitable if it outputs more tokens probably. [chuckles] Depends on your business model. [laughs] You know, it really depends.
67
00:10:28.244 --> 00:10:40.144
Uh, but I will say that, you know, as a writer with like experience a lot with AI-generated output, I do try to move from text to JSON text- Mm... which is very long JSON, right?
68
00:10:40.204 --> 00:10:48.124
Like, like there's paragraphs in every field because I'm trying to structure it, right? Right. I want you to first make factual statements, then make opinions- Mm-hmm...
69
00:10:48.184 --> 00:10:53.684
then make bullet point summaries, have dates, have ent-entity references- Mm... have your sources for references, all these things.
70
00:10:54.144 --> 00:11:03.504
Anyway, so, so these are things that like I think people who really, really experiment with structural output have to really care about. But, um, let's, let's sort of recurse up the stack a little bit.
71
00:11:03.884 --> 00:11:13.064
Before we started recording, you actually mentioned something which is really cool, which is that there's a lot of engineering that- inference engineering that goes on when a new model provider releases a new model, right?
72
00:11:13.504 --> 00:11:27.944
So let's call it GLM 5.2, Kimi K3. I had pre-previously assumed, especially if it's like, well, GM- GLM 5 to 5.1 to 5.2, like you know that you've supported them before. Is it that much work? It's a lot of work.
73
00:11:27.964 --> 00:11:31.424
[chuckles] Yeah. Okay. So like, you know, a lot of people, all you guys, right- Mm...
74
00:11:31.444 --> 00:11:37.984
whenever a new model launch like, uh, people rush to say like, "Oh, Hugging Face supports this, Firework supports this, Space10 supports this." And I'm like, "Yeah, of course we support it."
75
00:11:38.344 --> 00:11:44.064
But what, what, what goes into that? What goes- I think it's more than just support it too, right? Uh, it benefits the consumer a lot.
76
00:11:44.104 --> 00:11:54.804
Like I think it was with, uh, Kimi K2.5 or GLM 5.2, the latest, there was sort of an inference war, right? X provider is at 90 tokens a second. The next day we're at 150.
77
00:11:55.124 --> 00:12:05.624
The next- I, I kinda kicked that off with the GLM 5.2. [laughs] I wrote a Twitter article about... It got like half a million views, um- Based on being number one- Yeah... or special analysis. [laughs] Yeah.
78
00:12:05.664 --> 00:12:06.204
Which, uh- Oh my God...
79
00:12:06.224 --> 00:12:25.008
which then got everyone really excited about, hey, how can we, you know, bend, bend tracks a little bit further and, um- There's a difference between support the model as in like I can make a token out of this model and support a model as in I have a production-ready API from this model.
80
00:12:25.128 --> 00:12:25.248
Mm.
81
00:12:26.068 --> 00:12:46.607
Getting to the point of I can make a token out of this model is not that hard because generally the, um, open source influence engines, you know, VLMs, SGLangs of the world oftentimes even receive weights ahead of time maintainers do, or the people making the model merge PRs to ensure support.
82
00:12:47.428 --> 00:13:01.508
So you generally can, you know, just kind of get it working on the standard open source stack without too much pain in most cases. The challenge is, you know, every influence company is gonna have own proprietary stack.
83
00:13:01.768 --> 00:13:10.868
Um, you know, some open source components, some, some in-house stuff. And for any arbitrary model, there's going to be some new stuff.
84
00:13:10.928 --> 00:13:19.288
Sometimes you get lucky like K, uh, two five to two six was like pretty, pretty similar. Yeah. It was pure continued post-training- Yeah... if I remember correctly.
85
00:13:19.668 --> 00:13:25.988
Even in those cases, there's still stuff you have to do. You have to redo the quantization work. You're taking the model from...
86
00:13:26.108 --> 00:13:33.188
Generally, these models are not released in NVFP four, um, and we want them to be in NVFP four for maximum Blackwell compatibility.
87
00:13:33.768 --> 00:13:43.348
So we have to perform that quantization, um, and, you know, calibrate the quantization to make sure that we're not causing any kind of regression in the model's intelligence.
88
00:13:44.068 --> 00:13:47.368
And then we also have to train the speculator, as we've talked about.
89
00:13:47.468 --> 00:13:58.568
Generally, we have-- obviously we have ZDR zero data attention on our model APIs, so we don't know exactly the traffic that people are sending us, but we know what's popular. We know that coding use cases are popular.
90
00:13:59.108 --> 00:14:08.468
Uh, we know that agents, agentic use cases are popular. So we can get public data sets that are representative of that kind of traffic and, and train gener-general speculators.
91
00:14:09.148 --> 00:14:23.048
Now, with speculators today, you need to train the speculator using the base model itself because you're getting hidden states out of the model from running inference on these specific prompts, and that is the training data you use to create the speculator.
92
00:14:23.708 --> 00:14:34.788
So there's that process which you need the, the wheel model weights for. Uh, and then there's of course just the process of, you know, standing up all the infrastructure behind it, loading all this stuff in, testing it.
93
00:14:35.188 --> 00:14:35.428
Um,
94
00:14:36.368 --> 00:14:48.088
and then when there's a new model with a newer architecture, I think that like obviously the DeepSeek models tend to be the most challenging as they have like the most novel architectural stuff going on, uh, model of a model.
95
00:14:48.568 --> 00:14:58.308
But every new model has something. I mean, Kimi K two had... Oh, sorry, uh, GLM five two had- Spurs. Yeah, the, the DSA. Right. Which is brought from DeepSeek. Yeah. Yeah.
96
00:14:58.328 --> 00:15:06.408
And, and you know, we- So you can copy-paste into it [laughs]. It kind of works. [laughs] Um, you know, so like- Mm... we had to like build support for that into our runtime.
97
00:15:06.708 --> 00:15:15.348
Um, and, and you're right, like it actually is really interesting the way that all of these open source labs borrow from each other. For example, like GLM five point two doesn't have vision.
98
00:15:15.748 --> 00:15:29.468
So something that, uh, Hailey, a guy on our team, if, if we could take a look at this, um, he like kind of grafted the Kimi, um, vision encoder onto GLM five point two. Only training the projector. Exactly.
99
00:15:29.528 --> 00:15:38.868
So if you think about like the encoder, there's the encoder, which is the part that looks at the image and, and turns it into latent information, and then there's the projector which kind of like- You can say latent space.
100
00:15:38.908 --> 00:15:47.828
It's okay [laughs]. [laughs] And then there's the projector that, that maps it onto, uh, the, the model itself, and then there's the model weights.
101
00:15:48.608 --> 00:15:55.628
You don't wanna mess with the model weights because you run a chance of making the model dumber at something else for the purpose of giving it vision.
102
00:15:56.148 --> 00:16:06.048
So instead, Hailey started with just a projector, which is only a, a handful of millions of parameters. It's okay, yeah. Yeah. And- Can you show the, the training one? Y- Like the way it groks is- Yeah...
103
00:16:06.148 --> 00:16:12.028
very, very interesting. And maybe- That right there... you know, may-may-maybe Ali you should, you should take it from here. You've got a, a better- Ooh, double the sand... understanding of this than I do. Yeah.
104
00:16:12.168 --> 00:16:19.448
You can see like he-- the way he trained this is, is really, really cool. At the beginning, he was training it using just like, "Here's a picture of a mountain. Can you describe what's in this mountain?"
105
00:16:19.968 --> 00:16:29.088
And that caused it just like the first, the first, you know, learning walls. Like here you can see this all we're trying to teach it is to translate the encoded... Like it's, it's already taken the encoder from Kimi K.
106
00:16:29.168 --> 00:16:38.068
It's taken the, the image. It's- Yeah. Frozen, frozen- Frozen, frozen... with adapter. Exactly. Yeah. So, so the understanding the brain is frozen and the eyes are frozen. It's just we're trying to- Align...
107
00:16:38.108 --> 00:16:43.308
interconnect between the eye and the brain, right? So the projector. And so you take the tokens and then he's like, "Oh, can you describe what's in this image?"
108
00:16:43.328 --> 00:16:48.508
And he's like, "Oh, it's, it's a mountain," or it's a person or it's a human, whatever the case is. But that didn't cause complete understanding.
109
00:16:48.548 --> 00:16:58.908
So he changed it such that every image was associated with a data set of questions. Like, does this image have a white male? Does this image have birds in the top corner? Does this image have a scientist in it?
110
00:16:59.028 --> 00:17:07.068
All of that stuff. And it would have to answer questions correctly. And using not just training on describing an image, but being able to answer question, answer question, answer question, like question answer over time.
111
00:17:07.408 --> 00:17:15.748
Like you can see the grokking, which is like genuinely insane that retrofitting vision into a large LLM can learn to that extent.
112
00:17:16.128 --> 00:17:24.308
And even for images that it doesn't perform well on, for instance, if you ask it a picture of like Stephen Hawking, "Who is this?" Maybe it doesn't get it, but it will say something like, "This is Albert Einstein."
113
00:17:24.348 --> 00:17:32.268
Like it still understands- Close enough... this is a scientist who is a man who has, you know, done significant achievements, all that stuff. So that's like really, really cool. Yeah.
114
00:17:32.308 --> 00:17:41.648
And so, uh, we've covered Hao Tian before, who was the author of the Lava paper that did this, uh, a while ago. And I think that that's very foundational work for anyone who hasn't done vision work before.
115
00:17:41.728 --> 00:17:51.628
Same with the CLIP and Meta CLIP, where you go from just captioning to building out questions- Right... off the image and how much better you can get performance. Right. Right. Right. Yeah.
116
00:17:51.648 --> 00:18:01.448
But what's, what's so exciting about this is if you look at a model like this... Now, obviously, this is a little bit more of a research project. It's not, you know, it, it got to fifty-six percent on MMLU Pro, I think.
117
00:18:01.528 --> 00:18:09.988
So not, not quite Fontia. But if you're running this model, you haven't suffered any loss on your GLM five two quality.
118
00:18:10.128 --> 00:18:18.728
If you don't have an image, it'll just behave exactly the way it used to, and ultimately the- Which, which in the inference code, you literally do not include the other part, right? Yeah.
119
00:18:18.848 --> 00:18:27.308
I mean, you would just skip the encoder if you don't have an image input. Okay. Um, just confirming. Yeah. Does it affect a lot on the overall inference side? Like you're not adding much.
120
00:18:27.328 --> 00:18:33.088
You're adding a very small vision encoder. These are typically like- They're super tiny... less than a billion parameters, right? Yeah. It's...
121
00:18:33.168 --> 00:18:41.060
I mean- There's a little bit less standardization among vision encoders- Yeah... um, so the s-sort of support matrix can be a little bit, uh, sparser.
122
00:18:41.700 --> 00:18:46.960
But overall, um, yeah, it's, it's a pretty, it's a pretty minor component of the overall system.
123
00:18:47.020 --> 00:18:55.720
And, and ultimately what you get out of the system is all of a sudden you have Chemi Vision, GLM weights, and DeepSeek attention all in one model.
124
00:18:55.740 --> 00:19:05.100
[laughs] And that's, I think, a lot of the power and beauty of open source is that you can take all of these different components and, and combine them together into a system that's better than anyone- Yeah...
125
00:19:05.120 --> 00:19:13.020
can be individually. People used to say that you would also do Franken merges where you would take like- Yeah... layers from each model. Mm. No- does anyone do that anymore?
126
00:19:13.740 --> 00:19:16.620
Well, to, to your point previously when you were mentioning like, um,
127
00:19:17.580 --> 00:19:27.700
the work that goes into supporting a model when it first comes out, like GLM five two or Min- Minimax M three or whatever the case is, sometimes you do have to r- like, you do have to switch out some things.
128
00:19:27.740 --> 00:19:30.620
Like for instance, the Minimax M three head uses full attention,
129
00:19:31.580 --> 00:19:43.640
and with full attention you end up with this like insane bottleneck in Specter 'cause you're doing auto aggressive token generation for three tokens, and you're doing this like, like O of N squared over all of the tokens that are in your sequence.
130
00:19:44.480 --> 00:19:47.820
Um, your KV cache is like very large because it's not sparse, it's not top K.
131
00:19:48.480 --> 00:19:55.540
So we find it better to like, okay, we're gonna replace this, you know, we're gonna replace this layer with a layer from another model that's using like GQA for instance.
132
00:19:56.080 --> 00:20:07.900
And then just with the right training you can get it to have the same acceptance rate. So it is, it is very possible to, to retrofit layers from other models and very much needed actually, if a layer is like inefficient.
133
00:20:08.440 --> 00:20:11.280
The training just becomes the challenge, like how do you ensure that you train it properly?
134
00:20:11.600 --> 00:20:21.420
Which again to your earlier point is like the, the mesh between training and inference as in like, like you need very good training in order to do fast inference. That's like I feel like more and more becoming true.
135
00:20:21.880 --> 00:20:26.100
Yeah. Anything else on the support side when you, when you say like get it to fully production ready?
136
00:20:26.680 --> 00:20:43.579
Yeah, I think that there's also a question of just, you know, we can test a model to a pretty extensive degree, but we're trying to get it out quickly and then you see a bunch of other people test it and you get interesting results.
137
00:20:43.840 --> 00:20:55.880
There was a- an issue with, um, GLM briefly where we had some like mode collapses where it would just output the same token over and over again for certain prompts on certain temperatures.
138
00:20:55.940 --> 00:21:08.140
Like once you expose an endpoint to, to the wheel world, there's going to be, you know, so many more varieties of, of things given to it that, that you're able to, you know, discover and, and patch things.
139
00:21:08.200 --> 00:21:20.920
So it's not just a, you know, day zero process, it's then like for the first week, for the first month if a model remains popular, like how do you both fix bugs and then continue to push the envelope on performance?
140
00:21:21.460 --> 00:21:31.140
What do you mean you don't want your model outputting S, S, S, S, S, S? [laughing] [laughs] I- is there loop detection on that stuff by the way? It still happens like quite a lot, which is surprising. We have like we...
141
00:21:31.180 --> 00:21:41.000
in our, in our endpoint, like if a model was to output the same exact token like four plus times, we just cut the generation. We say like, "Oh sorry, this... Like try again." Or like we- Ah... reprocess the request.
142
00:21:41.340 --> 00:21:49.060
'Cause we know then like if it... like if, yeah, it's four times the same token, it's probably collapsed. Yeah. Is there a way to opt out in case I really actually want that? [laughs] You actually want that?
143
00:21:50.120 --> 00:21:52.720
[laughing] [laughs] I think, I think there's a way that we have to handle it.
144
00:21:52.760 --> 00:22:00.740
I'm not exactly certain, but I feel like in certain models like when they output something like you can imagine, um, like a table for instance, and so they want, they wanna draw like 12 dashes and 12 dashes.
145
00:22:01.320 --> 00:22:07.180
Yeah, I think there's a way for that to happen. I think we only do it on certain tokens, like we exclude certain special characters. Yeah.
146
00:22:07.220 --> 00:22:17.380
So we only do it on like certain like- like S is the most common almost, GLM five two. Oh. Um, and I think it was DSV four as well. Like you'd just have like looping issues where like- It was-...
147
00:22:17.380 --> 00:22:23.680
you literally just have like S, S, S, S. Yeah. Is there, is there a special- something special about S? No, just randomly- It just seems to be the one token [laughs] involved. Yeah.
148
00:22:24.360 --> 00:22:30.560
Uh, and it's, and it's only temperature zero or- No, no- Even at other temperatures... even at like point nine or whatever it will still, it will still collapse. That's weird, right?
149
00:22:30.560 --> 00:22:32.980
It's, it's an inference, it's an inference problem to be honest, like a software problem.
150
00:22:33.000 --> 00:22:43.980
Like oftentimes, um, the image you run, like NVIDIA will release an image for instance, and if we will upstream the changes from their latest CRTLLM image into our stack, we'll find that it, it fixes it.
151
00:22:44.020 --> 00:22:49.740
Or oftentimes this will only happen in an inference engine that you're using like SGLang. But if you were to switch to VLLM, that isn't the case.
152
00:22:50.360 --> 00:22:58.500
So it seems to be like an extremely like non-deterministic kind of software issue and not really a model issue. It's not like a weights problem. Like I'll- we'll say like, "Oh, it's a problem with the quant.
153
00:22:58.560 --> 00:23:06.080
We, we did PTQ wrong," right? But that isn't... that doesn't make sense because the same exact weights used with a different inference engine does not repeat this p- the problem.
154
00:23:06.660 --> 00:23:18.760
And sometimes it's, um, the kernels that are being used in the backend have like these very subtle sometimes race conditions where if you were to use this model hosted on one cluster, you will never get this problem.
155
00:23:19.120 --> 00:23:28.280
Oh my God. But if you host it on a different cluster, you will. And the reason is the KV cache transfer from a node to node in that one cluster is using a slower interconnect than the node to node in another cluster.
156
00:23:28.640 --> 00:23:34.920
So that exposes the race, whereas in another cluster it doesn't. So then you end up just like, "Okay, this model is not gonna be hosted on this cluster.
157
00:23:34.940 --> 00:23:42.500
We're gonna host it on, you know, another cluster because that cluster exposed that problem." But then it ends up with like, okay, is it the software? Is it the model weights or is it the hardware?
158
00:23:42.900 --> 00:23:51.580
There is a thing about this with temperature zero still not being deterministic, right? Right. Mostly because of hardware. Even at temperature zero- Mm-hmm... same model, you won't always get the same output. Mm-hmm.
159
00:23:52.000 --> 00:24:02.500
Even... But I'm surprised by the race condition one because, uh, I thought PyTorch was a graph that like guarantees that you at least, you know, execute things in the right order. [laughs] Well, yeah, true.
160
00:24:02.580 --> 00:24:05.100
Like, like I guess, I guess I'm not, I'm not saying that this is...
161
00:24:05.240 --> 00:24:12.200
Like I guess, well, you have things like PTL optimizations where like you can, you can start a kernel before the end of the previous kernel, and that's like because you want to do that because- It's like pipelining...
162
00:24:12.200 --> 00:24:16.540
there's no expense. Exactly, exactly. Yeah. But it's... But you don't do it cleanly. Like you, you overlap a little bit of the execution.
163
00:24:17.440 --> 00:24:25.780
No, I guess it is very possible that the kernel itself, like that one block that is supposed to be running in this instance of time, that kernel itself has a race condition. For instance, like a missing barrier.
164
00:24:25.840 --> 00:24:36.140
Like often if you're designing a kernel and you want it to make it to be very fast, if you don't test it extensively you'll, you'll have certain threads access data points from registers before they've been written to by other threads- Yeah...
165
00:24:36.180 --> 00:24:38.780
for example, because like your barrier is wrong or your synchronization was wrong.
166
00:24:39.500 --> 00:24:47.640
But yeah, like the testing itself is, is very, very difficult in those like, and the- And there's no like borrow checker for- What does that mean? Uh, like Rust. Like the, the...
167
00:24:47.700 --> 00:24:54.460
if you, if you're trying to have like memory safety if- Mm. It, it, it sounds like a comparable problem. Well, yes, but you're working in CUDA, right, NVIDIA GPU.
168
00:24:54.580 --> 00:25:02.420
It's like- You just need a higher level language like modular [laughs] Maybe that's what modular is supposed to do. I don't know. [laughs] How do you see keeping quality of the model?
169
00:25:02.480 --> 00:25:09.160
So you talked about all these steps of, okay, you gotta do quantization, train your own speculative decoder- Right... run on different hardware.
170
00:25:09.640 --> 00:25:22.860
Um, looking at other model providers, okay, you kicked off a inference speed race on the consumer end. Um, what goes into keeping quality the same across them, right? Sure, you can run benchmarks, but like- Yeah...
171
00:25:22.880 --> 00:25:32.500
how do you determine how much quantization are there standards? What actually goes into- There's a few things on quality. Most inference optimizations are lossless.
172
00:25:32.920 --> 00:25:47.860
KV caching, for example, you are just recomputing or preventing recomputing the same values. Speculation, of course, if a draft token is wrong, it gets rejected. The main lossy optimization is quantization.
173
00:25:48.300 --> 00:26:06.040
Um, and that really comes down to, number one, data format, uh, number two, uh, which parts of the model you choose to quantize, which layers, and number three, like doing a lot of calibration on the quantized weights, uh, to ensure that you're sort of preserving all the outliers.
174
00:26:06.720 --> 00:26:17.540
There's other sort of tricks that you can do, though. A big one is long context, 'cause one thing you asked at, right at the beginning is, "Oh, what's gonna happen if I send a 200,000 token request in?"
175
00:26:18.220 --> 00:26:28.260
So obviously, with a long input sequence, you need to, you know, store a lot more information. Uh, you need to process a lot more tokens.
176
00:26:28.900 --> 00:26:41.540
And so even if a model has a context of a certain length, you might, as an inference provider, choose to build an API with a shorter context length and of course a full length one as well.
177
00:26:41.980 --> 00:26:51.700
Because if someone doesn't need the full million token context, for example, you can get them better performance. I don't know if that's exactly like quality of the model.
178
00:26:51.780 --> 00:26:57.420
The, the way that I think about quality is to what degree are we faithfully serving the original model?
179
00:26:57.700 --> 00:27:13.200
If you think of a sort of golden implementation of a model that performs exactly the way the model is designed to perform, I think of quality as how close are we getting to that, you know, 100% fidelity of the model. Mm.
180
00:27:13.240 --> 00:27:18.120
You can also, of course, think about quality from the training side and how do you push yourself past 100%.
181
00:27:18.480 --> 00:27:27.720
But when I think about purely influence optimizations, it's getting faster while staying as close to that 100% fidelity mark as, as possible.
182
00:27:27.880 --> 00:27:37.240
And certainly our standard internally is that, like you should not be able to tell the difference between our API and a, you know, sort of official API.
183
00:27:37.560 --> 00:27:47.040
I think Kimi in particular does a good job of vendor benchmarking here- Yes... where they have- They released an actual vendor benchmark. Exactly. Yeah. 'Cause they accused, uh, some people, Amazon.
184
00:27:47.480 --> 00:27:55.860
There was some provider that was not doing very well on Kimi's benchmark. Yeah. [laughs] So, uh, with reflective, reflective we probably- This was a long time ago, right? No, like, like three- They also-...
185
00:27:55.920 --> 00:28:05.180
four, five months ago. This also happened with, I don't remember which model, but they, they pulled out quite a few, and then they started a whole chart about this. It might have been- Kimi vendor verifier. Yeah. Yeah.
186
00:28:05.700 --> 00:28:07.580
Yeah, 'cause you, 'cause you'd be pissed, right? Like if you're- Yeah. [laughs]...
187
00:28:07.620 --> 00:28:16.180
if like, um, like if I'm a consumer, and I'm using like Amazon's endpoint, for instance, and I, I'm use Kimi and I'm like, "Oh my God, like this is bad." I'm not gonna say, "Oh, Amazon quantized the model in a bad way."
188
00:28:16.220 --> 00:28:22.780
I'm gonna say, "Oh, Kimi sucks." Right? Yeah. So it seems like that, that makes sense. Yeah, they care. They care. Um- Justifiably. [laughs] Yeah, justifiably.
189
00:28:22.880 --> 00:28:33.020
Uh, uh, this is probably a stupid question, but just checking, has anything improved from mean quantization? Yeah. Like, is quantization always strictly worse? N-- Uh... Well, technically, no. It's, it's, it's a lossy.
190
00:28:33.140 --> 00:28:40.820
Quantization- Yeah... is a lossy, it's, it's a lossy implementation. It- Speed improves. It speed improves. [laughs] It, it obviously number, like you- No, I'm always look for inverse scaling laws. Yeah.
191
00:28:40.840 --> 00:28:45.400
This is something I learned from Noam Brown, where like things that normally act in one direction sometimes do.
192
00:28:45.760 --> 00:28:52.940
Well, technically, when you run a benchmark, because these models [laughs] are non-deterministic, sometimes your, you know- [laughs] Yeah, yeah...
193
00:28:53.060 --> 00:29:01.300
MBFP4 quant is like, uh, you know, two basis points higher than your- That's noise. That's noise. Yeah, exactly. Yeah. It's, it's, it's within... That's why I always say within margin of error.
194
00:29:01.900 --> 00:29:12.479
And I actually stopped saying that because everyone assumes that what I mean is, well, within some margin of error, we're barely inside of that to the worst, so we're saying... But yeah, sometimes it's just like,
195
00:29:13.380 --> 00:29:22.980
uh, gives you a higher output score. But like, like Ali said, that's noise. Um, to my knowledge, you're not necessarily making the results better. You're just trying to, again, like- Mm-hmm...
196
00:29:23.180 --> 00:29:33.000
keep your fidelity as close to 100% to the original model. There is, to your point, research that we did on MP. I don't know if you are able to pull- Yeah... um, a tweet we did.
197
00:29:33.200 --> 00:29:43.220
One of our research interns, Joshua, um, I think it's a tweet on, on how we have 20% better quantized JLMF2 than NVIDIA.
198
00:29:43.940 --> 00:29:49.360
Um, essentially what we found throughout like this two-month research is, okay, quantization is, is a lossy.
199
00:29:49.600 --> 00:29:58.820
It's, it's you're compressing the data from, you know, occupying 16 bits to occupying, you know, four bits, for instance. And so you're obviously losing some, some information, and you're trying to minimize that.
200
00:29:59.340 --> 00:30:05.440
And so when I say that I'm gonna quantize the model, my, my job becomes how do I find the layers that I can quantize and how to find the layers to not.
201
00:30:05.500 --> 00:30:13.320
For instance, with image models, I don't quantize modulation layers, and I don't quantize out projections because those two are... Like out projection is what you see as the user.
202
00:30:13.680 --> 00:30:24.280
Modulation is what the model sees or understands. Right, exactly. Um, and so I guess to his, his paper, do you have the... I guess it doesn't have the, the, the... Yeah. It's a long paper.
203
00:30:24.440 --> 00:30:29.340
Uh, [laughs] I don't know if I can find- If there's a part to search or it's probably in the thread. It's probably in the thread. Yeah.
204
00:30:29.420 --> 00:30:36.360
But basically, the, the long and the short is it is very possible that quantizing more of the model makes the results...
205
00:30:36.580 --> 00:30:42.880
Like if I have a model that I quantize layers one, five, and 10, and another model where I only quantize layers one and two-
206
00:30:43.892 --> 00:30:49.872
It is possible that the model in which I quantized more information is going to perform better because the quantization errors have canceled out.
207
00:30:50.352 --> 00:31:01.492
And so what Joshua showed in his mathematical proof where he had like a verifier in, is that you can predict which layers are going to have quantization errors that will cancel out with each other, and you choose to quantize those layers.
208
00:31:01.832 --> 00:31:20.292
And so the result of doing this mathematical quantization is you end up with a model that's twenty percent more quantized than another provider, so you get twenty percent more throughput of it because there's more layers than running an MBFFour, and your quality is better than that other quant because the layers that you chose to quantize have their errors cancel out, like one layer skewed to the right, one layer skewed to the left, one layer skewed to the right.
209
00:31:20.652 --> 00:31:28.132
Your final logistics distribution is more similar to the original distribution of the model, so you have better fidelity. And so the way we proved this was with KL divergence.
210
00:31:28.152 --> 00:31:35.932
So instead of just scoring on the benchmarks, we scored the KL divergence between the logic distribution of the quantized model and the logic distribution of the original full precision model.
211
00:31:36.272 --> 00:31:37.712
And we showed that with this technique we get...
212
00:31:38.272 --> 00:31:46.872
If your, if your, you know, if your probability distribution on the logic switch token it wants to select is more of the same as the original model, you're probably gonna end up staying true to the original model.
213
00:31:47.352 --> 00:31:55.372
So yeah, so it seems like, like previously before this, it seemed like the industry was, well, the more you quantize, the worse it's gonna be because the more loss you introduce. That's not exactly, not necessarily true.
214
00:31:55.752 --> 00:32:04.232
So yeah, doesn't improve it, but can cancel out. I think it might be this, but reminds me a good bit about pruning actually, where you can prune off certain layers. Mm-hmm. But very interesting.
215
00:32:04.292 --> 00:32:13.332
Didn't know this was a whole paper you guys put out [laughing]. It-- Well, it's a fun fact, it was originally seventy-two pages, this paper, and then we decided- Wow... we can't tell. We couldn't release it.
216
00:32:13.612 --> 00:32:22.892
So it's now forty-five. [laughs] Still, still thirty-nine pages, so very, very substantive. We talked about evals and, and all these things and, uh, like what's possible in terms of speed up?
217
00:32:23.212 --> 00:32:29.912
I guess like it's like probably like the number one- Mm... thing that people do wanna care about, and it's something that you wrote about in your post.
218
00:32:30.532 --> 00:32:41.952
Um, like official API, seventy tokens per second, and you push it up to ninety. Is that like a normal thing? So what's cool about working in Influence, the reason that I think Influence is going to be a
219
00:32:42.992 --> 00:32:55.012
useful place to do engineering for a long time, is that if you look at highly optimized domains like say finance, if you're in finance, you measure how much better you got in basis points.
220
00:32:55.052 --> 00:33:02.812
It's like, "Oh, I got five basis points better, like one-twentieth of one percent better," that's huge news because everything is, is so optimized.
221
00:33:03.772 --> 00:33:12.192
When we publish optimizations, it's, it's twenty percent, it's a hundred percent- [laughs]... it's two hundred percent. So there's still probably like a lot further to go, honestly.
222
00:33:12.232 --> 00:33:19.312
Like you'll, you'll know that Influence is pretty much solved when researchers start publishing about how they got one percent faster at, at something.
223
00:33:19.872 --> 00:33:29.832
Which by the way, uh, because I am from the finance background, in the seventies, that was the margin at the time. When you did quantitative finance research, you would find- And like twenty percent, ten percent. Yes.
224
00:33:30.012 --> 00:33:35.192
Yes. Yeah. And now it's- Yes, uh, tiny fractions... for, for those, for those people interested, look up Andrew Lowe's paper.
225
00:33:35.292 --> 00:33:47.992
Uh, he had a, a really interesting illustration of quant, uh, stat arb, uh, distribution, uh, narrowing down from like those kinds of twenty percent differences in the seventies, uh, down to nothing today, um, which is very, very cool.
226
00:33:48.432 --> 00:33:57.372
Exactly. And, and we're at the beginning of, of the same type of, of, of thing. Now benchmarking is hard. Um, I think any-anyone will tell you that.
227
00:33:57.412 --> 00:34:05.772
And, and benchmarking provider speeds is hard because there's so many variables that go into it. What hardware are you using? How much load do you have on the system?
228
00:34:06.132 --> 00:34:19.272
What's the exact nature of the prompts and input and output sequence lengths, all that kind of stuff. But overall, when you start stacking these improvements, you're looking at multiples. Um, you can look at it...
229
00:34:19.592 --> 00:34:28.552
The, the most common form, of course, is TPS, tokens per second, which is bad naming by us in the industry 'cause there's actually two tokens per second.
230
00:34:28.592 --> 00:34:36.912
There's tokens per second, the throughput number, and, and the latency number. TTFT. Yeah. Um, to- like total tokens per second out of the, out of the GPU as a throughput number.
231
00:34:37.332 --> 00:34:45.552
Most people only care about tokens per second as the latency number, which we should call ITL, intertoken latency, but we don't. Um- [laughs] Anyway, so
232
00:34:46.712 --> 00:35:03.031
you can imagine a sort of standard API without many optimizations for a one trillion parameter model operating somewhere in the thirty to fifty tokens per second range for, for reasonable traffic profile.
233
00:35:03.552 --> 00:35:10.472
Um, and we generally see the goal of, you know, pushing to 10X that.
234
00:35:10.952 --> 00:35:28.312
Um, but, you know, not necessarily day zero, but by stacking enough optimizations, um, if you have, say like four optimizations, each of which doubles performance, or sorry, three optimizations, each of which doubles performance, uh, then you, you stack that up, that's an eight X gain.
235
00:35:28.932 --> 00:35:38.732
That's kind of the, the order of magnitude that we're working with in this, in this space. We're trying to make things substantially faster, not just go from like seventy to ninety.
236
00:35:38.812 --> 00:35:47.392
Are you saying you've-- you, you have done that? So let's say you have as a reasonable baseline thirty or forty tokens per second. Uh, you can, you can achieve 10X that.
237
00:35:47.452 --> 00:36:10.332
So like on GLM five point two, um, if you wanna get unquantized, uh, perhaps on HOPOS even, um, and you're just using an off-the-shelf influence engine with no particular optimizations, no, no speculator, um, nothing, nothing extra around like KV routing, no disaggregation, uh, you're, you're probably, yeah, looking at that like thirty to forty.
238
00:36:10.372 --> 00:36:17.072
Do you think that's like a reasonable baseline? Right. Right. To get to something like 10X, there's a lot of trade-offs that you're making.
239
00:36:17.512 --> 00:36:30.912
If we're running at sort of more like a three hundred, four hundred tokens per second range, um, obviously you are using the best hardware possible. Uh, you have a optimized speculator.
240
00:36:31.112 --> 00:36:37.216
You have done all of your quantization work. You are Seeing a pretty high cache hit rate.
241
00:36:37.916 --> 00:36:48.556
You are running with a reasonably small batch size and a parallelism configuration that is tuned for latency versus throughput, but it is possible.
242
00:36:48.996 --> 00:37:02.696
Um, so the spreads that you see if you, like, go on artificial analysis or you go on OpenRouter and you look at, you know, the, the worst provider to the best provider, oftentimes can, can hit that kind of range.
243
00:37:02.876 --> 00:37:08.356
Uh, 10X is of course very aggressive. It's oftentimes maybe more of a four to six times improvement.
244
00:37:09.076 --> 00:37:20.916
But that's the, the kind of performance that makes us really excited, is when we can get these huge gains, not just go from 70 to 90 tokens. It's also, like, hardware dependent. Like, if you- Yeah...
245
00:37:20.916 --> 00:37:27.416
if you obviously have a thing where you're serving it on just, like, a node of H100s and then you throw, like, you know, you shard the model across, like, four nodes of B200s.
246
00:37:27.996 --> 00:37:35.496
Like, like, y- you can definitely increase the speed with just throwing more hardware at it. Like, normalizing for the same exact hardware and the same number of GPUs. Yeah.
247
00:37:35.596 --> 00:37:45.396
Then you're looking at, like, a two to 4X improvement- Right. Right... depending on the influence optimizations. So yeah, it's... Some of it's, you know, what's the call, and some of it's who's the driver.
248
00:37:45.396 --> 00:37:54.776
[laughs] If you break down the two to 4X, say, say the example is run GLM 52- Yeah... on B200s- Yeah... single node, right?
249
00:37:55.316 --> 00:38:06.736
What's, like, the cost trade-off for effort to get, like, the last bit of juice out versus what should people just think of, right? Spectre quantization. Yeah. Spectre quantization. That's, that's, that's, like, 95%.
250
00:38:06.816 --> 00:38:12.376
Like- And how far does that get you? And how, how easy is that for the average person to do? So say right now- Mm-hmm...
251
00:38:12.656 --> 00:38:23.236
I wanna throw the weights of GLM 52 on a node of B200s, how easy is it to find speculative decader- decoder model or already quantized model? How much work goes into it?
252
00:38:23.696 --> 00:38:34.176
If you're doing it up front, it's quite a lot of work. If you're doing it today, there's going to be people who have published things that you can just, you can just grab some NVFP4 weights. You can grab a speculator.
253
00:38:34.956 --> 00:38:43.976
Uh, yeah, if we're thinking about, like, what are the 2Xs we're stacking, going from, uh, s- BF16 to NVFP4 is, it's not quite a 2X, right? It's like...
254
00:38:44.676 --> 00:38:59.476
I, I think it's about, like, 30 th- 30 to 40%, um, from 16 to 8, and then another 30 to 40% multiplied from, um, 8 to 4. So that doesn't quite get you a 2X, but, like, roughly a 2X. Speculator, roughly a 2X.
255
00:38:59.656 --> 00:39:05.816
Disagg on top of that, if you're able to get enough hardware and, and put enough traffic through it, another roughly a 2X.
256
00:39:06.436 --> 00:39:20.276
And then you add in some, you know, double-digit percent increase from having just a, a better runtime with, you know, the, the latest kernels and stuff behind it. Um, and that, that's kind of how it stacks up.
257
00:39:20.656 --> 00:39:29.656
Um- Yeah... so building each of those, like, building the, um, quantized weights is, for someone who really knows what they're doing, hours to days of work.
258
00:39:29.856 --> 00:39:40.576
Um, building the speculator, again, like, hours to days of work. And the, um, disagg setup, hours to days. Well- [laughs]... okay, but like once, once you have it- Once set up. Once set up, yeah...
259
00:39:40.616 --> 00:39:44.496
yeah, yeah, getting disagg working for the first time, I'm saying, of course, is, is very difficult. But- Mm-hmm...
260
00:39:44.596 --> 00:39:55.436
the, the, the marginal implementation is- If you're just grabbing, like if you are a person, like just a normal consumer who has access to like a node of B200s and you're wondering, "How can I just host it myself?"
261
00:39:56.036 --> 00:40:01.736
You don't need to quantize the model yourself. There's always gonna be, like, an open source quantized checkpoint. Nvidia's gonna push one out if no one else does.
262
00:40:02.256 --> 00:40:08.736
You-- usually the providers will have their own spec deck that they've trained as well. You don't need to train your own spec deck, you can just use that as well.
263
00:40:09.156 --> 00:40:17.916
Um- Yeah, like, like, Chemi, uh, uh, GLM 52 has its own MTP. Right. Right. Minimax. It's multi-token prediction. [laughs] Yes. I'm just a, I'm just an expert- [laughs]...
264
00:40:18.096 --> 00:40:28.136
so I, so I can do it for you in case I get it wrong, you know? No. Uh- Yeah, actually, you should correct if we're wrong, but their multi-token prediction can be used for self-speculative decoding. I'm actually not sure.
265
00:40:28.156 --> 00:40:30.816
I'm not gonna correct you. Okay. I'm semi-confident in that- Okay. Yeah, yeah... but someone can check.
266
00:40:31.256 --> 00:40:41.356
Um, but you know, it's useful to paint the story of, okay, not just the average person, but say a company wants to switch from serverless inference to- Mm-hmm... I wanna throw this up on, you know...
267
00:40:41.876 --> 00:40:51.316
I wanna rent some GPUs, throw it up. These are the steps you take to do significantly faster than just put it behind VLLM. Right. I was waiting for a mention of Dynamo. Uh- Mm-hmm...
268
00:40:51.336 --> 00:41:02.836
I feel like, uh, that's supposed to be the baseline that you measure against. I would think of Dynamo as less of a sort of out-of-box system and more of a toolkit for building with.
269
00:41:03.456 --> 00:41:14.736
So when we talk about doing KV aware routing, when we talk about doing KV out offloading, when we talk about doing, um, PD disaggregation, Dynamo fundamentally is...
270
00:41:14.876 --> 00:41:19.876
By the way, Dynamo is an, an open source library from Nvidia. Um- We, we've done a pod with Kyle- Okay... Kyle Cranin. Cool.
271
00:41:19.936 --> 00:41:29.356
So, so then your, your listeners know then that it supports all the different inference frameworks, um, and it actually is kind of multi-hardware, which is interesting. But it's just a router.
272
00:41:29.396 --> 00:41:39.116
It's not like an optimizer layer. Yeah. A- all it does... Like, what Dynamo is good at, it is a library for moving information around your cluster, around your hardware.
273
00:41:39.536 --> 00:41:48.056
So if you have, you know, KV cache on one place and you need it to be somewhere else, Dynamo coordinates Nixal for you to, to move that around.
274
00:41:49.136 --> 00:42:01.736
That doesn't mean that, like, out of the box, you just say, you know, "Pip install Dynamo," and then you get like a, a massive performance speed up. Um, it's, it's more of a, a developer toolkit. Yeah.
275
00:42:02.016 --> 00:42:07.176
I would have said it would... It, it comes with a set of defaults that you can then swap out. It does.
276
00:42:07.376 --> 00:42:19.776
Um, if the industry at large, I think, was, like, rolling out all of these deployments, um, standard, then I think it would be like a, a credible baseline. But, um- [laughs]...
277
00:42:19.836 --> 00:42:22.976
we've got to, we've got to benchmark against, like, what we're seeing in the wild.
278
00:42:23.516 --> 00:42:31.876
I did wanna talk a little bit more about PD disa- disagg, because that is probably like number three after quantized and speculative decoding. In your book though, uh, I was just gonna pull out the book. Yeah.
279
00:42:32.216 --> 00:42:44.512
Like section 522 on Medusa, 523 on Eagle- Yeah... 524 on NM- It's, it's 55, um, would be, um- Would be disaggregation Yeah. Well, no, I just wanted to dwell a little bit on- Yeah... the other.
280
00:42:44.552 --> 00:42:52.932
Like, so what do you choose to include? What do you choose to not to include? Because there was all these other techniques, I guess. Yeah. Are these still relevant?
281
00:42:52.992 --> 00:43:02.192
Because I think they came out like a year and a half ago maybe. Medusa is quite old. Yeah, Medusa's old. Um- But is it in the book as a good, here's the baseline- Baseline... vanilla understand it?
282
00:43:02.192 --> 00:43:09.012
Like you, you like you should know this. Like I read the paper, I'm like, "Ah, it makes so much sense." Like [laughs] Yeah. So w- with the book, I had a couple goals.
283
00:43:09.292 --> 00:43:19.352
Uh, one was to give people just a working vocabulary for the space as a whole, and the other was to give them some intuition about how each of these techniques works.
284
00:43:20.052 --> 00:43:29.432
As I mentioned in, in my AI engineer talk, which is kind of the first public addendum to this, the speculation space has moved much faster than everything else.
285
00:43:30.012 --> 00:43:42.792
So yeah, even at the time that I wrote the book Medusa, I very much included as a way for people to understand how the h- h- how, how the space evolved rather than what the most modern technique is.
286
00:43:43.212 --> 00:43:54.252
And now of course there's DFlash, dSparke, um, there's, there's newer techniques even than, than Eagle, although Eagle is still very commonly used. SpecSpecta. Yes. [laughs] Speculative speculative decoding.
287
00:43:54.712 --> 00:44:04.192
What can you- Oh, it's, it's bas- the paper by Tri Dao and it's like it's basically doing speculative decoding- Uh-huh... for the speculative decoder. Oh, and spec... Oh my God. It's, it's literally just an another...
288
00:44:04.232 --> 00:44:05.772
It's like, yeah, that's the most simple way to explain it.
289
00:44:06.092 --> 00:44:15.452
And it seems like he got non-trivial speed ups there, but it seems that the complexity with training, it's almost like, like in our mind at least, it's almost as complex as training GANs.
290
00:44:15.492 --> 00:44:24.132
Like it's like a very delicate balance and oftentimes you, it's just literally [laughs] but yeah, it's literally speculative decoding on speculative- Speculative, speculative. Yeah. We, we saw this paper.
291
00:44:24.192 --> 00:44:33.652
It's interesting, right? Yeah. I wouldn't even expect it to be very particular to train, I would- Right, right, right. The naive part of me is like, okay, train speculative decoder. But I mean, like, and it makes sense.
292
00:44:33.692 --> 00:44:39.392
Like the whole idea of speculative decoding is you, you... It's like, it's like almost like the iPhone auto predict version- Mm... but for normal model, right?
293
00:44:39.452 --> 00:44:46.232
Like you're just, you're just, you know, generating three tokens and you're like, okay, I'll do prefill on them. And so you save those three turns for your original model.
294
00:44:46.472 --> 00:44:58.692
Now your speculative decoder is doing three turns of auto regression. So why not just have an even smaller model? I guess the, the other question there is what are the size of speculators? So say for GLM- Right.
295
00:44:58.832 --> 00:45:06.312
It's like a billion to- a billion parameters. Like for MiniMax it's, it's... Yeah, yeah. It's like one layer. It's like one 60th of the original model usually. Yeah.
296
00:45:06.812 --> 00:45:16.472
Actually, I think we should do a paper when we get back to the office. Speculative, speculative, speculative- Speculative... decoding. No, it, it seem... It does seem like, like how, wh- when do you stop?
297
00:45:17.232 --> 00:45:22.972
But then it also seems like kind of, um, like if you're able to train spec, spec decode for instance, right?
298
00:45:23.012 --> 00:45:34.252
Like if you're able to have a small model that is ac- accurately predicts what the intermediate speculator is gonna predict, that is able to predict what the original target model's gonna predict, then why not just use that smallest model directly, right?
299
00:45:34.392 --> 00:45:37.072
Yeah. This is a- Like it seems like a-... adjacent to the routing problem. Right. Yeah. Right.
300
00:45:37.552 --> 00:45:47.632
The thing with speculators is one of the practical constraints on using them is that you do have to run a small model on the same hardware that you're running the big model on.
301
00:45:47.672 --> 00:46:05.252
There is a orchestration and resource competition problem inherent in that, and that is one of the sort of constraints on speculation in general, is that draft tokens cost resources to create and cost software complexity to manage.
302
00:46:05.392 --> 00:46:17.312
And so if you have sort of like infinitely recursive speculators, you add in quite a bit of that complexity on the actual implementation within the inference engine as well, not just in the training process.
303
00:46:17.712 --> 00:46:24.232
I was gonna say, I would wonder if you could do similar, like distillation and pruning of, you know, it's the same thing, it's just a model.
304
00:46:24.312 --> 00:46:36.432
Can we not just distill a lot of the weights, quantize the speculator, but- Mm... out of my domain? Um, I guess the, the question that also comes up is, this is all for big server workloads, right?
305
00:46:36.932 --> 00:46:51.392
How much of this applies to, say I have this MacBook, I wanna run Gemma really efficiently. Um, similar problems? Not the same? Pretty different. I talked to Selo, um, about this on, on his podcast a couple weeks ago.
306
00:46:52.132 --> 00:47:03.332
The difference between influence engineering for the data center and for production workloads versus influence engineering for local AI, is that we start with fundamentally like different constraints and different goals.
307
00:47:04.192 --> 00:47:16.412
With local AI, it's how do I fit this model onto my hardware and then make it less dumb? And with data center influence, it's how do I load this model and then make it less slow?
308
00:47:16.972 --> 00:47:28.752
And obviously, you know, we care about less dumb, and they care about less slow. But the local AI influence engineering ecosystem, I think actually has a lot for us to learn from in the data center space.
309
00:47:29.052 --> 00:47:41.572
They are experts in various forms of quantization, including dynamic quantization that we just kind of don't touch, in the pruning, in the distillation, in the, uh, you know, layer removal.
310
00:47:42.472 --> 00:47:48.532
There's- Layer removal matters less. Yeah. There's- No one does pruning, really. Yeah. Well, but the, but they do it- Which is surprising, right? But that's, that's a whole different thing...
311
00:47:48.552 --> 00:47:55.312
just to, just to fit something on the laptop. Right. Right, right. Um, so yeah, I mean, it's, it's a, it's an interesting, it's an interesting space.
312
00:47:55.632 --> 00:48:11.572
Not, not necessarily that like their techniques make sense for us to do in the data center, uh, because obviously we have different resources and different goals, but more that the, the process as well as the openness of that field is, is something to, you know, admire.
313
00:48:12.232 --> 00:48:18.552
Yeah. Like to your point, like, um, certain optimizations that would... Like for instance, Turbo Quantum Shaver, like it made such huge hype on that. And
314
00:48:19.572 --> 00:48:24.192
we, we did like a whole deep dive on Twitter and I was like, "What is it? How does it work? Why is it good or not?"
315
00:48:24.892 --> 00:48:32.772
And it took off, and it was implemented on local devices because your, your memory bandwidth is, is so slow on like a MacBook, for instance.
316
00:48:33.312 --> 00:48:39.828
But try putting the same thing on like an Nvidia GPU on a B200 Turbo quant would not be, like it would not be used.
317
00:48:39.868 --> 00:48:53.868
Like NVIDIA ha-- like NVIDIA made it clear that this is not a good optimization, and we've seen it firsthand where the overhead of doing dequantization quantization of, you know, in the kernel itself of turbo quant kernel, each end-to-end is actually much, much slower than the time that you save from doing the bandwidth.
318
00:48:53.868 --> 00:49:02.528
'Cause on the V200s, you have like three point five terabytes per second. You don't need to- Hmm... you know, decrease the, the storage that much. You don't need to do, you know, FP4 KB cache.
319
00:49:02.588 --> 00:49:09.088
You don't need to use a requant. There's, there's, there's better optimizations to be made. But on, on Edge devices, it's extremely important, it's extremely useful.
320
00:49:09.528 --> 00:49:19.268
So you know, s-seems to be like different optimizations there, but then they're all uniquely combined with like, oh, you wanna quantize the model, you wanna do speculative decoding, like certain common prefixes with both- Principles.
321
00:49:19.328 --> 00:49:20.268
Yeah, exactly. Exactly. Exactly.
322
00:49:20.308 --> 00:49:35.348
They also do a lot of work on, um, p- model parallelism, especially over, uh, you know, heterogeneous topology, where you have, you know, some sparks and they are wired together with, you know, Ethernet, G-DGX sparks.
323
00:49:35.388 --> 00:49:44.528
Um- Yeah, yeah. This is the Exo Labs guys. Yeah. You have, you know, a number of, uh, Mac Minis stacked up. [laughs] There's, you know, the inter... They, they...
324
00:49:44.568 --> 00:49:50.988
One thing that, that I think we both have to deal with, although they have to deal with a lot more, is, is the interconnect between machines.
325
00:49:51.288 --> 00:50:03.588
Um, which is why like, you know, one thing that we do a lot is work with tensor parallelism. Mm-hmm. And that's where, uh, you are using all of the, you know, all eight GPUs, um, and, and sharding the model across it.
326
00:50:04.048 --> 00:50:11.188
Uh, tensor parallelism is not a good fit for local AI because it assumes a very high bandwidth interconnects like NVLink. Uh, was, you know,
327
00:50:12.108 --> 00:50:18.248
they might be forced to do something like pipeline parallelism, which we're never gonna do unless we're doing some kind of- Yeah. For image... multi-node inference.
328
00:50:18.348 --> 00:50:26.728
But since you mentioned it, I actually wasn't sure if we were gonna cover it, but let's briefly explain tensor parallelism and expert parallelism, uh, since you have very nice images. Do you wanna pull the book? Yeah.
329
00:50:26.988 --> 00:50:36.968
Yeah. Let's, let's get the- So I just wanna show off your images. [laughs] Yeah. Uh, shout out, shout out to Luke from Base10's design team for making these, these beautiful images. Oh, that's a... That's actually...
330
00:50:37.028 --> 00:50:50.188
Before we get into this, just one other difference is we, we talk a lot about the active parameters of a mixture of experts model, and for local inference folks, that matters a lot because if you have a batch size of one, you're only activating that many parameters.
331
00:50:50.748 --> 00:50:56.548
When we do- Yes. I was gonna-... inference in the data center- I was gonna bring it in the diffusion conversation. Uh, yeah. [laughs] Yeah. We, uh... At, at...
332
00:50:56.588 --> 00:51:07.368
When we go through like a, a MoE model, um, and we host it, um, for an API, we assume that all parameters are gonna be active because- You're batching... throughout your batch- Yeah...
333
00:51:07.388 --> 00:51:16.948
you're gonna, you're gonna hit everything. Um, cool. So broadly, tensor parallelism you can do with any model. Expert parallelism, you can only do with MoE models.
334
00:51:17.188 --> 00:51:27.948
Um, effectively all models today are MoE models, um, that are, you know- Sort of- At least all models large enough that you would care to parallelize them across multiple GPUs. So that, that's...
335
00:51:27.988 --> 00:51:35.488
That, that nuance is, is less important now. Um, with expert parallelism, the idea is you put the entire expert on a GPU.
336
00:51:35.568 --> 00:51:42.648
Generally, you have more experts than GPUs, so you might put like N experts per GPU, like eight experts per GPU or whatever.
337
00:51:43.248 --> 00:51:49.428
And then you, uh, you replicate the router, which the router is very small, across each of the GPUs.
338
00:51:50.068 --> 00:52:02.428
And then by moving the, the generation from expert to expert, with each expert being inside a GPU, they're not competing for resources. You massively increase the throughput that you're capable of doing.
339
00:52:02.948 --> 00:52:15.568
And the, uh, GPU-to-GPU connection b- is, is not as important 'cause there's not as much communication. Uh, tensor parallelism requires that you are able to do this like all gather, all reduce.
340
00:52:16.248 --> 00:52:30.168
So you basically shard the model across the GPUs entirely. Um, and then for each step, you're combining the results of each of the GPUs, which is why the interconnect matters a lot. And it is generally...
341
00:52:30.248 --> 00:52:35.228
Of course, this is a, this is a very high-level generalization. There's a lot of places where this is not correct.
342
00:52:35.548 --> 00:52:47.928
But generally, uh, TP is, is helpful for latency, and in many cases, you will use some combination of these two parallelisms, um, across the, the model rather than just like picking one or the other.
343
00:52:48.328 --> 00:52:55.087
Do you want, you wanna, you wanna add some color there? Yeah, usually, like in a model, it's not... They're not mutually exclusive. You do tensor parallelism and you'll do expert parallelism.
344
00:52:55.128 --> 00:52:59.228
Pipeline parallelism less solely, it seems to me like we never use VPN. Yeah.
345
00:52:59.268 --> 00:53:14.368
The only reason you would have to do pipeline parallelism, um, which is where you separate like different layers and you put like half the layers on one hardware and half on another, is if you are forced to do multi-node inference, um, because a model is bigger than you have the...
346
00:53:14.388 --> 00:53:26.808
Like- Mm-hmm... let's say, let's say you're doing a deployment on H100s for whatever reason, and you're putting a, a trillion-parameter model on there. You have to use multiple nodes of H100, and so you... The...
347
00:53:26.828 --> 00:53:39.688
Because the interconnect is so slow between the nodes, the only viable way to parallelize there is, is pipeline. But then you would do expert and tensor within each node. And the limiting factor for H100s is HBM? Yeah.
348
00:53:39.708 --> 00:53:48.608
They just don't have enough of- How much is- What, what, what's the magic numbers that we need to- Like on a, a B200 is 180 gigabytes per GPU, and then a node of eight, so you're talking like 180 times eight.
349
00:53:49.108 --> 00:54:01.828
And we have before, so each parameter takes half a byte, so that's 800 gigabytes. On a H100, it's like 140? It's 80. It's 80? Yeah. Oof. Yeah. [laughs] I'm old. I've been doing this a long time.
350
00:54:01.888 --> 00:54:11.348
I actually remember H100 specs. [laughs] Yeah. No. So, so one thing- Do you wanna tell me about the T4s? The T4s. Oh my God. Let me tell you what it was like to run a model on a T4 back in the day.
351
00:54:11.388 --> 00:54:20.188
[laughs] Um, o-o-one thing I was surprised to see that more people didn't do, uh, Jamba. I don't know if you guys remember Jamba from AI '21.
352
00:54:20.288 --> 00:54:29.928
They would actually specifically pick a hardware, and then they, they, they designed the arc dimensions for the hardware, and then it would obviously saturate the hardware. Like it makes sense.
353
00:54:30.348 --> 00:54:39.488
And like somehow all these models don't do that. Don't they do this for the training side, though? Uh, I don't know. The what, sorry? The- Training. For training the model. Like deciding which GPU, which GP- Yeah.
354
00:54:39.648 --> 00:54:45.224
Well, how to- Yeah, yeah. They, they do And with training, it's more of like a, a math. Like you can run the math- Yeah... and see the flops and maximize it.
355
00:54:45.784 --> 00:54:52.784
With inference, it's more of like an auto-tuning, like if you feel like GPU kernel auto-tuning. But like it's, it's basically like you, you define that, "Oh, I have two GPUs.
356
00:54:52.864 --> 00:54:57.724
I can do TP1, TP2, EP1, EP2," for instance, right? And you...
357
00:54:57.904 --> 00:55:06.624
So that, that gives you like total of like two squared combinations, and then you just like you shadow the same traffic, like real prod traffic, and you just see which configuration gives you the best TPM, TPS, and then just use that.
358
00:55:07.184 --> 00:55:14.504
I don't like the fact that it's-- you cannot reason about which one's gonna give you the best performance or that there isn't one specific configuration that's always best.
359
00:55:15.164 --> 00:55:20.784
But it seems like auto-tuning is just the way that you find the best one. And with kernels and GPU kernels, it's much of the same.
360
00:55:20.844 --> 00:55:28.244
After you, you design your kernel and des- you design your configuration, how many threads do you launch? How many, you know, how much shared memory do you use? You just, you just auto-tune.
361
00:55:28.284 --> 00:55:34.264
You just sweep the parameter space on the side, and this is the best one empirically. But yeah, but they are, they are combined. They're not just- Yeah... in like, like separation.
362
00:55:34.624 --> 00:55:44.644
There's a few bits of training that are kind of like hardware targeted. If you look at, for example, NVIDIA Nemotron models, they run very, very well on Blackwell. That's, that's unsurprising.
363
00:55:45.004 --> 00:55:56.944
Um [laughs] So there's some degree of that, but I think that most open labs are trying to make models that can be run on as wide of hardware as possible rather than targeting just like a single chip. I see.
364
00:55:57.204 --> 00:56:04.024
For u- usefulness. Yeah. Uh, okay, one more thing while this chart is still up. All gather, all reduce is expensive. Mm-hmm.
365
00:56:04.284 --> 00:56:15.584
One of the things that is a movement in Silicon Valley is mega kernels, just keep fusing kernels. I don't know [laughs]. Is it that simple? Well, I, I mean, like a fused kernel can't save you.
366
00:56:15.684 --> 00:56:20.404
Like, like here with, with, with tensor parallelism, you're... The half the matrix is on one GPU and the other half is on another.
367
00:56:20.444 --> 00:56:30.104
And if I need the entire matrix in order to do like a nonlinear operation on the next step, which is, for instance, like if I'm doing attention, I need the softmax or I need to do like, like exponentiation, I need to have the entire row.
368
00:56:30.544 --> 00:56:39.724
So I need to know what the partial result was from GPU 2 and what the partial result was from GPU 1 in order to be able to do the softmax in the next stage. So I, I...
369
00:56:39.844 --> 00:56:49.084
Like I have to make them communicate with each other, even if I had a fused kernel because of the non-linearities within each one. Also with, with like mega kernels, like honestly, I'm, I'm, I'm very bearish on- Ooh...
370
00:56:49.124 --> 00:56:58.864
on... I'll be honest. Please, please, please. [laughs] No, it's just like mega kernels, it, it was a good research direction and it seems like a very... Like, like intuitively, theoretically, it's nice. Like, oh my...
371
00:56:58.984 --> 00:57:03.624
Like you have a lot of launch overhead from launching one kernel- Yeah, just keep fusing it... moving the data. Just fuse everything together.
372
00:57:03.764 --> 00:57:14.504
[laughs] But yeah, but like, like the, the kernel complexity itself is, is, is very difficult to write a very optimized mega kernel. It's, it's, it's very, very difficult to do so. And even the...
373
00:57:14.784 --> 00:57:34.324
Like not to name any companies, but like even the companies that have, have worked or people that I've spoken to who work at companies that do fused mega kernels, they very, very often don't end up running those in production because the CRTL and modular kernels that launch are faster because you can optimize each individual component and you can just have them parallelize with each other.
374
00:57:35.124 --> 00:57:44.424
With the Rubins, I don't know if you guys saw the Rubins Twitter post yesterday, but they're also, um, he- Ru- Rubins? Like, like- No, no, like Rubin, like the GPU. NVIDIA GPU... Yeah, GPU. Yeah.
375
00:57:44.444 --> 00:57:49.624
They have a Twitter account for Rubins only? No, no, no, no, no. Okay [laughs]. [laughs] I was like, "What are you talking about?" Sorry.
376
00:57:50.064 --> 00:57:57.364
One of, one of the tech leads at NVIDIA like launched a Twitter post said like, "We're pulling the curtain on Rubin, and here's the, here's the specs." And, and the, and the third tweet showed...
377
00:57:57.504 --> 00:58:08.344
Like not to get too technical into it, I ha- and I need to read it much more, but the GPU is, is, is designed in such a way that it basically kills mega kernels. You don't need to use mega kernels that much anymore.
378
00:58:08.864 --> 00:58:15.804
So it seems like that entire research field goes into like, won't be continued. But yeah. Can I speculate about Rubin for a minute, please? Go.
379
00:58:16.184 --> 00:58:20.504
Um, you know, I've been through now, we w- we- And by the way, they are covered in the book. Yeah.
380
00:58:20.524 --> 00:58:28.144
But yeah, they- Well, I mean, they're covered in the book in the sense that like [laughs] I am aware- The Wikipedia entry... from the blog post- Yeah [laughs]... that Rubin is going to happen in the future.
381
00:58:28.164 --> 00:58:35.864
[laughs] And you even had the, the name of the one, Feynman. Yeah, it's like, "Hey, this is, this is gonna be- I was like, "This is very up to date." Like [laughs] I'm trying to future-proof this thing, okay?
382
00:58:36.404 --> 00:58:48.904
Uh, I don't wanna publish a new one until like next year or something. Uh, anyway, so we were discussing the degree to which I am old. Um- Mm-hmm... and you know, I've now been through three hardware launch cycles.
383
00:58:48.964 --> 00:59:02.104
I've been through the Ampere launch cycle, the Hopper launch cycle, and the, um, Blackwell launch cycle. Now, when I say launch cycle, I don't necessarily mean like the, uh, the actual shipping of the hardware.
384
00:59:02.143 --> 00:59:14.324
Like Ampere's were racked up well before I got in this industry. But there is a lot of time between hardware being racked up and hardware being sort of feasible for inference.
385
00:59:15.144 --> 00:59:23.864
So if you look at like the original vLLM and SGLang, or vLLM especially, like that was written targeting Ampere and then had to be updated for Hopper, updated for Blackwell.
386
00:59:24.804 --> 00:59:30.844
With each of these cycles, it becomes faster and more urgent, but also substantially more complicated.
387
00:59:31.704 --> 00:59:44.584
When I look ahead to, you know, what's going to be new with, with Rubin, I think that like Dynamo gives me a lot of technical hints around like what kinds of work is going to be very valuable.
388
00:59:45.124 --> 00:59:54.644
Obviously, we're continuing some trends from Blackwell, right? NVF4 is big. The amount of compute that they have behind NVF4 tensor cores is, is massive.
389
00:59:54.724 --> 00:59:59.204
Um, we'll, we're gonna talk about video, I think, at some point, and, and that's the, the big barrier there.
390
00:59:59.864 --> 01:00:10.364
You've got, you know, much, much faster memory bandwidth, um, but which was the same thing that, that made Blackwell so good. Um, but the, the big thing is more systems thinking.
391
01:00:10.464 --> 01:00:17.624
You have more emphasis on the CPU to GPU interconnect, more emphasis on the interconnect between GPUs.
392
01:00:18.184 --> 01:00:26.404
And when you look at Dynamo, it's a system entirely designed around, "How do I move the KV cache to where it needs to be when it needs to get there?"
393
01:00:26.964 --> 01:00:58.084
Um, so I think that themes around like KV cache offloading, KV-aware routing, and, and disaggregation are going to be substantially more important in the Rubin era, which means that inference engineering becomes not just a like CUDA kernel problem, but also like a very traditional hardware infrastructure problem, which is something, you know, we've been building toward for, for a long time, and something that's like very exciting to me because we're gonna see Sort of multiple domains colliding and
394
01:00:59.044 --> 01:01:07.584
the ability to reason from the kernel level, like up to the hardware level and, and back down is, is going to be very valuable. I will take what Phil said one step further, actually, into that.
395
01:01:07.644 --> 01:01:14.344
It's, I think, trending towards becoming exclusively an infrastructure problem, where like problems of PG, DISAC, Training, Spectac.
396
01:01:14.964 --> 01:01:27.584
But trotting kernels is not going to be much of a problem because the GPU is moving more towards being an ASIC, where it's-- you're just, you're just trying to orchestrate what happens on the GPU, but you're not actually controlling it thread by thread level.
397
01:01:27.624 --> 01:01:35.884
And you see this with like QTAL, QDSL, like you're, you're just working at levels of like tiles of data, but you're no longer working at controlling what each thread does on the GPU that's being taken care of for you.
398
01:01:36.484 --> 01:01:50.744
Um, so I guess, do you agree that a GPU and future GPUs are trending more and more towards becoming ASICs that just need to be launched and then they do the data operation based on your conversations with other people?
399
01:01:50.824 --> 01:02:01.184
Oh, I mean, yeah, no. Uh, that, that, that is a section of the market. Right. And obviously, ASICs can do, uh, a lot more performance for only their, their workload. Right.
400
01:02:01.544 --> 01:02:09.824
And the, the G in GPU makes, makes them continue to be very general. Yeah. The, the-- I think that there's like a spectrum- Actually, it's, it's graphics, but- Yeah.
401
01:02:09.924 --> 01:02:16.724
[laughs] I, I, I keep saying this, I have to correct myself in s- in case people come at me for getting the, the G wrong. Yeah. It's, it's like, it's like a spectrum, right?
402
01:02:16.944 --> 01:02:29.204
Of, of a very, very general-purpose compute to something like a Talos, where you've got the, the hardware built for a specific set of model weights. The weights burnt into the chip. [laughs] Yeah. No loading.
403
01:02:29.884 --> 01:02:40.964
I don't-- I wouldn't say that like, that, that we're, we're, we're going all the way there. It's, it's more like along the spectrum, it's a step in the direction of more specialization within the hardware. Yeah.
404
01:02:41.444 --> 01:02:49.904
I'm curious, I feel like he was driving towards something. I guess my point is being bearish on... Like, you say, you say, like everything else apart from burning the weights into the chip.
405
01:02:49.964 --> 01:02:55.144
Burning weights into the chip is like impractical because you wanna fine-tune, you wanna optimize, you wanna, you wanna quantize, you wanna release new checkpoints of the model.
406
01:02:55.444 --> 01:03:05.504
If it's burned into the chip, the chip's useless in like a month or two, right? I guess, I guess my point is, how can you not-- uh, like seeing Nvidia more and more specialized, like take its GPUs from a general
407
01:03:06.644 --> 01:03:09.924
programming paradigm where you're just-- it's a general computer that you can use to program threads.
408
01:03:10.624 --> 01:03:22.264
And with every new generation, you're putting more and more specialized instructions, specialized tensor cores, specialized, you know, MMA instructions, uh, things that will, will allow you to just control it almost as an ASIC, almost as a collection of ASICs.
409
01:03:22.424 --> 01:03:33.344
Mm. How can you look at this trend and then still be bullish on companies that are coming up with ASICs for AI? Mm. In the sense that, in the sense- Yeah, because they're, they're sort of re- Right...
410
01:03:33.444 --> 01:03:45.144
they're evolving towards that direction. They're almost evolving towards an-- like, like as an, as in Rubin, I guess comp- like compared to Ampere or, you know, T4, Rubin is, is, is basically an ASIC.
411
01:03:45.184 --> 01:03:52.784
It is-- It's basically just a thing that is, is used- Programmable ASIC? Yeah. It's like, yeah, like you can program obviously, obviously like, like I, I, I guess I like... It's very controversial to call it an ASIC.
412
01:03:52.844 --> 01:03:57.784
It is, it is a GPU. It is a-- It is general. It does have threads. I can write CUDA to control it and change its operations.
413
01:03:58.344 --> 01:04:06.244
But, but it has the systolic arrays and tensor cores and TMAs and, and tensor memory, and it has these things that are almost exclusively useful for loading model weights.
414
01:04:06.284 --> 01:04:14.044
It has, um, you know, tensor core instructions that are almost exclusively shaped around the head dimensions of models that exist in the market today.
415
01:04:14.504 --> 01:04:20.164
To say that you're gonna come up with an ASIC and you're gonna etch something into it, well, but the next architecture is basically gonna be useless. Yeah, I don't know. I don't know.
416
01:04:20.964 --> 01:04:31.244
I think that the thing to remember is just how long these hardware cycles are. Yeah. So if a chip is coming out today, that means the design process for it was kicked off years ago.
417
01:04:31.844 --> 01:04:41.844
And they've-- at Nvidia, they've done a very good job of predicting where the market is going to go and, and, you know- I mean, they have the most information, I'm sure. [laughs] Of course.
418
01:04:41.944 --> 01:04:55.864
But if you, if you look at, you know, there being public open source model architectures that look more or less like early versions of the one today, Rubin is honestly the first chip that was fully built in that world.
419
01:04:56.504 --> 01:05:05.584
And so you can see a lot of the understanding of the shape of the workload that this chip's going to be asked to do in the way it's designed. Yeah. Okay.
420
01:05:05.684 --> 01:05:09.284
So, so I'm not gonna be the best person to, to directly answer those questions.
421
01:05:09.364 --> 01:05:16.884
I think these are very fair questions that are-- obviously, the first one that's based on Rubin that like I've, uh, heard artic-articulated so well.
422
01:05:17.404 --> 01:05:30.044
I do think that, uh, I, I will make a case for a vertically integrated model lab ASICs. Mm-hmm. So like the OpenAI, Broadcom, what-whatever, Jalapeno- Sure. Yeah, yeah... uh, chip, which like totally makes sense.
423
01:05:30.124 --> 01:05:38.884
Like, uh, so I-- we first had this on the pod with, uh, Martin Casado, where he was like: "Look, if you have a, a, a trillion-dollar or five hundred billion dollar training run- Mm-hmm...
424
01:05:39.064 --> 01:05:46.904
then take fifty billion of that and make a ASIC. Like it's fine. Like, like you will get more than ten percent efficiency from, from the ASIC." And like, that makes sense. Right. Right?
425
01:05:46.984 --> 01:05:58.804
So, so like a model-specific chip, yes. But ASIC companies, the interesting thing is I feel like you are focus-- you're hyper-focusing on like, like you say, like the Talos stuff. Right.
426
01:05:58.844 --> 01:06:14.164
They are doing a lot more sort of like, uh, surface area engineering or like, like, like the actual allocations of memory and hardware and like the, uh, the communication between, between chips that, um, probably still won't be touched by Rubin, but I don't know the details.
427
01:06:14.444 --> 01:06:23.824
I see. I see. They-- Typically, they often talk, talk about things that I would expect to have bigger orders of magnitude- Mm-hmm... than would be programmably accomplished by whatever Rubin does.
428
01:06:24.224 --> 01:06:35.924
But who know-- who knows? [laughs] No, I see, I see, I see. [laughs] Yeah. It seems, uh- Yeah, like I mean, you know, like think about what is-- what are the real blockers to ten x to one thousand x faster inference.
429
01:06:36.304 --> 01:06:45.864
It is not the stuff that can be rearranged, uh, just within the existing GPU design. Inter communication. Yeah. Okay. Like these, these guys are aiming for three hundred thousand tokens per second.
430
01:06:46.224 --> 01:06:51.084
They're not fucking around. [laughs] Like, um- Might have to put on some X6. [laughs] Maybe.
431
01:06:51.724 --> 01:07:00.204
I think, you know, it is interesting to me that you're so bearish on so much of this kernel engineering work, given how much of it you've been doing recently. Right. Right.
432
01:07:00.224 --> 01:07:07.032
But like the more I do it, the more it just seems to me that- It's not mega I would also add like- There's generations of models being out, right?
433
01:07:07.272 --> 01:07:17.152
I think on your guys' end, you see a lot of, okay, one day it's GLM, Qimi, DeepSeek, uh, Minimax, throw in the others. Some are doing completely different stuff, right? Gemma, no encoder.
434
01:07:17.512 --> 01:07:28.632
The latest thinking machines is all from scratch. But when you look at the other side, like how long have we been on the GPT 5 generation, right? Right. They've been serving that thing for quite a while.
435
01:07:28.752 --> 01:07:37.712
Sure, there's maybe more pre-training. There's, there's different checkpoints, but like you actually can squeeze quite a bit out and you do a multi-billion dollar train run.
436
01:07:38.032 --> 01:07:43.832
If you can make it X percent more efficient, they, they serve it for a while. Same with, uh, say, the Claude Five set, uh, family, right?
437
01:07:44.092 --> 01:07:50.792
Like if they release a new model, like if they release GPT 6 now or whatever- Yeah, yeah... and they release a new model every year, and we-- well, we don't know.
438
01:07:50.852 --> 01:08:02.972
But if we assume that they're changing some bits of the architecture and not just doing like post-training, like you're gonna be spending fifty billion dollars a year every single year coming out with new ASICs for the model and throwing out the ASICs of the previous year away.
439
01:08:03.152 --> 01:08:12.532
Yeah. Yeah. [chuckles] Easy. So, so I think, okay, um, I, I would slightly disagree based on my con-- again, my- Yeah, yeah... uh, it's all secondhand, uh, on the longevity of a model. Right.
440
01:08:12.552 --> 01:08:23.452
There's still people out there using 4.0. Yeah. Yeah. Llama... Not Llama 2, but Llama 3. I still see Llama 3 workloads. Yeah. Because if, if it's done, if it's trusted, uh, don't change it. If it works.
441
01:08:23.632 --> 01:08:34.172
[laughing] Which is one of the promises of open source, right? Like the whole 4.0 save, 4.0 movement. Like you don't gotta have a save Llama 3 movement. You just gotta have an A100 somewhere.
442
01:08:34.252 --> 01:08:45.252
[chuckles] I think at some point there's also the question of, okay, if a model can do enough and use enough tool calls and be agentic enough, uh, can it just web search, tool search- Mm-hmm... write code?
443
01:08:45.652 --> 01:08:51.752
Do you really need to keep squeezing more? We will because you guys will make it cheap and fast and smaller, and I can swap it in.
444
01:08:52.172 --> 01:09:01.892
But at some level, like you give me 5.2 today or say whatever 120 B model, I can run with it for quite a while, right? This is assuming like you don't need intelligence.
445
01:09:02.332 --> 01:09:08.552
I think there's a lot of intelligence where we are- You need, you need reliability and predictability. Like I'm in enterprise like- Mm-hmm... like this is tried and tested.
446
01:09:08.612 --> 01:09:18.712
It is signed off by like my five thousand stakeholders. Right. Like I'm not touching it. [chuckles] It runs a batch job every day- [chuckles]... and I like the results. Yeah. The results are predictable. Yeah.
447
01:09:18.752 --> 01:09:28.732
Yeah, yeah. It doesn't make sense to keep using them. Like stuff gets sparser, cheaper, better. Right. But that doesn't mean that old models, GLM 5.0 isn't usable, right? Mm.
448
01:09:28.832 --> 01:09:38.372
If we hit a stall, say, for whatever reason, there's still a lot that can be squeezed out. We're gonna run out of time. I, I, I did wanna also make sure... Yeah. Yes, actually we happen to have this diagram.
449
01:09:38.412 --> 01:09:47.992
Pull-- compare this versus any Cerebras diagram, right? Um, I don't think Edge10, uh, medics have, have put out public, uh, charts yet. Uh, but the complete re-- the real estate is very different.
450
01:09:48.012 --> 01:09:55.552
The size is very different, right? This is not wafer scale, right? This is-- there's probably like, I don't know, a few hundred of these on a, on a wafer. Um, I don't, I don't know how big- Right, right...
451
01:09:56.072 --> 01:10:03.532
the, the, the comparison is, but like, uh, it is a, it is a very like real estate allocation- Few dozen... uh, difference. Few, f-few dozen, I would say. Few dozen. Yeah.
452
01:10:03.552 --> 01:10:12.732
Before we move from hardware, I have two quick questions. One, the latest Qimi, which is really big, uh, three trillion- Yeah... doesn't fit on most hardware on single node. Yes.
453
01:10:13.012 --> 01:10:25.432
You, you need, you need GB300 to fit it on a single node. You need GB300 or AMD. Um [chuckles] It's simple math. Um, NFVP 4, two point eight trillion parameters, uh, one point four terabytes.
454
01:10:26.052 --> 01:10:38.992
Um, the GB300s have, uh, two hundred and eighty-eight gigabytes each. Um, so a-across eight of those, um, you have enough room for the model and honestly like...
455
01:10:39.112 --> 01:10:48.432
So the other thing with, with GPU VRAM math is you have to leave space for the KB cache, and that's going to depend on, to some degree, on the context length.
456
01:10:48.612 --> 01:10:56.592
Um, so when a model is b- both has a very large number of parameters and a very long context length, you're kind of like fighting over space.
457
01:10:56.832 --> 01:11:09.812
Um, which is why, you know, the KB cache offloading, uh, would become like a, a more salient topic, I think, with these huge models, uh, 'cause you, you just-- you're very crunched for space.
458
01:11:10.332 --> 01:11:16.312
With the Rubens, you now have what? NVL 72 rack of- What?... 20 terabytes of your- Yeah.
459
01:11:16.352 --> 01:11:24.492
Now, now you, you still have NVL 72 on, on, uh, Blackwell as well, but, um, you, you can't necessarily assume you're gonna do inference on that. Mm-hmm.
460
01:11:24.572 --> 01:11:37.192
Um, there's a whole lot more 8X racks in the world than there are NVL 72s. Yeah. I guess my last quick question on hardware was, do you notice anything with hardware generations for new pre-trained base models?
461
01:11:37.252 --> 01:11:50.212
So one of the things you said for efficiency is you can swap hardware. That's one of the 2X gains. Um, when we see new stuff coming out training-wise on Rubens, uh, any changes on logs?
462
01:11:50.292 --> 01:11:57.492
Does this affect what type of models we will be seeing when these are more available? And can you- They, they get bigger.
463
01:11:57.552 --> 01:12:10.532
Like people understand the ceiling that you have in terms of how many parameters of a model you can run, uh, given the sort of latest inference hardware, and that kind of forms a ceiling.
464
01:12:10.552 --> 01:12:25.392
And so, for example, when DeepSeek R1 came out, it was, you know, it was six hundred and seventy-one billion parameters, which at the time was really huge and I think did a lot to push us to really quickly adopt Blackwell and, and get good at serving on Blackwell.
465
01:12:26.112 --> 01:12:42.032
Um, so yeah, it's, it's mostly in my mind about, uh, model size and then about matching the architecture and the native quantization to the, the target hardware, like we talked about with like, you know, all Nemotron models of NVFP 4, for example.
466
01:12:42.712 --> 01:12:53.612
So we talked a lot about LLMs. Mm-hmm. You have a lot more in the book. What about audio, video? What's the other side of inference engineering? Ali, you're pretty big in video diffusion.
467
01:12:53.932 --> 01:13:03.052
Video diffusions, I think, are like they're just shaped... A lot of the stuff that you can think about, reason about with LLMs being autoregressive. With video diffusion, it's, it's, it's not the case.
468
01:13:03.112 --> 01:13:09.400
For instance, you don't do- You don't do batching. You-- every request just comes in on one GPU and it serves one GPU. You don't have to shard.
469
01:13:09.860 --> 01:13:20.160
The models are a lot, are a lot smaller, like Wan2.2, for instance, is a twenty billion parameter model. Um, you don't need to worry about... So it's like orders of magnitude smaller than the best LLMs.
470
01:13:20.860 --> 01:13:31.240
And it's one of those spaces where the, the open source models are like with LLMs, we see Chemica 3 is almost comparable to, you know, Mythos or like GPT 5.5.
471
01:13:31.940 --> 01:13:39.900
The, the, the difference between the best open source LLM and best open s-- uh, closed source LLM is very small. Like it used to be six months. I don't think it's six months anymore.
472
01:13:39.940 --> 01:13:43.980
I think it's like basically almost on parity. Video models are definitely not. There's a huge gap.
473
01:13:44.040 --> 01:13:51.440
If you look at the best video that you can generate today with an open source model like Wan2.2 versus something like with Kling or Veo, difference is night and day.
474
01:13:52.020 --> 01:13:58.240
So it, it creates this disparity where media companies will choose to go most of the time to closed source models, for instance.
475
01:13:58.320 --> 01:14:05.560
And I were to tell you, "Hey, I can generate an entire three-hour movie for you with this model, and I'll optimize it so that you only have to pay me ten dollars."
476
01:14:06.260 --> 01:14:12.920
But if they were to do it on a closed source, they'd have to pay a thousand dollars, which is a hundred X. Like I'm a hundred X cheaper, but it's still a thousand dollars.
477
01:14:12.980 --> 01:14:15.240
They're still gonna choose to do all of their cuts with Veo on Kling.
478
01:14:15.820 --> 01:14:23.920
So the-- it's like a chicken and egg cycle where less demand causes less innovation in the field, causes, you know, less open source checkpoints to be released.
479
01:14:24.500 --> 01:14:32.420
And some of the labs that we're releasing open source models like Wan will have closed source their latest models, like Wan2.7 is not open source. We're still on Wan2.2.
480
01:14:33.160 --> 01:14:42.020
The, the challenge with, with video models especially is the number of, of tokens. So, so, so, so video models, you want to generate a high quality model, a high quality video.
481
01:14:42.700 --> 01:14:48.340
So let's say you're doing sixteen frames per second, that's like the absolute minimum you'll do, and let's say you'll do like 480p video.
482
01:14:48.900 --> 01:14:55.800
So you can think about your like dimensions and I think have like a good, um, just like a diagram that shows the number, the sheer number of tokens, right?
483
01:14:56.580 --> 01:15:05.640
Let's say you're looking at like just one video of like, you know, Sparta, Sparta 300 or whatever. So let's say we're looking at like four frames, right? Those four frames of that video, if you go just...
484
01:15:05.840 --> 01:15:17.200
if you're doing full attention, if you go a little bit up, like you're looking at, uh, 480p by 720 by 81 frames in just five seconds, 'cause sixteen FPS by five, right?
485
01:15:17.540 --> 01:15:23.500
And then you compress it down to latent space, but you're still doing thirty by like fifty by twenty-one tokens. Yeah.
486
01:15:23.580 --> 01:15:29.640
Which means that for attention for just five seconds, you're running attention on thirty-five thousand tokens, right?
487
01:15:29.980 --> 01:15:38.740
So the attention becomes such a huge bottleneck, and because it's open squared, if you're doing like-- if you extend that to like ten seconds, well, it's just squared, twenty seconds, thirty seconds.
488
01:15:39.080 --> 01:15:46.500
So to generate a good cut scene of like one minute, it's almost impossible to do within the same compute time. And it's just, it's, it becomes unfeasible. You can't do it.
489
01:15:47.160 --> 01:15:54.780
And so you end up with moving towards two directions. Either you decide to do attention on the entire video at once, in which case you are forced to do sparse attention.
490
01:15:54.820 --> 01:16:05.220
So if you scroll back down to the origin, the, the video image, like you can see whereas on the left for instance, I would be doing full attention where every single token in that Sparta 300 scene attends to every single other token, and you can see the sheer number of like red patches.
491
01:16:05.800 --> 01:16:12.320
On the right, I'm only attending to each token only attends to like the top K or top twelve point five percent that's important to it, which can be like spatial.
492
01:16:12.420 --> 01:16:20.240
So like, you know, the token that represents the crown attends to like the head, the face, and then the head on the other frame and the previous frame, temporal locality, spatial locality, that kind of thing.
493
01:16:21.060 --> 01:16:29.860
This results in terrible video quality and the whole point of the, the post or the article here is to show like how you can train and you can do all these things, but you will still suffer your quality a little bit.
494
01:16:30.540 --> 01:16:39.260
So you end up with one of two things. Either you bite the bullet, you have huge compute, and you do full attention over like a million tokens because you're trying to generate like two minutes of video,
495
01:16:40.240 --> 01:16:50.020
or you move towards autoregressive video. Autoregressive video seems to me like that is the bet that the future's gonna be making, but there are no good open source autoregressive video models out there today.
496
01:16:50.580 --> 01:16:51.240
And that seems to be the...
497
01:16:51.300 --> 01:16:59.400
If you want to get like an hour movie, if you wanna see video models generating like an, uh, like, you know, Hollywood level movies, they have to be autoregressive in order to exceed that five second frame.
498
01:16:59.960 --> 01:17:07.980
Or there has to be some insane leap that happens in compute that allows us to do full attention over like millions of tokens at the same time in a, in an efficient manner.
499
01:17:08.600 --> 01:17:17.120
Even millions of tokens, it's like you're, you're quadratic, so you're gonna get there really quick. Right. I think, uh, can you explain the pros and cons trade-offs of autoregressive?
500
01:17:17.460 --> 01:17:27.620
So one that comes to mind is, you know, the consistency- Right, right... across frames. You will, ten minutes into generating autoregressive diffusion, you're gonna forget. But what, what are pros and cons of this?
501
01:17:27.860 --> 01:17:32.560
Well, like, like autoregressive LLMs, you can take a lot of your... Uh, sorry, autoregressive diffusion models.
502
01:17:32.600 --> 01:17:36.840
You can take a lot of your optimizations that we discussed with LLMs, like Spectac and stuff like that, and you can apply it there.
503
01:17:37.480 --> 01:17:47.980
And you can, if you have a very high quality scaled up model, there is no reason why I can't stream the outputs as in I can show you the first frame and then I'm like, like, kind of like GPT back in like twenty twenty-three when you're...
504
01:17:48.500 --> 01:17:56.020
Like now it's just almost like one shots the text, but back then you could read and it's generating as you read. With video models, you can watch and it's generating as you watch, it generates the frames.
505
01:17:56.120 --> 01:18:08.560
And, and so token by token generation will allow us to, to scale a lot up and apply the attention mechanisms there. The downsides is every single autoregressive video model is shit. It's just terrible quality.
506
01:18:08.640 --> 01:18:14.000
If you-- uh, like it's just, if you put, if you put the quality of any opens like Wan2.2 versus any other autoregressive model,
507
01:18:14.920 --> 01:18:23.260
you can see like a, like a video generated by Wan2.2 is like, you know, a cat and dog fighting. Autoregressive model will give you like degraded Tom and Jerry quality, I don't know.
508
01:18:24.040 --> 01:18:28.820
The, the solution to generating long output then becomes, "Okay, we're not gonna use autoregressive model. We're gonna..."
509
01:18:28.920 --> 01:18:38.620
If you look at some of the things that like, um, like Groq Imagine or Groq Video does, and they do it really, really well, is they'll, they'll try to stitch these, you know, seven second chunks together.
510
01:18:38.680 --> 01:18:43.900
And so you generate seven seconds and then you're like, "Okay, I'm gonna... Can you extend this video?" And they'll chunk two videos together.
511
01:18:44.560 --> 01:18:50.440
Open source doesn't seem to have the tricks that they have there and, and by definition it's closed source. We don't know what they're doing.
512
01:18:50.960 --> 01:19:00.680
But the closest you can get is taking the last frame of a video and feeding into like a, a text and image to video where it will take the text, the prompt, and it will take the image of the last frame, and you'll ask it to generate the next five seconds.
513
01:19:01.060 --> 01:19:08.596
And that's kind of like how you, you can extend this level of a model to generate like a moo, where you're just, you're constantly streaming frame by frame. But you get drift.
514
01:19:08.876 --> 01:19:17.716
So you start with like you take the image, and then you generate a video, and then that next five-second video is like lower quality, and the third chunk is like even lower, and the fourth chunk is even lower.
515
01:19:18.136 --> 01:19:28.816
And like sometimes you'll see things where like the, the new video is like just ever so slightly darker than the first one, and the next one is darker than the second one until like twenty-five seconds and you have black screen.
516
01:19:28.856 --> 01:19:37.776
[chuckles] Like it's just-- it's, it's, it's a... We tried to have a demo that would show this, but it was like-- it was, it was extremely embarrassing to show. Like we just decided not to because it seemed to like...
517
01:19:38.356 --> 01:19:47.816
But it is, it is, um, I think models will get there. They just need to, in my mind, scale up significantly and move towards being autoregressive. But the training techniques don't seem to be clear there.
518
01:19:48.156 --> 01:19:55.096
For those, um, who are interested in Grok Imagine, we did a pod with Ethan Ha from, from that team- Right... uh, who dropped a little-- a few hints, but not that- [chuckles]...
519
01:19:55.396 --> 01:20:03.836
not, not, not enough that we can fully reconstruct everything. Specifically on this part, the-- he explains a bit about- Yeah. So we talked about memory and, and, uh, longer context and all these things.
520
01:20:04.276 --> 01:20:09.816
But as far as I know, the- it's not autoregressive, even the... Like no one in industry is autoregressive. Yeah. It seems to be. Yeah.
521
01:20:10.156 --> 01:20:20.256
The key thing to understand between a autoregressive model and a diffusion model is that diffusion attention goes in both directions, while autoregression, it only goes forward in the sequence.
522
01:20:20.856 --> 01:20:24.856
So that's why you see this sort of like going off the rails behavior both in...
523
01:20:25.456 --> 01:20:36.216
If you sort of naively construct a video generation model as simply generating a linear sequence of frames, you can't then go back in that sequence and fix something to make the whole thing consistent.
524
01:20:36.336 --> 01:20:42.316
While, of course, the reason that we need all this latent space for the video model is, like you said, we keep all the tokens in memory. We
525
01:20:43.256 --> 01:20:51.596
iterate over that, that full sequence, and c- uh, you can, you can adjust the past in, in order to make the future make sense.
526
01:20:51.696 --> 01:21:08.076
So if we think about the architecture that's gonna get us there to these longer, richer sequences, it's probably, like you said, gonna be a mix of the, the autoregressive and the diffusion, um, wo- working together to do what each piece is good at.
527
01:21:08.556 --> 01:21:15.935
Well, if you get... Like you, you intuitively get why. So like English, for instance, or just writing in language, it's like it's just left to right. You can stream your tokens. You can stream your chain of thought.
528
01:21:16.076 --> 01:21:17.636
Just even as a human, you write like you just...
529
01:21:18.215 --> 01:21:22.756
You write and then you think about what's the next thing you're gonna generate, and then you write that, and then you think about your ideas, and then you generate forward.
530
01:21:22.776 --> 01:21:32.376
And sure, you can argue that as you write, you need to go back and you wanna edit some things. But you need to do that, you know, less often than you'd think. Whereas with video, there is no sequential...
531
01:21:32.896 --> 01:21:42.476
You know, the, the pixel in the top left corner of the video and the pixel in the bottom right corner of the video, they both need to attend to each other to understand how the video quality is gonna be almost as equally.
532
01:21:42.876 --> 01:21:52.556
Whereas with text, you don't need that as much. Is there a parallel to audio? Like I'm not a hundred percent confident on this, but there was a point about a year ago where there was audio LM.
533
01:21:52.596 --> 01:22:03.036
There's diffusion for audio and autoregressive, and for the points you mentioned, mostly on the inference side, even though they're shorter clips, you know, most music is three to five minutes, um- Mm-hmm...
534
01:22:03.056 --> 01:22:11.636
we've basically swapped over to autoregressive for- Yeah, I can't speak to music, but speech is autoregressive. Yeah. You, uh, effectively...
535
01:22:12.056 --> 01:22:16.096
I mean, this, this was even back with like the Orpheus architecture a year and a half ago.
536
01:22:16.156 --> 01:22:27.236
You just add a bunch of waveforms to the vocabulary so that the LLM can output tokens that represent those waveforms, and then you construct speech, and that's how you stream it. That's it. Wow.
537
01:22:27.356 --> 01:22:37.516
That's my AIE talk from 2025. Ah, nice, nice, nice. Nice. But it's not... With audio, it's not the same challenge, though, is it? Because you... Like audio is solved with an LLM that generates everything.
538
01:22:37.616 --> 01:22:44.796
Like, like with audio, it's still a, a transcript that you can generate with an LLM. Yeah. So your, your audio model just needs to like transcribe it, text to speech.
539
01:22:45.236 --> 01:22:54.836
For music, there was a phase of a ta- a trade-off between diffusion for music and- Right... autoregressive, and they were both pretty on par. There's probably more pros than cons to either.
540
01:22:54.876 --> 01:23:03.536
I just wanted to poke and see if you had takes. Yeah, I don't know about music specifically. No. [chuckles] You know, i- what... With what you said about editing- Mm-hmm...
541
01:23:03.556 --> 01:23:10.476
you writing, you know, obviously, I think my editor would tell me I actually need to do that more often [chuckles]... and, and go back and fix things.
542
01:23:11.036 --> 01:23:23.296
I can imagine music or poetry, for example, where you have a rhyming scheme and you might wanna go back and make a change to make it, to, to make it easier to set up a rhyme that you wanna make later on.
543
01:23:23.816 --> 01:23:29.255
Uh, there, there being some advantage to being able to attend in both directions. Uh, but yeah, to my knowledge,
544
01:23:30.516 --> 01:23:41.476
you know, I, I very much bifurcate this, this influence problem into the autoregressive models, which have a set of constraints and techniques, and the diffusion models, which have a s- set of constraints and techniques.
545
01:23:41.516 --> 01:23:54.356
And, um, I think of text, um, embedding, uh, voice in and voice out as being in the autoregressive side, and then image and video being in the diffusion side. There's some overlap between the two.
546
01:23:54.396 --> 01:24:04.916
It's not a, a perfect split, but that's the broad categorization I use. I should point out, I think it's confirmed, right, NanoBanana and, uh, GPT Image are autoregressive image.
547
01:24:05.456 --> 01:24:17.056
It's kind of this blended approach that we're talking about, um, but in the, the image s- space, uh, it hasn't like made its way over to the video space, at least in the open source world. Yeah.
548
01:24:17.336 --> 01:24:25.776
But like I, I assume that's not too far away if, if that is possible on the- Right... on the... At le- at least the Qwen Image guys are trying it. Yeah, yeah, with the- Yeah...
549
01:24:25.856 --> 01:24:35.936
I'm really excited for Qwen I- Qwen Image III. I hope they open source it. And then I should, I should also mention on the diffusion for tech side, there's been some movement. Not a lot. Yeah, we've got Mercury.
550
01:24:36.296 --> 01:24:45.036
Um- You host Mercury? Yeah. Um- Nice, nice. There's the- Gemma as well, right? Diffusion Gemma? Uh, Diffusion Gemma is open source. Yeah. Um, and then, uh- And we- Yeah...
551
01:24:45.056 --> 01:24:59.076
on the science pod, we just have been releasing, uh, some, uh, virtual cell models that use diffusion as well. Yeah, they, they have built... It's definitely still in the sort of cheap, fast tokens, um, world. Yeah.
552
01:24:59.156 --> 01:25:08.916
Uh, we're trying to- It's... I think it's the wrong marketing, and I've told them this before. I was like: Look, like you're not gonna beat the optimizations that, you know, the other LLMs are gonna do.
553
01:25:09.476 --> 01:25:19.856
Uh, but you can, you can have different APIs. Like you, you should be able to use it differently than chat response, chat response. Mm-hmm. Me also. Because it's diffusion. Because you can do like...
554
01:25:20.216 --> 01:25:31.468
What, what is like context-free guidance for diffusion look like? Uh, for, for text. Like give me a po- give me a poem, give me a s- uh, plot structure that like diffuses into place Exactly.
555
01:25:31.508 --> 01:25:39.288
So that's where, you know, like I mentioned with, with poetry, for example, where you might want to ensure consistency across your website. I've done a lot of LLM sonnets.
556
01:25:39.888 --> 01:25:47.048
It used to be one of kind of my go-to benchmarks, and even models today- They cannot count. Yeah... yeah, they, they, they don't get the syllables right.
557
01:25:47.088 --> 01:25:58.308
And if you can attend across all of the different tokens, uh, you can, you can get the syllables right. Yeah. Uh, b- and, uh, David Holtz from Midjourney was, uh, investing in text diffusion.
558
01:25:58.368 --> 01:26:06.248
I don't think anything came out of it, but like the idea was that you can storyboard a long movie, and then you can generate the scenes with video- normal video gen.
559
01:26:06.728 --> 01:26:18.708
But the, the idea of like coherence across, uh, a thing that would just appear where like the, the, the end should attend to the start and you should not have this autoregressive path dependency does make sense in principle.
560
01:26:19.408 --> 01:26:26.448
Uh, just the API should be different. The marketing should be different. None of the m- most heavily used open source or closed source models use diffusion.
561
01:26:26.968 --> 01:26:35.768
But isn't that like- [chuckles] Like doesn't that point to almost like a- It, it's, it's chicken and egg because what if you just give it more scale? [laughs] What's the, what's the largest diffusion LLM?
562
01:26:36.068 --> 01:26:44.088
I, uh, I don't think it's very big. [chuckles] Uh, I don't know the parameter count on this one, but diffusion Gemma is- Like under 20B. Diffusion... diffusion Gemma is not large. I think it's a 20 something. Yeah.
563
01:26:44.128 --> 01:26:52.148
Un- yeah. You know. Oh, it's- Like you haven't actually tried. You haven't given it a big one and you haven't- [laughs] So it's like very unfair... and Fusion Gemma is a 25B and it's old.
564
01:26:52.168 --> 01:27:01.328
And, and that's what I'm saying is, is like for its size, it, it does pretty well, um, in terms of, in terms of quality. It's almost like the same challenge with video models to have the same size.
565
01:27:01.368 --> 01:27:10.408
It's like you're comparing it to models that are much larger in scale. Yeah. Well, unless you do the whole thing where you, you have a text, uh, backbone and then you- Right, right...
566
01:27:10.608 --> 01:27:20.668
you know, you, you like glom some kind of decoder thing that, uh, does that. Like, you know, we- so we- we started off the podcast doing this for the, uh, for the inverse direction from image to text. Right.
567
01:27:21.108 --> 01:27:30.628
Uh, and I think like it's, it's roughly intuitive that you can do the opposite direction. I agree. [laughs] I see it. I see it. Yeah. I mean, the... we're, we're speculating on, on research in general. [laughs] Yeah.
568
01:27:30.728 --> 01:27:42.028
Um, one part that we can end off with this is, is the, the topic of your talk where, uh, inference engineering used to just be like, let's take an open model, make the GPU go brr- Mm-hmm... and then that's it.
569
01:27:42.048 --> 01:27:50.868
That's the job of, of Phase 10. Uh, now it looks like people are using inference more and more in post-training. Yes. Um, and training and inference. Yes.
570
01:27:51.088 --> 01:27:58.788
It's training for inference and inference for training both have become big topics. Well, inference for training in the sense that like obviously you just need...
571
01:27:58.988 --> 01:28:02.128
you need to do, you know, you need to do rollouts when you're doing like oral training ones.
572
01:28:02.168 --> 01:28:23.708
And so if, if your rollouts are taking a long time, if like, you know, you're using a vLLM for instance, or as opposed to CRTL LM or if, if the model that you're trying to train is not supported in CRTL LM and you have to fall back to an older inference engine, your rollouts are gonna be slow and you don't wanna do training on rollouts that are too off policy, so you have to wait for them, so you bottleneck your entire training pipeline.
573
01:28:24.308 --> 01:28:29.088
And so like obviously the techniques that we, we do inference optimizations for, um, will help them there.
574
01:28:29.828 --> 01:28:35.868
The training for inference mostly comes down to like just the spec deck training, Eagle-i training, um, and sometimes post-training.
575
01:28:35.928 --> 01:28:45.448
For instance, if you want to quantize a model, you'll quantize it down to like NVF before. How do you... S- like sometimes, sometimes you get lucky and you can just do PTQ and that works.
576
01:28:46.068 --> 01:28:50.188
Sometimes you quantize it down to NVF before and the model is, is terrible, like the quality is too bad.
577
01:28:50.788 --> 01:28:58.908
And you have to do post-training on the model in order to make it understand that it's going to now be an NVF before and let it still output the same logits.
578
01:28:59.548 --> 01:29:03.828
You can do this with normal SFT, PC, um, you know, quantization aware training, all of that stuff.
579
01:29:04.248 --> 01:29:18.208
But more and more so we're seeing techniques like NVIDIA released a quantization or a distillation paper where you, you establish a version of the model that's in NVF before and a version of the model that's in full precision, and then you'll do distillation training based on the logits of the two models in order to make the F before model understand.
580
01:29:18.868 --> 01:29:30.008
And so more and more of the team, the engineers, like of the inference engineers that work on our team, they have to be very familiar with like training techniques and, and just being fine writing training pipelines for it.
581
01:29:30.708 --> 01:29:36.788
Yeah, it just seems like, uh, they're meshing together in a sense. Models are coming together. Yeah, absolutely.
582
01:29:36.848 --> 01:29:45.787
I mean, if you think about the ultimate goal potentially of having a continuous improvement system- [laughs]... you know.
583
01:29:45.847 --> 01:29:54.448
Yeah, I mean, it's, uh, it's, it's kind of funny, but at the same time it's also kind of happening and, and I think within a, a few months to a couple years, like a lot of
584
01:29:55.548 --> 01:30:07.388
leading agent builders are, are going to have these loops like really up and running in production where you are doing inference, learning from the inference.
585
01:30:07.648 --> 01:30:15.728
We obviously for a long time have been sort of like learning from inference as it's live and dynamically adjusting the system.
586
01:30:16.048 --> 01:30:29.188
Uh, you know, any kind of dynamic adjustment is going to beat a static configuration across, you know, your, um, exact config, across your speculator, across that kind of thing. And then the...
587
01:30:29.388 --> 01:30:41.788
you know, you can take the, the traces that you're generating from your product, continuously post-train the model, roll those out, A/B test, um, get better signal, get better model, get better product.
588
01:30:42.348 --> 01:30:50.128
That, that loop is, is really promising. Um, the technologies and the infrastructure to build it are coming along quickly.
589
01:30:50.868 --> 01:31:03.668
And so the sort of unification between training and inference, I think is, is only going to accelerate. I actually w- was chuckling, but I wasn't-- I didn't think it was funny, like it's actually real.
590
01:31:04.228 --> 01:31:13.788
One of the big things for AIE World's Fair was that, uh, you know, we have, uh, RSI into AGI. It's the, it's the rough tagline. Uh, which like, yeah, I mean, we have, uh...
591
01:31:13.948 --> 01:31:23.308
I saw you pull a parameter call of like we have models training models, and, uh, the next step is obviously models training, uh, the... or optimizing their own inference, which is kind of funny.
592
01:31:23.508 --> 01:31:30.428
I wonder if, uh, models will be like on policy better at training themselves than training models that they are unfamiliar with.
593
01:31:30.888 --> 01:31:42.838
This-- these are all like very interesting open areas of research One big part of my job a couple years ago was for any arbitrary model that came out on Hugging Face, writing a config foot and kind of getting it up and running.
594
01:31:43.458 --> 01:31:50.338
And now the get it up and running config is, is one-shottable. Mm. Um, and so, you know, I don't have to do that anymore.
595
01:31:50.858 --> 01:32:00.838
Yeah, I mean, that's not exactly a model optimizing its own influence so much as a model, like being able to read the SGLang docs. But, uh, yeah, I mean- Well, we do see it. We do see it, like- Yeah...
596
01:32:00.958 --> 01:32:06.198
with GLM 5.2 for instance, GLM 5.2 is very, very good at writing GPU kernels. And so for like...
597
01:32:06.358 --> 01:32:37.857
It was very funny internally, we had a GLM 5.2 endpoint that we were using to, like that we plugged in in our cloud code harness, so every engineer on team uses like our GLM 5.2 and it will do a forward pass on the GLM 5.2 instance of the, you know, the node, and then it will get the profile trace and it will analyze it and it will find the kernels that are the bottlenecks in SGLang, and then it will write the new kernels and then we'll do another profiling trace and when it's done it uploads the image to our thing and then we can pull that image down and repeat the cycle.
598
01:32:38.398 --> 01:32:43.878
And so for quite a bit of time we had like literally GLM 5.2 optimizing itself- Writing and optimizing GLM 5.2... at GLM 5.2.
599
01:32:44.378 --> 01:32:53.478
And, and like some of the GPU kernels that run GLM 5.2 within our inference engine is written by GLM 5.2 and the trace and the kernels were guided by GLM 5.2 as the driver. So it seems like...
600
01:32:54.258 --> 01:33:04.278
I do see, I do see that, that circle being there. I think a bit more time is needed. There's definitely a lot of things that it can't do. The models just aren't there yet, even though they're like really, really smart.
601
01:33:04.938 --> 01:33:12.878
Like, they still try to like reward attack their way into like the, the cheapest or like they're very... Like, they're not good at like decision-making almost it seems. But yeah, I do...
602
01:33:13.978 --> 01:33:17.378
Like yeah, like a model optimizing its inference is, is already a thing that, that happens.
603
01:33:17.418 --> 01:33:26.418
[chuckles] Do you think GLM 5.2 was uniquely good at optimizing itself or did it just happen to be the best coding model that we had access to and it would- [chuckles]...
604
01:33:26.638 --> 01:33:36.638
do an equally good job of optimizing, uh- Would it-... a DeepSeek or a Qimi or something? Well, to Surks's point, maybe it's gonna be off policy when it tries to optimize another model. Would it secretly hurt DeepSeek?
605
01:33:36.638 --> 01:33:44.358
[laughing] To try to boost itself? [chuckles] Uh, no, for, for what it's worth, I don't believe that. Yeah. But it's just... Let's just find out. It's interesting, yeah.
606
01:33:44.678 --> 01:33:52.358
Just, just, just, you know, you have more compute than me. Just go- [chuckles] Just go try it... try it. Yeah. Any other upcoming trends in inference engineering that we didn't cover?
607
01:33:52.798 --> 01:34:00.018
Like right now, you know, you- 'cause you guys are so close to it- Yeah... you can obviously see it, that the world- rest of the world doesn't know about. The big ones are obvious.
608
01:34:00.358 --> 01:34:13.538
Models get bigger, hardware gets more powerful, users get used to a certain level of speed and demand a higher one. I think that some things I'm excited about are systems level.
609
01:34:13.818 --> 01:34:27.038
You know, we, we still have a lot to think about in terms of composing multiple models together. If you think about a voice agent, there's three to five models involved in that and the communication between those models.
610
01:34:27.598 --> 01:34:32.778
There's a lot of new modalities that are coming out. Uh, there's like the Cosmos, the new world model.
611
01:34:33.078 --> 01:34:41.078
Um, there's more research, uh, speech to speech is still like not entirely a thing, uh, but it's, it's getting, it's getting closer.
612
01:34:41.778 --> 01:34:47.478
Uh, there's gonna be just a lot of new modalities to build around, which is gonna be exciting.
613
01:34:48.418 --> 01:35:06.018
And then, yeah, I think that the other thing to solve, which is something we've been solving for a long time and, and are not done with yet, is, is just going to be continuing to operate at another 10X, another 10X, another 10X scale as an industry.
614
01:35:06.138 --> 01:35:20.738
If you think about the degree of usage that AI has worldwide compared to, you know, some of the more mature technologies both on, on consumer and business, it's, it's pretty clear that there could be multiple 10Xs more of demand.
615
01:35:21.638 --> 01:35:35.718
If you look at the infrastructure work industry-wide, obviously it's been stood up very, very quickly to meet a unprecedented spike in demand and that is like not stopping.
616
01:35:36.058 --> 01:35:46.738
Um, so yeah, there's just, just a lot of problems to solve around like long tail reliability and, uh, figuring out where we're gonna get the next like 10X and 100X of tokens from.
617
01:35:47.118 --> 01:35:57.178
I'm gonna say it's gonna be a really boring answer, but I think the answer is just faster next, like faster network chip communications. It seems to me that like more and more memory is the bottleneck.
618
01:35:57.198 --> 01:36:03.778
You wanna have larger models. Right now when you're doing serving at large, you have to transfer KV cache from one node to another.
619
01:36:04.678 --> 01:36:15.118
But the way that you do that is you tran- you find the KV cache, you find where it is, you transfer it to another node, you put it on that node's memory, and then you transfer it from that node's memory into the GPU and for like, like into the sensor cores of the GPU.
620
01:36:15.618 --> 01:36:24.738
So there's like a two-stage transfer here that makes it such that you're very bottlenecked with, with just KV cache transfers at large, which affects the time of decode and PD this side.
621
01:36:25.398 --> 01:36:34.618
You have to do this because the HBM is so mu- it's like extremely fast, like 4.5 terabytes per second as opposed to like, which is like magnitudes better than, than NIC communication speed.
622
01:36:35.318 --> 01:36:46.038
If you were to somehow be able to, in like, in like this theoretical dreamland, have extremely fast NICs, you could in theory spare that HBM and you could just transfer KV cache trans- like directly from one node to another.
623
01:36:46.558 --> 01:36:54.398
This would give you like almost 100X speed up when you're doing this aggregated serving between nodes and nodes. I'm not familiar with the technical challenges of making NICs faster.
624
01:36:55.018 --> 01:36:59.778
I'm certain there's a reason why they're like orders of magnitude smaller- [chuckles]... than like, like slower than, like HBM.
625
01:37:00.378 --> 01:37:10.218
But if someone were to figure that out, it would literally be like a mag- like two orders of magnitude faster to do decode. That would be my take. Be a good chip. Cool.
626
01:37:11.038 --> 01:37:20.218
Uh, I don't know if you have a nomination for things that are trends. Uh, I, I got one. Cool. Uh, so I think inference engineering for continual learning.
627
01:37:20.998 --> 01:37:40.914
Um, so what if you just, uh, like if you, if you just had the, the idea that you were supposed to learn from everything that you ever process Do you do anything differently or, or do you just have the same paradigm of like, well, stick it in a memory.md and then like it somehow gets consumed in KV cache and like this system works, it's not broken?
628
01:37:41.414 --> 01:37:53.334
Or like how do you like reshape inference so that it learns while you inference? Yeah. I think maybe one relevant topic there is your absolute best friend in the entire world's work on KV compaction. Correctly.
629
01:37:53.374 --> 01:38:04.174
Like what changes? What changes when- If you're trying to continual learn. There's two takes and there was like, um, like Charlie and I had this Twitter, uh, sort of, uh, argument where the...
630
01:38:04.454 --> 01:38:11.494
like continual learning could take one of two paths. It could either be that the model learns and so it's continuously pushing its new knowledge into its weights.
631
01:38:12.194 --> 01:38:21.194
In that case, you just need to have, like your inference just needs to continually fetch new weights or yeah, like you just literally need to do fetch new writes and reads of weights.
632
01:38:22.214 --> 01:38:31.914
Or the other path, which is you do KV cache compaction and if you do- And there's a LoRA layer if you just only update LoRAs. Yeah, exactly. Exactly. Which is, that's the engram approach- Yes... which we covered.
633
01:38:32.354 --> 01:38:40.294
The argument against doing weight pushing is that you can only fix one hop knowledge, as in, as in you can only- Yeah... feed it a new feature of like, oh, what is the best university in the world?
634
01:38:40.334 --> 01:38:45.834
The best university in, in the world is, is Waterloo. But then a second derivative [laughs] That's not changing. That's not changing. That's not changing.
635
01:38:46.194 --> 01:38:53.854
But like a second derivative question of, of which university should I hire an intern from? So if you know that the best university in the world is Waterloo, then the answer should be Waterloo.
636
01:38:54.074 --> 01:39:02.114
But if I wasn't just one-shotting the question and I was to ask it to like, use its knowledge to think and then give me a second answer, or like should I hire an intern from Waterloo or MIT?
637
01:39:02.234 --> 01:39:08.374
It'd be like, "Oh yeah, both are good." But no, like I liter- I just edited in your knowledge base that Waterloo is the best. Why didn't you use that to do reasoning?
638
01:39:08.714 --> 01:39:14.554
So that's the fundamental problem with trying to change a fact in an MLP within the weight. KV cache compaction fixes that.
639
01:39:14.954 --> 01:39:28.834
With, with KV cache, um, or like rather not KV cache compaction, but like if you're able to have something like, like the still paper which we, we came out with, which is you're able to sort of make your KV almost infinite and you're able to compact in such a way that you don't lose any of the knowledge.
640
01:39:29.534 --> 01:39:33.354
In that case you can actually do continual learning and you can actually solve continual learning.
641
01:39:33.894 --> 01:39:43.714
And this as a, it's, it's a result of, um, this, this argument that Charlie and I had that I do concede that his point was correct and I, I do see that KV cache is, is, is the way forward.
642
01:39:43.754 --> 01:39:49.234
And, and in that case, I don't think inference is going to change that much because we still use KV cache in inference.
643
01:39:49.294 --> 01:39:56.814
You're just gonna update the KV cache or it's gonna be like an additional step, but nothing changes in the weight, so nothing changes in inference time, nothing changes the spec that I had. Okay.
644
01:39:57.054 --> 01:40:07.034
Surprisingly great answer. Uh, we have it up on the blog. It's, it's a relatively recent blog, so, uh, we can- people can go see it. Mm-hmm. Uh- Hyper parameters. Yeah. Otherwise, this is super enjoyable chat.
645
01:40:07.094 --> 01:40:13.494
I know we've like already gone two hours. [laughs] Oh, wow. I didn't even realize. I know, like time flies. Yeah. Yeah. So much we didn't even cover. Yeah.
646
01:40:13.534 --> 01:40:20.174
This is like, we also wanted to talk about the book and all that, but you've covered the book. I mean, yeah, I mean, every- everyone, everyone knows about the book. [laughs] Yeah.
647
01:40:20.274 --> 01:40:30.494
Uh, high- highest ROI thing in the history of base 10, right? For the hour. Without a doubt. Without a doubt. Yeah. Absolutely. So congrats on that. I, I, I, you know, and we've covered that in, in our meetup- Yeah...
648
01:40:30.534 --> 01:40:38.014
which we can publish separately. Uh, but no, thank you to you guys for being so generous for sharing. I think it's a fun conversation that, um, we don't get to have enough.
649
01:40:38.094 --> 01:40:46.154
I think inference engineering, we never really covered head on and so to have you guys come on, uh, is, is a, is a treat. Oh, it was amazing. Yeah.
650
01:40:46.194 --> 01:40:54.734
Thanks- thanks for having us and, you know, hopefully in a year everything shifts and we can, uh, come back and say everything we were wrong about. [laughs] Yeah. Yeah, yeah.
651
01:40:54.794 --> 01:41:02.914
I'm excited for this Mega Kernels comment to get out and see what, see what people say. [laughs] We gotta stir stuff. [laughs] Should I go into hiding? I know I'm gonna get like the Mega Kernel community after me. Ah.
652
01:41:03.414 --> 01:41:14.234
[laughs] Yeah. One thing I really respect about you is you are not will- oh, you are not, uh, scared to kick the hornet's nest ever. [laughs] It's not, I, I don't think it's that, that controversial. I don't know.
653
01:41:14.754 --> 01:41:28.714
We'll see. [laughs] We'll see. We'll see. [laughs] That's right. All right. Thanks guys. Thanks. Thank you so much. [outro music]