TRANSCRIPT 🔗
WEBVTT
00:00:03.759 --> 00:00:11.199
Now we enter in the session three, foundations and tool chains, and time for Matthias, liquid.ai.
00:00:11.519 --> 00:00:13.679
Matthias, uh the floor is yours.
00:00:13.839 --> 00:00:14.560
Thank you.
00:00:15.119 --> 00:00:15.839
Hello everyone.
00:00:15.919 --> 00:00:17.519
Um, can you see my screen?
00:00:17.839 --> 00:00:18.320
Daniel.
00:00:19.440 --> 00:00:20.399
Yeah, I see.
00:00:20.800 --> 00:00:21.359
Yeah, sure.
00:00:21.600 --> 00:00:21.920
Perfect.
00:00:22.000 --> 00:00:22.239
Awesome.
00:00:22.399 --> 00:00:22.640
Yeah.
00:00:22.879 --> 00:00:23.199
Awesome.
00:00:23.280 --> 00:00:29.440
So my talk is about uh LFM2, which stands for Liquid Foundation Models, which is basically how we call our models.
00:00:29.600 --> 00:00:33.119
Um, designing the next generation of foundation model architectures for HI.
00:00:33.280 --> 00:00:36.880
I'm Matthias Leichner, I'm a CTO and co-founder of Liquid AI.
00:00:38.079 --> 00:01:00.399
So if you look at uh all of the device classes where we see like demand for AI, uh like from a smartphone, where you have like a personal assistant to like laptops, um, again with voice uh to factory robots with like a lot of vision and video data processing to on-prem um servers in a way that uh handle privacy, you know, sensitive data to cloud services in a way with larger models.
00:01:00.560 --> 00:01:06.159
Uh we see that the requirements on the compute devices and um models is quite diverse.
00:01:06.239 --> 00:01:16.400
You know, on the cloud, we have like um multi-node setups with like a massive amount of memory, um, but also we expect a high quality view um working with a with a cloud model.
00:01:16.560 --> 00:01:36.079
On the other hand, like on a smartphone and so on, we have like uh much more uh uh limitations on terms of memory and latency and memory bandwidth, as well as compute um uh compared to kind of large models on a laptop, it's uh quite also different because usually we have more memory available, but the memory bandwidth is often not that much more than a smartphone, for instance.
00:01:36.400 --> 00:02:02.319
So um what we actually need in a way from a foundation model perspective, in a way, with um inference-aware architectures in a way that are device um specific, for instance, like a uh architecture that's specifically designed for the low memory available on a smartphone versus like on a laptop, we would need uh an architecture that is kind of uh allowed to have more memory, but like is aware of this kind of uh low memory bandwidth uh compared to like an H100 GPU and so on.
00:02:02.640 --> 00:02:15.520
So if that's actually what we need in a way, but uh if you look, what we have currently is like the transformer, like all the large language models that we see out there, you know, Lama to Queen models, um, to GPDs and so on, based on this transform architectures.
00:02:15.680 --> 00:02:18.960
And it in terms of quality, it's um a very strong architecture.
00:02:19.039 --> 00:02:22.639
It's it basically fits any that type of sequential data that you feed into it.
00:02:22.719 --> 00:02:26.479
Uh, but it also comes with costs in terms of memory um usage in a way.
00:02:26.960 --> 00:02:52.000
So, what we did as a company is like to look into the design space of architectures in a way, and um, you know, we decompose and create like a way to represent it more uh efficiently, so including like the attention um um but also like all of the alternatives that come out from research, you know, like Mamba, XLSDM, you know, uh hyena convolutions, linear tension models, states models as other uh linearized um features and so on.
00:02:52.159 --> 00:03:13.439
So um what we did um for the company is to kind of uh parameterize this design space, you know, a capture, and then um use that one to do an evolutionary uh search space, uh evolutionary search on top of this uh design space to optimize um um specifically for each of these um uh device classes that I mentioned before.
00:03:14.479 --> 00:03:24.240
So to give you an example, like uh this evolutionary algorithm that we call star, how it works in a way, we have like an initial population in a way of an architecture in a way that feeds an input X to Y.
00:03:24.719 --> 00:03:54.159
And what we do is actually we do an inf uh hardware in the loop um um uh evaluation um to kind of get the number of uh like this the size uh that it takes up on the on the device in a way, because um with caching is quite different, you know, like the actual just the like the amount of parameters that the model has is often just the indicator of roughly the how much memory it consumes, but exactly with caching and and a lot of disactivation memory and so on, there's a lot of dynamic components as well, you know, uh KV cache, um activation memory and so on.
00:03:54.400 --> 00:04:04.800
As well as like the uh we profile how much um uh how fast uh this this architecture actually runs the code and pre-fill operations on the device.
00:04:05.039 --> 00:04:14.560
So in our case, we used a Galaxy S24, so a Qualcomm and P Qualcomm CPU chip um to measure how fast this device runs.
00:04:14.639 --> 00:04:24.480
So we have like the parameter size, we have the catch, like how much memory it actually uses, as well as like the CPU um latency on a on a Qualcomm uh edge uh device in a way.
00:04:24.560 --> 00:04:43.040
Uh and then we um basically proven in a way unpromising candidates um and the promising ones in a way we also train them on like our um um reference data set in a way and measure like how well actually does this architecture fit the data because like just having a fast architecture doesn't mean that we can actually preserve the quality.
00:04:43.120 --> 00:04:50.560
So we have like both in terms of memory and latency some requirements, but also on quality, like hey, we don't want to sacrifice performance, go back to a transformer.
00:04:50.959 --> 00:05:03.920
So what
Load full transcript
we did is then basically prune the unsuccessful candidates like that don't fulfill the speed or perplexity or other requirements in a way, and then use that as with an uh evolutionary algorithm to recombine mutate architectures in a way.
00:05:04.000 --> 00:05:09.839
So then we improve kind of on the on the frontie of like error and uh versus latency.
00:05:10.399 --> 00:05:25.839
Um here's an example where we have like the first generation of models, and we we pick one that has the like the one of the lowest losses, um, and um use that one in a way to generate the second generation of model architecture candidates.
00:05:25.920 --> 00:05:35.839
Um, again, optimize here for loss um in move further along um in the design space of like latency on the y-axis and loss on the uh x-axis.
00:05:39.360 --> 00:05:54.959
So this is nice in theory, but also like um what we achieved here um to give an example um using a quantized version on like a Galaxy S24, um, compared to a Lama, compared to a phi model from Microsoft, and also the coin model from uh Alibaba.
00:05:55.120 --> 00:06:00.319
We show that um uh the plots on the bottom, you see the throughput, and the plots on top is the latency.
00:06:00.399 --> 00:06:05.600
So and on top is like lower is better, and on the bottom plots, um higher is uh better.
00:06:05.759 --> 00:06:21.680
And we see that in the orange that LFN2 in a way outperforms model, um even like larger models such as the Lama 3.2, which have more parameters, but they're actually slower on a Galaxy um phone in a way, specifically the D code is is a is a is a big uh improvement.
00:06:21.920 --> 00:06:30.319
And this architecture, although it was optimized for a Galaxy S24, um, the performance improvements also translates to other you know edge uh devices.
00:06:30.399 --> 00:06:54.560
So, for instance, here we have a laptop with an AMD Ryzen um um PC and uh also on CPU with a quantized model with uh four bits and eight-bit activations, and we see like very similar because the device classes are very uh much more related compared to like a cloud GPU in a way, and therefore, for instance, the low memory bandwidth um uh that was optimized for the most that we see on the phone is also present on the on the on the on the laptop.
00:06:54.720 --> 00:07:04.399
Therefore, we see also like um significant advantages um uh compared to um uh yeah, uh standard transformer-based models in a way.
00:07:04.720 --> 00:07:12.800
So um, as I mentioned before, uh, these are all like just speed and and and and throughput for pre-fill operations, so prompt processing and decode for the token generation process.
00:07:13.040 --> 00:07:17.199
Um, but as I mentioned before, like this is this is nice in a way to have a fast architecture.
00:07:17.279 --> 00:07:21.600
But what's also important is to have uh to maintain the quality of a transformer.
00:07:21.680 --> 00:07:29.360
So we know that transformer has is a very uh proven battle-tested architecture, so it
fits any kind of data um um to it.
00:07:29.600 --> 00:07:42.480
And um in this uh table here we show um that we trained actually three different uh sizes of this LFM2 architecture, like from 350 million parameters, 700 million parameters, and 1.2 billion parameters.
00:07:42.639 --> 00:08:20.079
And we then evaluated a lot of this standard benchmarks, including MMLU, um, the multilingual version of MMLU, MMMLU on the bottom, also uh grade school math, um, instruction for women and other capabilities, and what it shows really that it outperforms like significantly, for instance, a Lama 2, a Lama 3, sorry, uh 1B architecture in a um despite having much uh smaller size of the models, you know, and not just size of the model but also improved uh inference uh latency, but also the quality is really maintained and also like um as uh competitive to kind of transformer models out there.
00:08:22.160 --> 00:08:33.279
Um so one thing that I should mention is that this architecture um um yeah, it's as I mentioned for custom is not based on as it is it has some components from Transformer um that are derived from the architecture.
00:08:34.639 --> 00:08:36.879
Um but it's a it's an open weight model.
00:08:36.960 --> 00:08:38.159
So we released this model.
00:08:38.240 --> 00:08:53.919
Um if you're interested in working on AI applications and want to you know get access to uh language models um that are like optimized for like um uh speed on the edge, um please check it out.
00:08:54.000 --> 00:08:55.440
You know, it's all it's open weight.
00:08:55.679 --> 00:08:59.279
Um you can fine-tune it um and that app and see how it goes to your task.
00:08:59.440 --> 00:09:05.519
We also have like integration with like uh you know Lama CPP and other inference um frameworks um for edge.
00:09:05.679 --> 00:09:08.159
Um also an NPU compatibility.
00:09:08.320 --> 00:09:13.360
So um specifically like with core comm chips and uh AMD hardware is coming soon.
00:09:13.519 --> 00:09:17.759
Um and some with some third-party integrations already there basically implemented.
00:09:18.240 --> 00:09:21.759
So um this is a language model, so text in, text out.
00:09:21.840 --> 00:09:28.799
Um, but we use this backbone, this efficient uh edge uh backbone um to train a family of models, not just um that one.
00:09:28.960 --> 00:09:43.039
So um on the left, uh we see that we not just have the 350m, the 700m, the 1.2b, but we also train the 2.6 billion parameter model and a mixture of expert architecture that has 8 billion total parameters and 1 billion, 1.5 billion active.
00:09:43.440 --> 00:09:53.759
So that is more or less targeted to actually like um devices with more memory, but still like low memory um bandwidth, as such as a laptop and and the AOD devices that I mentioned before.
00:09:54.000 --> 00:09:58.720
And we're not limited to text only, but we're also trained you know, vision models on top of it.
00:09:58.879 --> 00:10:13.360
So basically, what we do is we added addition encoder to the language model backbone to kind of uh generate this what we call LFM2 VL series, vision language series, um that's ideally like for um you know video processing and and other tasks.
00:10:13.519 --> 00:10:22.960
So this is actually uh being used in production with some of our customers that have um basically like video um edge video processing with this um uh vision language models.
00:10:23.039 --> 00:10:29.519
Um again, like the 350M with a vision encoder uh comes to around 450 million parameters.
00:10:29.600 --> 00:10:36.399
So this is on the bottom up to the 3 billion parameter vision language model that's based on a 2.6p language model backbone.
00:10:36.720 --> 00:10:39.600
And not just um vision, but also audio.
00:10:39.679 --> 00:10:45.360
Um so we have a prototype that's still like a lot of improvements going to be happening in the next couple of weeks to months.
00:10:45.440 --> 00:10:49.039
Um 1.2 billion parameter audio model is again optimized for edge.
00:10:49.200 --> 00:11:00.000
So this is a bit more competitionally intensive with like the audio decoding, and this basically allows a native, so you don't need to transcribe the audio to text and then uh to a text-to-speech model afterwards.
00:11:00.080 --> 00:11:02.080
This model can do anything, uh everything it wants.
00:11:02.159 --> 00:11:06.879
So you're just feeding the audio into the model and it generates tokens uh and speech out.
00:11:07.039 --> 00:11:13.360
Um, but as I mentioned before, uh we expect that there's a lot of updates coming in the next couple of uh months on that audio model.
00:11:14.799 --> 00:11:18.720
So this is um in a way what we what we did, in a way, what we uh have done so far.
00:11:18.799 --> 00:11:51.759
But like in the future, we are you know um doubling down on like optimizing for other platforms in a way, different, like even my cloud that we're looking at, even though this edge is a lot of uh most of the demand at the moment, like on edge, but also there's uh big in uh demand for cloud um that the benefit from edge models because um the low, you know, that the volume in a way, if you have like an application with like a large uh user base and so on, you actually want to significantly reduce the cost offloading uh to the edge, but also like uh run the edge model on the cloud in a way, um in a in a very large batch review.
00:11:52.720 --> 00:11:54.720
And also for modernity-specific aspects.
00:11:54.799 --> 00:12:14.399
So, what I mentioned before, like the improvements on the audio is like one of the things where we double down again um on like really optimizing on the edge um the capabilities that we can provide in a way already, like basically just the same quality that we that we see in cloud models at the moment, we want to really bring to uh embedded devices.
00:12:14.559 --> 00:12:16.000
Um and uh yeah.
00:12:17.200 --> 00:12:25.519
With with the LFM2 series, we already did a first step in this direction, but
like there's a lot more um that we um going to release over the next couple of months.
00:12:27.039 --> 00:12:29.519
Um yes, this concludes my uh talk.
00:12:29.679 --> 00:12:37.600
So please, if you have any questions or follow-up, uh yeah, reach out to me or or yeah, I don't know how much time we still have for the for the session.
00:12:37.759 --> 00:12:39.840
Um, but yeah, uh thanks so much.
00:12:40.480 --> 00:12:43.039
Uh first of all, uh Matthias, uh, thanks so much.
00:12:43.279 --> 00:12:48.240
I think we have uh still uh five minutes uh for uh QA.
00:12:48.480 --> 00:13:01.440
And uh really your presentation is impressive, and I've been following uh liquid.ai for uh for a while, and I have to say that you are doing great performances and uh and developments.
00:13:02.000 --> 00:13:06.639
Let's check if uh there are uh questions uh for you.
00:13:07.039 --> 00:13:19.200
How long does it take and what is the cost, if you can discuss this, of finding a network architecture suited to specific hardware using your genetic search approach?
00:13:19.440 --> 00:13:20.720
That's a good question.
00:13:20.960 --> 00:13:23.120
Yeah, that's a very good question, actually.
00:13:23.200 --> 00:13:29.440
Yeah, um so so what we do is um um we studied extensively the scaling loss.
00:13:29.519 --> 00:13:35.840
So, in a way, like how does um um with increasing parameter size, in a way the performance of the model improves.
00:13:35.919 --> 00:13:40.399
So there's this um scaling loss that applies mostly to transformer but other architectures as well.
00:13:40.559 --> 00:13:54.720
Um basically like of fitting a uh training a couple of models at different parameter sizes and then basically fitting a regression plot in there to really see like okay, like if we if we train a hundred billion parameter model, how what quality should we expect from that in a way?
00:13:54.879 --> 00:14:06.639
Um and we did we did this extensively because of the reason um the question that you that you asked in a way, um that um training like let's say uh a three billion parameter model is actually quite expensive.
00:14:06.720 --> 00:14:12.320
So we we cannot use uh training of a three billion parameter for actually doing the architecture search.
00:14:12.480 --> 00:14:39.759
So what we do instead is we train models like roughly like hundreds of millions of parameters, so much smaller, um but not just one size in a way, but really like multiple sizes, but still on a very small scale, like as I mentioned for hundreds of million parameters, and then in a way fit um a regression plot to see how well this would fit into like a 3 billion, uh 14 billion and other um parameter regime in a way, and then we we extrapolate from that.
00:14:39.840 --> 00:14:57.759
So basically what we do is like we optimize basically for smaller size, um and then once every um in the uh evolutionary process, once every I don't know, 10 iterations or 10 um generations, we then test it on a bit of a larger run to make sure that they don't overfit in a way to that very small setup to make sure that it's transfers.
00:14:58.240 --> 00:15:12.080
Um but yeah, that's that's how we you know we try to keep the cost reasonable to train models, and yeah, I think yeah, the amount of actual cost of GPU hours, I think it really depends on on how much we want to optimize, you know.
00:15:12.399 --> 00:15:16.879
Yeah, that's uh that's great.
00:15:17.039 --> 00:15:19.279
Uh um uh Matthias.
00:15:19.919 --> 00:15:23.679
Um just a question from me instead.
00:15:24.159 --> 00:15:46.960
Um, you know, I was uh for a while interested on uh uh conversational human-machine interaction audio basis, and usually a a naive approach, well naive, an approach is uh to use uh uh you know speech to text and then LM model and then uh and then text to speech.
00:15:47.039 --> 00:15:53.679
While I understood uh one of your foundational models, especially on audio, does all the three functioning once.
00:15:53.919 --> 00:16:05.840
Can you comment a bit uh and uh uh and also I I wonder if you already bring in in one of these models you will bring uh reasoning out?
00:16:07.759 --> 00:16:09.120
Yes, that's a good question.
00:16:09.200 --> 00:16:19.919
That's actually like yeah, the the the audio model, like we did a lot of research, so that's why it's like um the least mature at the moment, like unfortunately, but like as I mentioned, we'll make a lot of improvements.
00:16:20.000 --> 00:16:22.159
We're currently doing a lot of improvements on the architecture set.
00:16:22.720 --> 00:16:33.360
Um yeah, we basically uh very similar to the vision um models in a way where we just use like a C clip or like a vision transformer in a way to encode in a way the images into like a tokens.
00:16:33.519 --> 00:16:36.480
The for audio is in the input space, it's very similar in a way.
00:16:36.559 --> 00:16:49.360
Like just instead of like a vision transformer, it uses, I think, some some FFT-based uh models in a way that we convert the the speech into like some latent representation that is basically similar to tokens in a way.
00:16:49.600 --> 00:16:55.679
On the output, uh we try like a lot of different approaches, uh, most of it decreases quality a lot.
00:16:55.759 --> 00:17:00.960
Um, so what we found to be working is um interleaving audio with text.
00:17:01.039 --> 00:17:09.599
So what we do is we predict an audio, we first of all we predict the text, uh, a couple of text tokens, and then we predict the corresponding audio to the text tokens.
00:17:09.759 --> 00:17:17.200
And the reason we do it and because we figured out uh we tested if you only predict the audio tokens, um the quality degrades a lot.
00:17:17.359 --> 00:17:24.319
So you need to really like them, you know, in a way, the model needs to think in token space because the most of the pre-training was done in like text token space.
00:17:24.480 --> 00:17:31.759
So it also needs, you know, the the kind of the text tokens are always kind of to ground the audio to to steer the audio in a way.
00:17:31.920 --> 00:17:41.440
And so basically what we are doing now is like um one of the improvements that we're now doing is to do parallel decoding, to not just doing like sequentially like interlative, but really like parallel um audio text in a way.
00:17:41.519 --> 00:17:49.119
But as I mentioned before, the the text uh one is very important, so you don't just predict the audio but concurrently, like in parallel audio and the text.
00:17:49.759 --> 00:17:53.440
Um that's uh that's uh good uh Matthias.
00:17:53.599 --> 00:17:56.799
Uh obviously, I would have uh other other questions.
00:17:57.039 --> 00:18:11.599
I hope really you consider as liquid.ai to be uh to attend to the other events or be part active part of the AGI foundation community because I think uh uh I perceive uh that community as your uh your own.
00:18:11.680 --> 00:18:15.519
So please uh uh consider to be active with us.
00:18:15.599 --> 00:18:16.960
Uh you are doing great.
00:18:17.200 --> 00:18:18.240
Congratulations.