Transkrip penuh untuk The Small Language Model (SLM) Show, Episode 1: All About Model Merging

all the papers that came out in the space, and I found a whole bunch of interesting literature on, emerging language models and I wanted to use those. But, as is often the case with, research papers, there's not often, like stellar production quality code released. 00;05;57;05 - 00;06;14;23 Unknown Often there's a Jupyter notebook or something released, and it might it might work, but probably it's designed to be used on a, you know, a cluster of eight, 100 or something or whatever. The researchers who wrote that paper originally, did their experiments on, you know, they're not going to put a whole bunch of resources to make a generalized everything. 00;06;14;23 - 00;06;34;06 Unknown They just they just want to make this technique work. But, you know, I wanted to use this, and I just had a sort of my, my GPU pool resources available. So I had to write my own implementations. So I did and I, I threw them up on GitHub. People started using them and you know, like it's kind of snowballed into the state of things we have today. 00;06;34;08 - 00;06;51;01 Unknown That's great. Yeah. You know, obviously we we love what you created. you know, we thank you for creating it. So, that's that's fantastic. Crowd library. It's a fantastic toolset. So over to you, Betsy. My guess I'll say the same thing. You know, what is model merging with you and then also secondary left because you've built a ton of great things on. 00;06;51;04 - 00;07;11;13 Unknown yeah. I'd love to hear about, you know, how you discovered merge Get and what that was all about. Yeah. I consider modern merging as, a fourth step in the, in the functioning of the training, pipeline you have, like, the training you have the supervised fine tuning, you have the preference alignments. And now we have model merging. 00;07;11;13 - 00;07;31;21 Unknown And we like the idea of, yeah, we have this rich ecosystem of models that have been trained in fine tuned and they're quite good. And we have like that have been trained different data sets. So they can be like quite complementary to. And now we can merge them to create better models. So I've started getting like curious about model merging class. 00;07;31;21 - 00;07;51;19 Unknown Yeah I think it was in some like there was some scripts, I don't know, people were exchanging scripts and I was like, this is this is not possible. Like, this cannot work. This, this is too stupid to work. Right. And then I got really interested in it, and I discovered it, and I thought, wow, that's super convenient because, it's not just a script anymore. 00;07;51;19 - 00;08;09;28 Unknown Like, it's quite reliable. You have this Yaml file. so I decided to give it a try because it was so convenient that. No, I didn't have an excuse, not to do it. and, at the beginning, I was still skeptical. I was still very skeptical. Was like, no, this cannot work. Like, I'm just hacking the of the board. 00;08;10;01 - 00;08;36;24 Unknown And I started writing an article about it. And during

like, the time I was writing the article, I became more and more convinced that actually, you know, if it's not just hacking, they didn't know, like the model was actually better because I started using it. And, and I found it really, really interesting. back then in December 2023, it was very easy to take, like the best models, the open reports you would merge them using, like, any technique, and it would create the better model. 00;08;36;26 - 00;08;51;27 Unknown Now it's really difficult to to do it because, it's really popular now. people match models every day. so yeah, like, now the the competition is quite tough. 00;08;51;29 - 00;09;09;11 Unknown So I was on mute there. Yeah, absolutely. yeah. I mean, I'm, I'm, I'm words. You know, you said people merge models. Every AI merging applied models per day. Still, I just go in, do some merging, have a little fun, try to beat this kind of like a game, right. And try to try to beat some models on the leaderboard and see what happens. 00;09;09;11 - 00;09;31;29 Unknown So, you know, through merge kid, you know, Charles has made it absolutely simple to do that, which is fantastic. and then you as well, Mike, you built some great tools on that. Know, I use, you know, your, your, auto email, notebook quite often to do the benchmarking. the open side of it was a bit broken, but I think a fix that I should give you a PR on that. 00;09;31;29 - 00;09;57;25 Unknown So, that that's that's. Yeah. but. Yeah. So, yeah, it's it's just a fantastic, you know, way to, to utilize the best of both worlds when it comes to both models. Right? So, I guess I'll say, you know, I'd love to hear what you both to as kind of the next not I'll say now in, in the future, but what is the biggest, you know, revolution when it comes to model merging and, and that's a kind of a two part. 00;09;57;25 - 00;10;27;16 Unknown Yes. Like what? What is it done? What does it revolutionize now? And what do you think it's going to revolutionize going forward? Charles, you can go first. Sure. so one of the things I like most about model merging and, and one of the things that, you know, initially excited to be most when I, initially open source merge kit was that, you know, the, the fact that it doesn't require any GPU resources or, you know, expensive servers you can have on a personal laptop if you want to, that is releasing that to the community. 00;10;27;19 - 00;10;48;29 Unknown It enabled this sort of huge scale, decentralized global experimentation that that I mean, you could sort of see it in the scores over time in the open leaderboard just by giving these tools to so many people and letting them all plug away with it and see what they, they come up with, we we end up doing more trials, more experiments than any one researcher could ever hope to do. 00;10;49;01 - 00;11;06;06 Unknown So that's, that that's one of the things that, you know, initially excited me most about it. What I'm most looking forward to is, you know, right now one of

my main research thrusts is, is trying to extend these merging techniques to, being able to merge, merge models that want fine tuned from a common base. 00;11;06;07 - 00;11;29;11 Unknown So, you know, when you merge models right now, you you'll say you'll pick like a couple of Lama based models. So they're all, you know, they were all trained from Lama 7 billion parameter base. and then, you know, you can merge those and that works great. if you tried to merge, you know, models that were trained from different bases, like a decoder model and, and, a model, for example, you just can't do that right now. 00;11;29;14 - 00;11;56;25 Unknown but so I'm investigating techniques to enable that. And should that become possible and suddenly we, we know we get this, this access to this whole find the potential again, you know, so if we can get some sort of synergistic effects between different foundation models, then that's that's hugely exciting. I have a question. I don't know if you have an answer to Charles, but to what degree are we still like in the very early stages of the potential of what can be done with model merging? 00;11;56;27 - 00;12;17;08 Unknown That's a good question. I think it's it's barely been touched, honestly. so, you know, there are a whole bunch of great model mergers that have been done in the community and, and, you know, those are fantastic. I think there are a lot of more, you know, I see that a lot, a lot of concrete applications that could benefit hugely from merge. 00;12;17;08 - 00;12;38;05 Unknown Get that from model merging, that, that haven't really been fully utilized yet. RC is actually a great example of that, the way that we're using it to speed up and reduce the costs involved in continual pre-training, there's huge value there that the market as a whole is yet to really tap into this. Yeah. It's incredible. 00;12;38;07 - 00;13;00;08 Unknown Maxine, tell us some some about what you've been doing with, with merge Kit, maybe the most fun you've had with it. yeah. I've done a few projects with it. started with, like, make it, merge. it I don't even remember the names, to be honest with you. macro 14, something like that. 00;13;00;10 - 00;13;20;05 Unknown it was part of, like, a really big merge of 14 different models. I thought it was insane. So I wanted to, to to use this one, plus another one, because I also needed a good base. And, and this is, quite funny because this is a model that has been reused over and over and over again as a little tool to plot the family tree of this. 00;13;20;07 - 00;13;46;08 Unknown merges. So you can see that the, the parents, the this was models and, it now it's quite wild because there's been so many merges, so many generations that you can see, I can trace my, latest model back to the first one. but it's it's a long way home now. because, there's been so, so many, people like, reusing them, in, in some strange ways. 00;13;46;09 - 00;14;11;16 Unknown Like, very strange, actually, because now I also have cycles in my family tree. So this is not something that you see usually in real life, but, that happens with attendance, I guess. so that's been a lot of fun. this was the new Diego model, that has been very popular. so it's it's based on, on the much, I think slap merge. 00;14;11;18 - 00;14;37;23 Unknown And, this has been GPU fine tuned, to make the model a little better after the merging. And this is the same recipe that I tried to also monarch. but this time I changed the new data sets to add more, more data that is a bit more, about multi turn conversation. And that allows the model to become better at this kind of stuff. 00;14;37;25 - 00;14;58;29 Unknown It's also funny to see that you can have like this really great base model after you match. And then you can kind of tweak it the way you want to using different datasets. and if you find it the way you want it at the latest, project I have related to, to two matching models, it's called auto merger. 00;14;59;02 - 00;15;31;05 Unknown And, this is my, my attempt at automating my own job, because, it's so difficult to merge models now. like, if you can really see, like, the diminishing returns, so, I leave it to, I, to, to do it for me and I, we'll see, wins. but no, like, the main, product of this auto merger project is, is mestizo to merge automatically a lot of models, and get insights into, okay, like, how am I supposed to design a good merge? 00;15;31;05 - 00;16;01;09 Unknown Like, which models should I choose to to do so? So right now, I'm still, waiting, to get more results. but, it's quite funny. I already have a model that is better than R5 monarch. without any deeper fine tuning. maybe it's overfitting the benchmarks. Actually, it's probably overfitting the benchmarks, but still, it's it's probably a good sign that says that this it's they are diminishing returns with this. 00;16;01;09 - 00;16;23;25 Unknown Yeah. Above potential. to to still improve this, 7 billion parameter model and make even stronger models. That's terrific. And we'll post in the comments after where people can find your work. Obviously some of the, hugging face leaderboard, but also is a towards data science. You publish and yeah, yeah, yeah. Then also on LinkedIn. but we can bring it a few comments here. 00;16;23;25 - 00;16;43;14 Unknown James Ralston says you guys are rock stars. I'm gonna include myself in that. he wants to know if this call is recorded. yes. You can find it on our seeds LinkedIn page, also on our YouTube. afterwards, Ignacio Ferreira has a question saying it would be good to have more insights into how the models are merged, which he hasn't been able to find. 00;16;43;21 - 00;17;17;23 Unknown easily. One wants to take that over. We'll give that to Charles. Charles, let's let's see. Let's go deep. So so that's a big question. I mean, you could talk for, for weeks about all of these techniques. so the foundation of most of these is the principle of linear mode

connectivity, which is is basically something we discovered that if you take a, a base model and then you fine tune it on different data sets for different tasks, they end up in, you know, they end up close enough together in parameter space that you can actually just linearly interpolate between those final weights. 00;17;17;23 - 00;17;42;08 Unknown And every point along that line is a valid model and give you, you know, some decent performance on, on, on all the tasks involved. So most of these methods are basically just, taking that principle and running with it. you know, the most basic is linear merging. You're actually just averaging the parameters slurp, which is currently, you know, it's a favorite of people right now. 00;17;42;10 - 00;18;12;05 Unknown It's the same thing, except it's interpolating spherically instead of linear linearly. and basically it's just preserving the magnitude of, of vectors. In addition to interpolating the direction, which works a little bit better with in the interpretation of some of these parameters. then there are a bunch of, cool papers that, you know, to take this and they apply different techniques to the parameters to reduce interference when merging multiple models. 00;18;12;07 - 00;18;33;22 Unknown actually, the, the origin of merge get was as an implementation of the paper. resolving interference when merging. I forget the name, but the method is called ties merging. And then basically it's, it's specifying it before you merge. And, and the original release of merge get was actually just that, a script that did that technique. 00;18;33;24 - 00;18;53;20 Unknown I, I'm, I'm quite fond of that line of, of of methods. so, I'm going to answer that for Ignacio. send in your questions, post tomorrow, LinkedIn page. And, we will get Charles back in future shows. and, you know, good to go more into depth. So if you start playing around with it and have specific questions and merge kit is remarkably easy to get started with. 00;18;53;22 - 00;19;13;29 Unknown let's bring in a question. We have about two questions from Shelby Nayak. she says great conversation on model merging and leveraging small model, something we want to do a good fit as well. That's not a question, but here's a question she says, I'm curious, what is the additional cost associated on top of fine tuning a base model? 00;19;14;01 - 00;19;37;03 Unknown Okay, I think you're I think she's probably referring to the additional cost of merging on top of the fine tune. Well, merging very efficient. It's not acceptable. So, very, very low cost on the merge perspective, Charles, you've probably never done anything like pricing, kind of costing usage on, on, mergers, I assume, because it's a little I actually have looked at it a little bit. 00;19;37;04 - 00;19;56;05 Unknown I just, you know, when I was initially writing this, entirely in my spare time, like nights and weekends, I was, I occasionally used run it for, for bigger merges. So I, you know, I

looked at the cost of it and you can round it to zero. the cost is 2 or 3 minutes of a CPU virtual instance if you spin up a new instance for it. 00;19;56;05 - 00;20;17;16 Unknown So you could maybe pay a couple cents if you were really spending about it. Actually a dozen that's like that. Oh, we got another question. We got another question for the Ignacio. Yeah, you read it, Mark, you're the co-host. You read it. Right. So basically we train a model for Neagle and one for math. If I use some of these methods, like the one that works for both. 00;20;17;18 - 00;20;40;25 Unknown That is exactly it. Ignacio, you're combining two models, into one, and you're using the information out of both models into one model. but, you know, the Charleston I mentioned before, it's not, you know, the model merging experiments. All right. It's an extremely you know, it's a it's a method that is extremely strong. I think it's going to revolutionize our labs, to be honest. 00;20;40;25 - 00;21;01;02 Unknown Right. And if it hasn't already, it's going to really transform the way people think of their lives. But, pairing it with continual pre-training is actually extremely powerful, right? Because you have the ability to, you know, you'd mention. So if I train a model on a trade amount. so right now, presently, like the most common way to do that is you would train two full models and then you would merge them. 00;21;01;02 - 00;21;17;14 Unknown So say you, you know, use Mistral as a base or llamas based on central. and then, you know, you train one model for math and then you train a legal model for this for all you merge, right? Seven. So you still have to do the train though, right? You still have to train over the two 7 billion parameter models. 00;21;17;17 - 00;21;37;28 Unknown so what we're, you know, striving towards is comparing, you know, kind of what Charles had mentioned earlier is, you know, merging different size models and merging different architectures. So what this does is it opens up the ability to train a much smaller bottle, right, say a 1 billion parameter model and then merge it with a 7 billion, the 13 billion, you know, the 30 billion. 00;21;38;01 - 00;21;56;22 Unknown so the training itself becomes much more efficient and cost at that, the model. And then you still reap the rewards and the massive benefits because you're, you know, using transfer learning and you're getting the, the great benefits of the already trained model that exists with great general capabilities, general reasoning, or, you know, whatever that may be, and you're combining it into one. 00;21;56;24 - 00;22;04;15 Unknown So hopefully that makes up a quick follow up on that from Ignacio. 00;22;04;18 - 00;22;22;19 Unknown What could be some of the use cases compared to fine tuning one model. Well, so by I mean, Maxine, I'll let you take I'll say one thing and then you can take it. I'll let Maxine know this. But, there's a real problem when you're fighting models, right? It's called catastrophic. Forget it. Right. Models. Forget what they used to know. 00;22;22;21 - 00;22;39;10 Unknown when you go over them with new data. so catastrophic forgetting is a real thing. So if you fine tune a model, if you do like the fine tuning of a model, it's going to forget some of its general capabilities and its general reasoning in the model is probably going to be accurate. But go ahead, Maxie, I'll let you let your to go. 00;22;39;12 - 00;23;19;09 Unknown yeah, I think that's, model matching. it's not just about matching to models. Now we have like, an ecosystem of merges and it's built, iteratively. So, each generation is better than the previous one. I would say that it provides really good base models. I've had a lot of feedback from people using the models I've made, which are merges, and they find you need to do something else, in like, different languages to, and I think this is the main strength is that it gives you excellent base models. 00;23;19;09 - 00;23;40;18 Unknown Then you can fine tune them if you want to, and you're going to get like, even better models, for your use case. but I think this is really the, the, the main point behind model merging beyond just, I have a math model, and I have like another model. I merged them, and I retain, both capabilities. 00;23;40;20 - 00;24;09;12 Unknown It's also my model is better in general, and that's very, very valuable. I think the reason behind it to, to give some intuition is that it makes the weights less redundant. it's better at compressing the information inside of the weights. And this is what you get by iteratively, merging the the parameters of these models. If you guys are okay with this, we're just going to keep going with viewer questions. 00;24;09;12 - 00;24;32;06 Unknown I suspect we get a lot of good questions. so let's let's do it and I'll read this one. And then whoever wants to answer this is from Sequoia. And, what would be the most effective technique when it comes to knowledge composition? That's mixture of experts, right? Emily? model merging or block expansion? I'll let you take that job. 00;24;32;09 - 00;24;54;20 Unknown Sure. So you know that that. That's a hard question to answer. model merging is great at it. block expansion is also great. I love block expansion. That that's a really cool technique, and it's one of my favorite things to play with lately as well. mixture of experts. it might be a good way to do this, but there hasn't been, 00;24;54;22 - 00;25;32;20 Unknown It hasn't really, been been demonstrated yet that it can be effective in this, is, you know, an actual, you know, classically trained, a mixture of experts. the experts do end up being, like, subject experts so much they end up specializing on very, very, you know, abstracts hard for human to parse things like parts of parts of speech or or even that, you know, that, that that's, that's more specific than an actual, trained mixture experts would specialize on, though there is a

type of, of merge that you can do, where you combine, two dense, two or more dense models into a mixture of experts. 00;25;32;20 - 00;25;56;15 Unknown And that works. but it actually it doesn't work quite as well as just straight up merging, the models into one. Yeah, that's one thing that I mentioned to Charles is like, merging models is big. That's that's something that's that's really exciting to me personally. I think that, that's something we, we definitely I want our roadmap to, to dig into work from a research side. 00;25;56;17 - 00;26;19;28 Unknown so definitely excited. Did you want to add something on that, Maxine or. No. Yeah, I wanted to say that. Yeah. So so my experience that, mixture of experts, really exciting, works really well. Like, better than, what I would have expected. But if you really want performance, like, if you want to be efficient, like model merging, it's very difficult to to beat. 00;26;20;00 - 00;26;43;25 Unknown Terrific. Another question from James Robinson. If you read this one, Mark, what role do you believe there is or is not for leveraging model merging and the eventual signature development of AGI? I made it. That's the trillion dollar question. And the answer to the question is the question assumes that we achieve AGI. Well, but let's just say let's say we do. 00;26;43;26 - 00;27;06;09 Unknown I'm not 100% sold that we do, but let's say we do. what do you think makes it? yeah. Well, I'm not very comfortable with this question. I would say that this stuff hugging face said that, auto merge was probably the way of getting to AGI. So you hear about the engine? Exactly. I don't know, I think he has opinions. 00;27;06;09 - 00;27;36;25 Unknown I don't I don't really. You have an opinion on that, Joe? Yeah. I mean. If it happens, it will probably involved in some small part just because it's such a convenient way to pack more more capability, more intelligence and into the same, you know, size of parameters. I can't really guess beyond that. I mean, whether or not the transformer architecture just by itself is enough to reach AGI, it's I don't think anybody knows that. 00;27;36;25 - 00;27;57;22 Unknown And frankly, it's even at this point it's hard to tell where the line for AGI is. Something that I find pretty funny is how like five years ago, the the Turing test was the gold standard. And then sometime in the last three years, we took the Turing test out behind the shed. But it doesn't cut it anymore. We don't have, you know, it's just, I don't know what what I see it kind of thing. 00;27;57;23 - 00;28;20;07 Unknown Yeah, exactly. Yeah. James, stay tuned to Future Show as well. We'll get back to you with a definitive answer on that. question. Yeah. From you genius. Giovanni, how can I be? And sorry that the question is blocking off our main experts there. we'll get your heads back in in a minute, guys. How can I be sure that when performing a merge, there's no overfitting on the benchmark tests? 00;28;20;10 - 00;28;41;17 Unknown I have been following Maxime for several months using merge. Get daily. Yay! Five models per day. But while I'm convinced that the models are improving, part of me also suspects that they might be overfitting. On hugging face, there is a thread about data contamination that discusses this. Yeah, go ahead Maxime, because I know you brought this up already. 00;28;41;19 - 00;29;12;29 Unknown yeah. This is this is a really good question. there is definitely data contamination, and, the benchmark is the most contaminated in that the misuse list. also, unrelated, but is also not, not, not very good. Benchmark is truthful QA. And, it's it's really been bothering me for a while. so, for a moment, I had this project of, like, rebuilding this much, pyramid, from scratch, using on these, like, models that were not contaminated. 00;29;13;02 - 00;29;34;08 Unknown But that was the quest alone in this efforts. And I believe it was really a lot of, a lot of work. So in the end, I kind of, gave up on that. I think that's the best way to convince yourself that, it's you're not just overfitting. One benchmark is to use different benchmarks. 00;29;34;11 - 00;30;15;20 Unknown so you have, of course, the open assembly, the boards. You have the new suite, which is pretty good at which is very correlated to it. So it's it's pretty nice. and then you have other benchmarks like empty bench. so multitasking conversation, you also have things like, IQ bench, and a lot of other benchmarks that you could use and that provides like, predict which of this view of the performance of the model in different areas and, I think that's the best way we can tackle this issue, because, yeah, if you create five merges a day, it's probably gonna overfit it. 00;30;15;22 - 00;30;36;20 Unknown And that's fine. Honestly, that's, that's that's also okay in about the data contamination issue. I think it's fine in the end. I, I mean, I'm at peace with it now. I think that indeed it's contaminated, but also the models are better. So in the end, it's what we want. Like, we we want to create better models first. 00;30;36;23 - 00;30;46;03 Unknown so, I think it's, it's okay as long as you acknowledge it. And you, you know, about this issue. 00;30;46;05 - 00;31;01;28 Unknown Let's bring in another question. But I will mention that Mark may have to drop off. Charles and Maxine are good to go for a few more minutes to answer questions. And, Mark, if you need to go, just give us a wave and, we'll see you in two weeks. All right. This is from Nabi Sharma. 00;31;02;01 - 00;31;32;22 Unknown great talk, and thanks for your work on merge kit and auto merge. The question is, in a scenario where you are using model merging to update the learning lens, are there techniques to ensure that catastrophic forgetting does not happen through ever wants to take that your natural? Sure. so it's not necessarily a way to, make catastrophic not forget catastrophic forgetting not happened. 00;31;32;22 -

00;31;54;11 Unknown It it's more, model merging offers an avenue to, undo the damage of catastrophic forgetting. So, you know, you can take one model, fine tune it with your huge new data set, and you get something that ends up really, really good on the task. You turned it on at. It's forgotten everything else. but if you merge that back into your base model, and there are their settings you can play with to make this work better. 00;31;54;18 - 00;32;33;27 Unknown to you, you end up getting the capabilities of both, you know, so you retain still the base model's ability to do the original task that was trained on. And then you also get, you know, most of the performance of your, downstream fine tuned model. Yeah. I'll saying that like, as an example, you know, when I, when I do merging, like perhaps perhaps adapter based models, you know, just a simple merge, with back with a base variation, like, it increases the, the performance know above average of, you know, 8 to 10%, of the model, across else, benchmarks based on, you know, increases it 00;32;33;27 - 00;32;57;09 Unknown from their original test model up to 10% just by re, you know, having that merge where, you know, back with the base model, as Charles says, right. It's it's not, you know, kind of it's it's allowing it to remove the barrier of catastrophic forgetting because it, you know, fuzes with the base model again. And it records that information that it had lost when you did the two. 00;32;57;12 - 00;33;17;20 Unknown Let's bring this question in before you drop off real quick. Mark, this is a question from, Vikram Runabout. Big question. But briefly, what are the limitations of SLM? yeah. I mean, obviously being a little biased because I'm all about it, but I mean, of course I that's the limbs are smaller export models right there. They don't know as much as large models. 00;33;17;20 - 00;33;36;28 Unknown Right? I mean, so if you are hoping you can get a 7 billion parameter model to function in the way that Collider or GPT four does, it's not going to happen. Right? but, you know, that's why I say when I, when I prep in saying like SLM or better, I'd say that by saying or 99% of business use cases. 00;33;36;28 - 00;33;56;05 Unknown Right. And and that's the reason, the reason behind that is because if you have, you want to utilize a language model for air, right? Do you care that it, you know, can you recite poetry or, you know, rap music or, you know, tell you who won, you know, Best Picture in the Oscars in 95? You can't. Right? 00;33;56;11 - 00;34;24;12 Unknown You want a natural, grounded use case, and task associated model. Right. So, is there limitations set? You know, there's limitations on the general capabilities in general reasoning of the smaller model in comparison to the large model. But I'll say that's what we're doing here with model merging. Right. Like SLM paired with model merging. When you pair it with model merging, you get the general reasoning capabilities back from the from the model that you do the merge with. 00;34;24;14 - 00;34;48;21 Unknown so all things that we're trying to, you know, kind of get past, in the kind of limitations that exist today with just smaller language models. Right. yeah. Hopefully that it that's actually a terrific place to wrap. And what I'm going to tell, to wrap the show about is, what I'm going to tell the people who still have questions, Ignacio Acuna and some others, we'll take those questions and build them into our our next show coming up in two weeks. 00;34;48;23 - 00;35;05;12 Unknown hopefully we'll have Charles back. Maxine, you're welcome anytime to join. Mark will always be here with me. And we'll get into, into some of those that are more in the weeds about, the specifics on model merging and merge kits and then, as always, you know, today was very much on model merging. We'll keep talking about that. 00;35;05;12 - 00;35;24;29 Unknown And then the bigger, issues, challenges and benefits of slams, not smiles. As I wrote. So, just fantastic. Thank you everybody for watching. We'll get it up on YouTube soon. We'll get the audio version out on podcast. and the recording will stay on our LinkedIn page. so keep chiming in there with the questions you all want answered next time. 00;35;25;05 - 00;35;51;25 Unknown Any final words from, Maxine Charles, Mark. Maxine. Well, we'll start with one. yeah. Keep merging. It's great. It's it's really easy to do. And if you don't have experience with it, it's like, you can do it in, in a few minutes. and then evaluated very easily. So, really, it's, it's a concrete, and I encourage everyone, to, to get this experience. 00;35;51;28 - 00;36;12;03 Unknown Hashtag keep marching. Love it. I don't Charles, if you want to say anything. And that was that was that I mean, what else what else is there to say afterwards? Stay tuned. Stay tuned. Because. Oh yeah, that was. Yeah. Charles or team and RC, we are doing some fantastic things in the world of model merging. things that are really groundbreaking. 00;36;12;03 - 00;36;32;22 Unknown You know, we're we're, we're, we're, we're ready to pop when it comes to the really great things in the emerging space. So stay tuned. We'll be releasing some great things in the coming weeks and months. So, we're really excited. I hope with everyone else inside. What is it? All right. Thanks, everybody, for joining. Take care and we'll see you, in two weeks for episode two of the Small Language Model show. 00;36;32;23 - 00;36;38;07 Unknown Take care. Bye.