ML Engineer: "You've been lied to about open source"

Nahshon Cook-Nelson: You can't blindly swap models. The main point, you need to understand your load. You need to understand like how much users you handle, how much of a complexity it is. If you have a problem that you want to have like a revenue for casting, you need to fully train it on your data. So there is no one who's gonna do it for you. It's very specific, it's very niche to you. In general, I think that hybrid routing is a real future. Not even a future, but that's what we have right now. That's the principle with which we have been building products already for years. A GPU of 800 level is twenty five to forty thousand dollars. API dependency for LLMs is not mandatory, as a lot of people tend to think. Like before, if you're running like a self hosted solution, you inherit all of the stuff which is currently done by Welcome back to the Return on Intelligence podcast, where we're all about helping you implement AI by bringing you real experts who are practically implementing solutions right now, not just talking about it. Here today, we have a very special guest, Vitalius. ⁓ he's an ML engineer. And I want to start Vitalius by asking people who just clicked on this episode, what are they going to get walking away from this? So, in this episode in general, we're gonna cover the Emerging trend of self hosting L models. Basically, what are pros, what are cons? Whether this is something useful for your business and like what it takes to get there. I love that. Now, why should they listen to you? ⁓ that's a good question. I mean, in general, I've been doing data science for more than seven years. I have PhD in natural language processing. Same as anyone in our machine learning team. We all come from this classical data science background. So back in the time, third party API vendors were not even like the case for them. ML models. So from day one we've been building models like layer by layer, then you know, hosting them and providing the inference level. That's what we have done. I personally deployed models like self hosted models for tens of thousands of businesses which already had millions of parameters. And that was like years and years before like the GPT release. In general, right now, even with all the new words ⁓ popping up in the news, this is something which we have heard like ten years ten years ago. So this is not something new to us. Principles are still remain the same. That's awesome. Because I think a lot of people can get a little bit of whiplash. From everything that's happening, go to Claude, go to chat. Now it's saying go to your own, go to your own self-hosted. So kind of give us an idea of what's happening in the landscape right now and what it means to be quote unquote self-hosted. So in general, I would say that API dependency for LLMs is not mandatory as a lot of people tend to think like before. In general, you can go two ways by the principle. So whether you go with a third party provider, so this is a serverless approach, you know, as with Amazon. with you know with open AI with whoever you choose to and you don't care about you know what's going on under the hood you just send the request you get the response back so everything is like a black box to you however it ties you to the provider we're gonna talk about you know what other cons are to that. But that's basically it. You can really spin up LLM in you know in a minute which is Which is a very cool thing to have. Another approach is self-hosting. So basically, you have a model ⁓ on premises on some of your servers, ⁓ where it is stored and like you manage everything. So it's not leaving ⁓ any data is not leaving your server. The model is spinning in there, so you send the data to your own server, it returns you back ⁓ the responses. That's basically a self-hosting approach. So you have ⁓ open weight models which are shared, or you know, in case you have enough budget, ⁓ you train your own models and you own everything. That's basically the approach. Before open AI chat GPT ⁓ got really popular, you said this was like the alternative that people chose who had enough money to pretty much build their own models. Were there foundational models before as well that they would start with? I wouldn't even say that that was like an alternative. I would say that was mostly the all like mostly like the only use case in machine learning. We look at two different ⁓ conceptually models. Like what was like the whole change with LLMs, like with the models which we had before, and why everything before was self-hosted? So Initially, models before you had to train them like on your data specifically. So if you have a problem that you want to have like a revenue for casting, you need to fully train it on your data. So there is no one who's gonna do it for you because it's it's very, you know, it's it's very specific, it's very niche to you. So basically, before we always had, you know, that problem of a data because if your use case doesn't have enough data, you can't train model on top of it. However, that has changed tremendously when we got introduced with LLMs, with natural language, because previously you had to teach models from the scratch, which was very expensive. expensive. Imagine training from the scratch the model to understand English language. Like it's you know it takes insane amount of computational it all has changed with ⁓ like two thousand end of twenty eighteen, like with Transformers introduced like the first model was birth ⁓ done by Google. That was that's what was used in the Google search. What has changed there that models were already pre-trained by some vendors and they were already good enough for the language. So for example, the BERT model, it was trained on full ⁓ English Wikipedia. So you already could get the model which kind of understood the English level and you could already use it from the scratch. That wasn't the case before. And that's how like the whole principle has shifted the third party API approach to MLs, because what was with Bert, Bert was open sourced. However, like GPT principles, they're all the same. So you have a pre-trained model, which was pre-trained by OpenAI, by Anthropic, which costs like, you know, hundreds of millions of dollars. And out of the box, it's already good enough for like a lot of use cases. Then you can tweak it a little bit. And that's what created all that, you know, all that transition. So for previously it wasn't it wasn't the case. You were building models, you know, you were building models from the scratch and training from the scratch, basically. So now I let's say deep seek. I can grab this open source model, Deep Seek Flash V4 or the latest thing, and I can get it. And now I'm pretty much starting from a world-renowned model that then I begin to tweak. So I'm almost going back to that process from before. But instead of starting from scratch, I'm starting from like an incredible model. Is that the way to see it? Absolutely. And the the Deep Seek is one of the exam like one of the good examples people got. I mean, it was one of the big news model distillation, or like so that's one of the techniques which we had like in data science, you know, tens of years before. Basically you train The first like foundational model, which is, you know, very expensive. You know, for example, we take GPT 5.5. Training it from the scratch is super expensive. However, if you already have trained this model, you can take a smaller model, which is gonna be learning not from the data, but actually from the responses from this data. So we are mimicking that initial foundation model. And training that, this is called model distillation. Doing that is a fraction of a cost, but you're getting, you know, it depends on your use case, but you're getting most like of the accuracy in there. That's why that was like one of the big things for like Anthropic to prevent actually other models like learning from each other, because it's very easy, you know, to do it when you have like a really good foundational model. That was I don't want to drop name, ⁓ you know, was this exactly the use case with the deep sea, but for a lot of models that was the use case. They are literally distilled versions of these like big foundational models. So now that benefits the companies because Now I have the option of saying, okay, I don't want to pay anthropic anymore. I can download it. Now, should we go self-hosted? Part of what you told me when we started preparing for this episode is it sounds cool to say, Hey, you'll never have to pay for tokens again. You can just use your own self-hosted model. But Actually, there's a little bit more to the story than what sounds. ⁓ absolutely. Because there is a lot what you add on the top. Because like right now, all the you know self-hosted ⁓ self-hosted hype on LLMs, we like most of them are only mentioning the the token price. You know, open eye costs like this much, you know, on like one, you know, it costs, you know, that much. But this is just like a small tiny fraction of what actually is happening, you know, under the hood like in a production. Because if you're running like a self-hosted solution, you inherit all of the stuff which is currently done by third party providers right now. So this is serving, scaling, load balancing, auto-scaling, failover, monitoring, batching, caching Quantization, GPU utilization, rollbacks, model upgrades, you know, guardrails like on top of it. So this is everything which you inherit right now. So I think it it's a bit similar to trend, which was popular beforehand. It's it's again a bit emerging of should I use cloud service providers or should I, you know, have servers on premises? Because this is a very similar, you know, this is a very similar like a comparison. So because yes, you know, it's it's cool to have. all of your data like in house, but it comes like with all the provisioning with all these like support costs. You can't look, you know, just you know at one bill and tell like this is better, like or this is better. And the same the same absolutely one with LLM. So like if you are talking just you know about token costs, it doesn't show you the picture. Cause as long as you have like a local LLM, you know, running somewhere just for yourself. If it's like a small model, you know, you can you can run like some small models, you know, on your device or like on your on your computer. Everything is fine with there as long as you know as this is a toy. But if you start Talking about this on production level, where you actually have customers where you actually start to scale. Here's the whole fun where, like, where it starts to happen. Just like an example, OpenAI has released I think two models, if I recall it correctly, like two open weight models, like 120 billion parameters, and like another one another one. As an example, that model, which is of course, I mean it it's not as good as GPT 5.5, but Anyways, for a lot of use cases, it's good enough. In order to run this model, just from computational perspective, you need a GPU of H hundred level. A GPU of H hundred level is something around like 30 okay, 25 to 40,000 dollars. So just to run this model alone, you need like a GPU which costs, you know, already 30 five forty thousand dollars. That's a lot of initial like investments. And even okay, you set it up, you know, theoretically, you put it down, it started working. What if you have like five users running like at the same time? What do you do now? You buy another GPU? Because like this is everything which you need to provision yourself, or you need to pay, you know, ⁓ Amazon or you need to rent something from them, but then again, you're introducing another, you know, another dependency because like you rely on the availability of these, you know, GPUs running on like on on someone else's server. So, yes, if you look at you know at the small model which you can run on Mac, you know, you can show very fancy demos, but if you look how Actually, does it scale? This is a very different ⁓ kind of a story. And as well, why initial comparison of tokens is not really relevant to show the full picture. Token from token differs from the model. So, for example, if I don't want to not to name drop just to show like the same principle, if one token costs you know X amount ⁓ for one model and Y amount to another model, it doesn't mean that your product is gonna c cost you know cheaper on one or another because A lot of these benchmarks have been proven that you know, yes, this costs fifty percent cheaper per token. However, for complicated use cases, this model uses three times more tokens. It's more verbose. So your to your per token price was eaten by that. So, like, first of all, it's already not that good in terms of a cost. And now imagine like if your open weight models, I think it was like with Quan, ⁓ I don't remember which exact version, it was like very verbose. So here you not only pay in money, but actually you pay in a lot of that server provisioning. Because imagine now per user you're generating four times more ⁓ more tokens. That hits your architecture decisions. You know, for like for your server, and now you really feel it because you know you're not open AI, you're really the one now responsible for all the auto-scaling, all the stuff which third-party providers do on a scale to reduce the cost, you know, as prompt caching, for example. If you not going too much into details, but like if you have like repeated parts of a of a prompt, you can reduce like up to 90% in terms of a cost, because that's what's something you know they have caching, which you have to build it yourself. If you're you know, if you if if that's something that you want to do, you know, they have thousands of people who do it. If you want to self-host. You know, you're gonna be responsible for that, or like your team. Like if you have a big enough team to do that kind of a work, great, but it really needs to make sense ⁓ from your product perspective. Yeah, I really like what you called out earlier before we started recording, and it you talked about this hybrid approach because I think people tend to see most things in life as binary, right? You either all in on this thing, you're fully self-hosted, or you are all trapped in clawed harness, but it seems like it's a lot more of a spectrum. And people can start to say this thing is super confidential and valuable and we want to be sovereign about that. But that thing over there, we're gonna publish on the site anyway. We don't need to. Talk to me about what that hybrid approach could look like for companies. Yes. So in general, I think that hybrid shrouting is a real future. Not even a future, but that's what we have right now. That's the principle with which we have been building products already for years. So in some of the previous podcasts, we've covered it about being agnostic to the provider. I like you know, cloud the Opus Five, you know, just put it like everywhere, it does the job. In general, future systems that travel between models intelligently. That's you know, that's the approach for the for the products. One model won't be best at everything. First of all, it's not gonna make sense in terms of accuracy, in terms of latency, in terms of in terms of a cost. So in general, that's why one of the important things when you build any kind of LLM solution is evaluating it. That's you know, that's what we do. In general, when you evaluate systems, you do it. I mean, at least we do it in in three different, you know, in three different ways. The first one is natural, like human in a loop. You can't, you know, you you always need like how how does the human experience it? That includes users, that includes QA testers, like feedback loop, like all of that. Then you have metrics of performance, like you know, queries per second, latency, cost, so all of these kind of metrics. And then you have models which are grading your responses. So like LLM as a judge or different kinds of models, so like how fluent the answer is, how accurate the answer is. And then you have like, you know, you have very clear understanding of how every module of your product is behaving. And having that as a baseline, you can actually start experimenting with, okay, maybe I can use here like a smaller model. Or maybe here, like I have a lot of costs. I don't care here about latency. That's the use case for the self-hosting. Because like I do batch once in a month. I don't care whether this runs for four hours or eight hours, but I spend like a tons of money on it. In general, you can't blindly swap models. But the main point, you need to understand your load. You need to understand like how much users you handle, how much of a complexity it is, have a baseline, and from there you can experiment with the models. And that's what we've been doing with different providers. And that's what's being done, probably should have been done as well with balancing with the self-hosting. Because in general, the risk is if you're gonna start with a self-hosting solution for the production, you're gonna spend very big amounts of time just setting up every like just setting up instead of actually validating whether your product brings value to your company or to your users. Because you know, with a quick demo, you know, open like open AI, whoever you just, you know, you put it, it does the job. Cool. Now when you go to production, you're like, okay, you know, this is not, you know, it's like tokens are eating my margin. That's when you start, you know, counting whether you know whether you want Unless of course you have a use case where you just you just can't share any kind of a data because you know it's like proprietary, it's medical. However, of course, you know, some of the providers, ⁓ most of the providers right now they have like zero retention policy, which you can you you can you can, you know, models don't retrain like on the data that you provide. There are like enterprise accounts where they you know they really make sure that that's not happening. But if for your specific use case you can't you don't want to share any kind of a models, then yes, ⁓ like self hosting is ⁓ can be a necessity. I like the fact that we look at the company becoming AI native holistically, because then when you talk about model selection, some departments are gonna require a different level of ⁓ data retention or ⁓ security or sovereignness compared to other departments. And then even within those departments, certain functions Are gonna be more valuable, something we're willing to spend higher token costs on, or like you said, it's a lower frequency that requires less, we're okay with a higher latency. I just appreciate that because then when someone I think about ⁓ we call her responsible Rachel, who's listening to this, who's responsible for rolling out AI to the entire company, she doesn't just think Claude Team is gonna solve the whole thing. Maybe we have a Claude Team account, but if she really wants to be model agnostic. She's hearing what you're saying. I think the last part I wanted to cover is let's say someone says, okay, I have a function that I see self-hosted could solve. I know that it's complicated. I know it's gonna cost more. Vitalius, you didn't scare me away. I still think I want to go ahead and do it. Who do they need? To make that happen. First of all, you're gonna need a machine learning engineer. ⁓ naturally you need like a person who's gonna select the model which is appropriate for your use case. And that should be combined together with DvOps or like MLL MLOps person who's gonna support all the infrastructure like under it. In general, it's not gonna be just like two people. Usually this is, you know, a load for like for entire department because you need to not to make sure that it works, but as well, like it's it's a bunch of monitoring, you know, to prevent like over you over you, like all of that. You need to upgrade models in case if you want to do fine tuning, it's it's it's gonna be more complicated than, you know, just calling an API with a third party provider. Like it's gonna, you you need to retrain, you need to update the waste, you know, so on, so on. So it's gonna be quite a big involvement and ⁓ it's gonna be on duty. It's not like you're gonna set it up, you know, in week or two and then you're gonna forget it about it for years. So you're gonna need TVOps, you're gonna need ML people who's gonna who're gonna support it across like all the time on the project. And we I know we love Sentry as like a monitoring tool. Other ⁓ Lang fuse, I don't even know what that's for. We're putting together a case study on that right now. What other like software or stack things they got the GPUs They've got the ML ops, the ML engineers, DevOps, et cetera. What stack do they need? ⁓ I think it very strongly depends on like where you self-host. So like whether you provision models on servers from AWS or whether you provision this on your own server. So this is again how much in-house do you want to go? For example, if you self-host a model Azure servers, but for example, you use like monitoring from Data doc and you send like all of your data to another like third party provider, then what's the point? You know, like then you're just sharing, you know, your your data again, but just like with a different party. So, like, really, how deep do you want to go there? It very strongly depends. Langfuse use case is more for your LLM product, it's not about the server infrastructure itself. So it's like it just shows you like what's the average latency, you know, of my endpoint, you know, and so like so on and so on. The cool thing about Langfuse, even so, this is a third party provider, which you can, you know, bait, use their API. They are ⁓ they are open source, so you can launch their entire stack on your servers. But that's gonna cost, you know. ⁓ so some of the users, because like we had a one very great, you know, ⁓ use case in the healthcare medical domain. Their CTO was, you know, quite defensible about you know what kind of data goes where. So initially they were like, Yeah, we really want Lank Fuse, but we want to self-host it. We're like, Cool, you know, here is like here's our DevOps, here's our ML, like We're gonna do it. So it required like just monitoring itself required, I think, seven, seven or eight different resources, you know, like different kinds of memories, diff different kinds of cash being spun up on their, you know, on their servers. And then the monthly bill was quite high, you know, and supporting all of it was, you know, quite and Then they just just went to the Langfuse, they they found like the subscription type which fitted them better, where it had like zero data retention. And then actually they switched back to the third party provider. So because they didn't want that, you know, they that maintenance burden of self-hosting created to them. Because you know, when you use API, it looks all you know simple. You don't see like what's happening under the hook. In Langfuse, because like they had multiple versions, but if you wanted to use like at that time, it was the latest version of Langfuse, like the third generation, where you have like a bit more functionality just to have a dashboard, like seven or eight different servers for you, like. load balancing, like, you know, monitoring all of that just for a single dashboard, you know. Or you just go and, you know, you pay like fifty bucks a month and you you know, you get all of it. So it's I can't tell you what's you know what's better like exactly for your business, but this is something to keep in mind, whether you really have such a load where it makes sense, you know, or or or such a use case where it makes sense for you to have to go self hosted because I think we we haven't covered it, but as well in some of the use cases, self hosting is only solution as well from the connection point of view. If you need like a hundred percent reliability, if you always need something to be available for you because everyone has a downtime. So imagine you need like a computer for like on the Like on the plane, not in all the spaces you know you're gonna have like a Wi Fi connection. I had one use case, it was with electrical vehicles, so it was like a forecasting, but as well like some LLMs for when your car is being charged. And like the problem which we encountered with the first party solutions that in a lot of ⁓ garages there is zero connection. So like we we don't know like what's going on like in your car. And in this use case, like sorry, like you need something you you need something running like on device. So like that's you know, that's a use case where and that's you know, that's a different domain, like I IoT in general, like in like even in your iPhone, you have like these models which are which are there, you know, to summarize your email because you're you're not connected to the internet like in a lot of time. For these like niche tasks, you can have some small models which are already like running on your like on your edge edge device. That's awesome. We also have an edge resource. A mobile edge resource that I believe Koba put together. We'll link that up in the description. Also, on our next newsletter, we're actually going to be sharing an evaluation suite. And we always share resources like this: like how to put together your own evaluation suite, how we recommend putting together your ROI for AI, because that's super important to what Vitalis is saying to understand where you can get the most results. Also, I think what we'll end up putting together from this, Vitalius, is I've noticed that you can use open source models that are lower cost, but they're hosted. In your country, like they have US hosted cloud providers. I think it's like Fireworks AI is hosting some of these open source models on cloud. And then you can use them, get a lower token cost, but still get the API feel inside of your model. So we'll also put out resources on that so you can decide, like Vitalis is saying, how self-hosted do you want to go? If you just want that cheaper token costs, maybe there's a better way to get there than spinning up your own ⁓ servers and and getting GPUs. Vitalius, is there anything we haven't covered that we should I think just to sum up that you know local inference is not a feature. It's you know, it's a responsibility. So don't you know think about infrastructure first of all. I mean it's it's a boring part, but it's like ninety-five percent importance of this. Yeah, think about like whether your product really needs this. Because okay, we can do it, but that's, you know, in a lot of use cases we just tell you like, yeah, it's just doesn't make sense for you. So like really, you know, you need to have like all of these metrics and that's again bringing up to the point where you really need to evaluate. You really need to know your product. Because like if you're starting up, ⁓ you c you can't blindly make these kind of a choices because it's gonna like it's it's gonna bit you like really, really a lot. Thanks Vitalius. This was a fire episode. Now I can confidently walk away from open weights and know that maybe it's just not the right time or the right thing and I'll happily pay my API cost. Yeah, I think again, not to be like a devil's advocate. So like one cons of being like fully dependent on a third party solution is first of all you have a token pricing trap. Right now like many companies behave like current API pricing is per like is permanent, which is not. So right now a lot of these use cases because in general if you would look at difference between the accuracy between LMs like even two years ago. I think it was like Stanford AI report. So like in one year, the difference between Top one and top ten model right now is like less than five percent. So basically it's kind of state of the art, but like for you for most of the use cases, it really doesn't matter that much, you know, that like whether you use like OpenAI or Claw, you know, like they are both great. They both have like a lot of great models, and they are currently fighting for the market. That's why you keep like on a weekly basis, you see like people, you know, companies like reducing the cost. They change, you know, token prices, model accesses, you know. You constantly see this right now, like within Tropic, like yeah. We have like a weekly limit, then we just remove for everyone, you know, just randomly weekly limits. Yeah, so as long as it's working towards you, cool, you know, because like we don't have a monopoly, you know, at some point that should end because a lot of you know, a lot of companies they are right now burning their money on provisioning DC LMs like to you, so there is a big chance of. that you know tokens are gonna be more expensive because they need you know they need to earn money so like from from from from somewhere. Because like the fact that right now you pay, you know, one million tokens to open AI, it doesn't mean that for open AI it actually costs one dollar to, you know, to provide you with a million with a million tokens. That's the first thing. Second thing Just training that model alone cost it for the company, you know, like hundreds of millions of dollars. Where does that money, you know, at some point they need to know, they need to earn that money back. Whether this is gonna be as, you know, as it was introduced in some of the solutions, ⁓ whether this is ads, you know, whether this is, you know, some extra latency, some policy, this is not something that you can control. As well, same with the models. The fact that Prompt worked on this model, it can behave very differently on the SMU model. If you are reliant on a provider, you don't have any power over deprecation of models because they just decided, yeah, this is model is not, it doesn't make sense in terms of return of investment to the company anymore. We're just gonna drop support to it. Because still, you can see the deprecation, like it's it's available in internet, you know, for open AI. So, like when does the sunset for model? And that's it, you know, your whole business is Depending on just prompts working on this exact model, in half a year it's gonna get down. You're gonna need to, you know, you're gonna need to update it. If you want to update it, you need to make sure that it's actually working, not worse, not worse as it was working before. Because in a lot of use cases, the fact that, you know, like Opus jump from four point seven to four point eight, it like it doesn't mean that that model is better. It's just, you know, it's just a release, whatever, you know, whatever that means. So maybe for your specific niche, it can actually be worse because in general, in all these benchmarks, you know. like funny ways like trust me benchmark like trust me ⁓ trust me bro benchmark it doesn't make sense for your business like doesn't it like it doesn't matter it it really matters like for your specific use case it can indeed you know outp outperform the previous model by how many percent you want on that specific like math equation or something but it actually can be quite worth you know for for your business use case and then you're in trouble because that's not something that you accounted before. That is the risk. That is the risk. This is the more you know maybe everything's gonna be good. Maybe prices don't go up but there is a risk for that. There is a risk for that because right now we have a lot of these players competing for each other because the accuracy is very similar. Like previously it wasn't, you know, like there was a very big gap, you know, be between the quality from open AI and Gemini, you know. Right now, like it's it's good enough. Like for for a lot of use cases, if your data is good, like it's it's like it it's good enough, you know. So that you know, jump from one version to another, it's not creating such a breakthrough as it was doing, you know, previously. It's not anymore like going from GPT two to GPT three point five. That was big, you know. That was big. Right now, going, you know, from Opposed 4.8 to 4.9, you know, it doesn't matter for most of the use cases, first of all. And it wouldn't justify to the user, you know, twice increasing the pricing. Right now it's like a very volatile, very, very, very volatile market. If you're not comfortable with that, you can go self-hosted, and then you're gonna have a lot of you know upfront cost, but then like everything depends on you. So like you you know all of it. And again, if for your use case, you know. This this matters, you know, you can safeguard yourself already. If you're not even sure that you need this kind of a product or your product is shaping a lot, probably you don't want to spend, you know, your first months ⁓ just setting up, you know, everything. Yes. Vatalius, this is great. Thank you. Thank you.