Fuld transskription af The 2026 GPU Shortage Crisis

00:05:41.519 Inferencing is very hot, lots of inferencing is happening to improve uh customer experience. 00:05:41.600 --> 00:05:51.040 Let's say, you know, you go into a support service, or you're you're calling support instead of a human person, you have uh a bot or an LLM that you're engaging with. 00:05:51.120 --> 00:05:59.439 That's sort of like the inferencing that we're typically seeing, or the fine-tuning, which you know increase is is really focused on your specific data sets. 00:05:59.680 --> 00:06:13.759 That's where we're talking about larger clusters that are typically uh dedicated to specific to your specific workloads, whether you're using video content, images, or it's text, or maybe it's audio, depending on your your business. 00:06:13.920 --> 00:06:20.399 Um, so what you need to do uh today, or what we're seeing customers look at is the business model as a whole. 00:06:20.639 --> 00:06:36.000 Do we can we make the case to own the equipment ourselves and make a long-term commitment where we believe that we will find more use cases to justify the purchase or the leasing of that GPU equipment? 00:06:36.240 --> 00:06:51.040 Or do we go forward with a rental uh where you know we want to still like have the flexibility to test and run proof of concepts and you know validate our pipelines that we're testing, but at a lower commit. 00:06:51.199 --> 00:07:00.079 So that's first like trying to figure out what is your long-term strategy and how much of a commit are you willing to take as a whole, as a company. 00:07:00.240 --> 00:07:08.079 Because if you're an AI startup, you want to move really fast, but you may not have the budget to make this long-term commitment. 00:07:08.240 --> 00:07:11.360 And that's harder and harder now to get closed. 00:07:11.439 --> 00:07:18.000 Meaning, if you really want like a three-month contract today for H100, or that's no longer possible. 00:07:18.079 --> 00:07:18.959 It's really hard. 00:07:19.120 --> 00:07:23.439 So finding the right the right neo-cloud partner is super important. 00:07:23.600 --> 00:07:31.040 And if you look at this ecosystem, you have neo cloud partners that really focus on being that broker between the capital provider. 00:07:31.120 --> 00:07:35.839 So they will find your capital provider and they will map you and connect your capital. 00:07:36.160 --> 00:07:44.800 And you then can get like the full end-to-end experience where they will come with the software to load your uh AI pipelines. 00:07:45.199 --> 00:07:53.199 Um, so you know, some of them have built-in uh orchestration, computer orchestration stack, and they will package that as part of the service. 00:07:53.360 --> 00:07:57.920 Uh, others will force you to rent the equipment from them. 00:07:58.079 --> 00:08:02.879 And so then you're a little bit more at their uh, how would I say? 00:08:02.959 --> 00:08:36.960 Uh you know, you're sort of like dependent on that infrastructure provider because, you know, last year last week,

for example, there are customers that were complaining that only had a one-year contract, their one-year contract was up, they thought they could renegotiate, but those interrupt providers, those neo cloud providers said they couldn't extend their contract because they were replacing that old equipment with brand new equipment that was yielding higher returns, was able to serve a higher level of customer or a customer that was willing to pay more money. 00:08:37.120 --> 00:08:47.200 So even if you have a contract in place, um, it doesn't necessarily mean that you can extend that contract automatically, you know, you're sort of like back in the queue. 00:08:47.360 --> 00:08:54.639 If you want to purchase the hardware there as well, you have to really understand that there's a huge uh shortage in GPUs. 00:08:54.720 --> 00:09:18.159 So you have to be uh aligning yourself with those neo clouds that really have the allocation and can provide you a commitment on getting uh an allocation because there is well, I've heard cases with customers last week that said that they had been waiting for months to get allocation of these GPUs and yet didn't get the allocation, and we're back in the queue. 00:09:18.480 --> 00:09:22.320 Um, so because a lot of customers want maybe smaller configurations. 00:09:22.480 --> 00:09:29.039 What I do see is there are some new clouds that have uh eight uh that have servers with eight GPUs in them. 00:09:29.200 --> 00:09:36.879 So the smaller, more deployable, I would say servers, but even that we're talking about more than half a million dollars in in price point. 00:09:37.039 --> 00:09:38.559 So it's it's expensive. 00:09:38.799 --> 00:09:52.000 So yeah, I would say understand your long-term commit, align yourself with the right new cloud provider, and then um look at the ecosystem of tooling that exists today because there's different orchestration stacks. 00:09:52.159 --> 00:09:54.080 Um we're seeing multiple pop up. 00:09:54.240 --> 00:10:00.080 Some of them are very specific to the neo cloud provider themselves, others are open source. 00:10:00.159 --> 00:10:06.159 Um, so you see SkyPilot is coming out as one of the open source compute orchestration stacks. 00:10:06.320 --> 00:10:12.879 Because in the end, what you want is you want flexibility to move your job, your compute job, to where the compute is. 00:10:13.039 --> 00:10:24.240 And what we're seeing now is that in contrary to like the hyperscalers, right, you would sign up for a hyperscaler, you trust that hyperscaler, and your workloads would stay within that hyperscaler. 00:10:24.320 --> 00:10:30.720 The hyperscalers made it really difficult to move out of the hyperscalers because they charge you an arm and a leg. 00:10:31.039 --> 00:10:40.559 If you want your data back, AWS, for example, charges you up to$90 for one terabyte of data that you want to move out of the cloud, super expensive. 00:10:40.799 --> 00:11:03.279 So now when you're deciding to move or take advantage of this ecosystem of neo clouds, and we're talking about thousands, right, of providers in this ecosystem, thousands of providers that are spread around the world, then you should be thinking about how do I move my data to these providers and where is my data located in the first place. 00:11:03.679 --> 00:11:23.840 Um, and that's you know the second challenge that I wanted to talk about because data orchestration now becomes a real problem, knowing that this new market is fully this disaggregated across multiple vendors and multiple types of GPUs. 00:11:24.080 --> 00:11:49.840 You have to be thinking about how do I set myself up for success, meaning how do I keep my data in a neutral place so it's not locked in with one of these providers, so that I have the negotiation power, the leverage to move to another provider when there's a better price, or it's uh maybe uh sovereign deployment, which we're seeing now as well, because sovereign deployments are now pretty hot overseas. 00:11:50.000 --> 00:12:06.240 If you look at Canada, if you look at Europe, the need to protect uh your assets within country boundaries and you know, uh leverage uh open source LLMs instead of some of the US frontier labs models keeps coming up. 00:12:06.320 --> 00:12:13.440 And so that's a whole new market as well, where compliance is sort of driving the need for local deployments and local control. 00:12:13.759 --> 00:12:27.039 So, yeah, depending on where you're located, depending on the strategy you're you're willing to take, the encommitments you're trying to make or willing are willing to make, you should very uh should be very selective uh in your job, in your selection. 00:12:27.600 --> 00:12:41.840 I wonder just hearing all this, it's gotta be so frustrating for business leaders because I almost wonder how is this legal that they're you know trapping them and trapping their data, their own data, not releasing it back to them. 00:12:42.000 --> 00:12:48.320 Do you think that we'll soon see some regulations that maybe control a little bit of this? 00:12:48.480 --> 00:12:56.559 Because I'm sure most business leaders, it's it's so frustrating to try to figure out which place to go with and not be trapped. 00:12:56.639 --> 00:13:06.159 And then we have longer and learn longer commitment terms and longer amounts of time you have to give notice to break away, and it's not easy to move somewhere else. 00:13:06.320 --> 00:13:08.960 So it seems so frustrating. 00:13:09.440 --> 00:13:10.240 Yeah, yeah, yeah. 00:13:10.399 --> 00:13:21.440 I mean, it's it is so a lot of customers are complaining about one there not being enough upper um access to GPUs, so that forces them to make a call, right? 00:13:21.519 --> 00:13:35.360 They have to either purchase, make a really large purchase, which they may not have the access to, or go with a new cloud provider that uh has certain boundaries on how to, you know, how how long they can use it for or what the minimum commit is.

00:13:35.519 --> 00:13:45.120 But the hyperscalers have always been more um sort of like they've been holding the data hostage of the customer because right now it's all about your data, right? 00:13:45.360 --> 00:13:46.799 Because what are you actually doing? 00:13:46.960 --> 00:13:52.720 You're running your fine your fine-tuning models on private data sets, your or your inferencing. 00:13:52.879 --> 00:14:05.360 I mean, training is too expensive for the enterprise customers to take this on themselves, um, which is why you would take existing trained models, fine-tune them on your private data sets, and then differentiate that way. 00:14:05.600 --> 00:14:17.919 But fortunately, in this new neo-cloud architecture, most of the neo-cloud providers, they will they will be more open about data moving freely across those neo-cloud providers. 00:14:18.000 --> 00:14:27.360 So that is actually a good thing because they realize that that is important to for them to attract customers that are willing or are interested in using your services. 00:14:27.519 --> 00:14:30.960 And so you see this movement away from vended lock-in. 00:14:31.519 --> 00:14:34.960 And then it comes down to well, where is my data stored? 00:14:35.120 --> 00:14:44.720 If my data is still stored in AWS or Google, then I'll have to really think thrice about how and do I have the budgets to move it out maybe once. 00:14:44.799 --> 00:14:46.240 And then where do I move it to? 00:14:46.399 --> 00:14:49.679 Which is why with a Kaveh, uh, we don't charge for egress. 00:14:49.840 --> 00:14:51.200 Customers can store it with us. 00:14:51.279 --> 00:14:58.000 We're presenting ourselves with a neutral data layer that can be used across multiple GPU partners. 00:14:58.159 --> 00:15:02.639 And then the customer can more freely select a GPU partner and move their data to it. 00:15:02.799 --> 00:15:25.360 But second, what we're also seeing is the cost of these GPUs and the storage that's embedded in these GPUs is going up so high that it is just not possible to store all your data in the storage, these NVMe devices that are extremely expensive, closer to that that are sitting next to the GPU. 00:15:25.600 --> 00:15:52.720 So instead, um customers are asking for a local, sort of like an edge object store that sits close to the GPU, but is is accessible over network, is accessible also remotely, and doesn't have these high price points, but still has the locality, the throughputs, and offers a digital repository, right, where you can actually like store persistent data in a persistent way. 00:15:52.960 --> 00:15:59.840 Because the feedback we're we're hearing is that customers will move from one pot, from one GPU pot to another GPU pot. 00:16:00.080 --> 00:16:05.039 And when they do that, you kind of lose all your settings, your configuration on the cluster. 00:16:05.200 --> 00:16:13.840 And when you want to restart your testing, you have to bring your snapshots back, you have to bring your LLMs, your

weights. 00:16:14.159 --> 00:16:19.840 And so uh the faster you can pull those back, the faster you can start using your GPU. 00:16:20.000 --> 00:16:26.320 So it's super important that the the cold restart or the cold boot restart problem, right? 00:16:26.399 --> 00:16:27.919 That you can solve for that. 00:16:28.080 --> 00:16:36.559 Because if you're taking if you're bringing your data back again over and over again from let's say hyperscaler, you're sort of waiting for that data to come back in. 00:16:36.639 --> 00:16:42.559 And these data sets are you know tens of terabytes, these snapshots, just to kind of get things going. 00:16:42.720 --> 00:16:53.120 So we're seeing um a need for a local digital repository to preserve those snapshots, these weights, these LLMs in a persistent way. 00:16:53.360 --> 00:16:57.120 So actually now we're moving more towards the edge, you can say, right? 00:16:57.440 --> 00:17:02.559 And it's great to see that trend because um we're sort of going back. 00:17:02.720 --> 00:17:08.559 We're coming back from this whole cloud move, right, to sort of a new architecture. 00:17:08.720 --> 00:17:11.119 I don't think we're replacing hyperscalers. 00:17:11.279 --> 00:17:20.559 This is more augmenting, right, the workloads that we currently see in enterprise, but they're more tailored, more focused towards these new AI pipelines. 00:17:20.720 --> 00:17:22.480 So giving them more control. 00:17:24.000 --> 00:17:24.799 Correct. 00:17:25.039 --> 00:17:29.759 And then what we're seeing as well is the interest in confidential compute. 00:17:30.000 --> 00:17:37.680 Um, so confidential compute is where, as you are moving towards these new neo clouds, the question then is do I trust the neo cloud? 00:17:37.920 --> 00:17:40.559 Do I feel comfortable with my production data? 00:17:40.799 --> 00:17:54.559 Um, so there are newer techniques now to ensure that when your data is executed or is being processed in these GPUs at these edges or in these at these neo clouds, that it's done in a confidential way. 00:17:54.640 --> 00:18:05.599 So there's um now ways to you know demonstrate, like basically enforce, you know, and on the on the chip level, that the data won't be decrypted outside of the GPU. 00:18:05.759 --> 00:18:24.240 So you can you can feel comfortable then that your data will only be de-encrypted uh or be unencrypted basically uh when it's actually being executed, so that the the owner, the the service provider that's serving the compute infrastructure will never have access to the actual data itself. 00:18:24.400 --> 00:18:43.599 And that this is sort of like the confidential computing working group at the Linux Foundation has been driving that with the major um compute vendors and also the cloud providers, because this is not only uh a fear that customers have when using neo clouds, but you know, still with the hyperscales, they have the same challenge. 00:18:43.759 --> 00:18:54.240 And thus, as customers are more sensitive now and more aware of the fact that these uh sensitive data sets or their private data sets, sort of their differentiator. 00:18:54.559 --> 00:18:58.799 Uh, they're trying to protect those data sets uh more carefully. 00:18:59.200 --> 00:19:09.920 And so we're seeing customers definitely in the enterprise space that are more interested in owning their own infrastructure as a sovereign deployment, maybe managed by NeoCloud. 00:19:10.160 --> 00:19:15.599 Some of them are looking at having it stored or installed, deployed in their own data center. 00:19:15.759 --> 00:19:25.839 But then again, a lot of the enterprise customers today have yeah, have moved a lot of their existing infrastructure into the clouds and to hyperscalers. 00:19:26.000 --> 00:19:39.920 So this opens up, you know, new opportunities for those neo-cloud providers, ourselves, our Cave as well, to build new services in this new world where compute is the decentralized. 00:19:40.240 --> 00:19:44.240 Well, that does, you know, that does bring to question, you know, the trust issue. 00:19:44.480 --> 00:19:49.839 This data that's in many different places now bring being brought closer. 00:19:50.000 --> 00:19:53.359 So perhaps it is a little safer, more trustworthy there. 00:19:53.599 --> 00:19:54.640 What are your thoughts though? 00:19:54.720 --> 00:20:00.640 And what tips do you have for business leaders to ensure that trust? 00:20:00.880 --> 00:20:13.519 And, you know, also just to be able to access the data when they need it, or when, you know, when something goes wrong or a mistake is made, um, being able to quickly be able to sift through their data. 00:20:14.079 --> 00:20:14.640 Yeah. 00:20:14.960 --> 00:20:29.200 Well, I mean, so uh the sovereign deployments is definitely like a way like to isolate your environment, but still have it in a a managed OpEx type of uh Post profile. 00:20:29.279 --> 00:20:45.839 Meaning, you know, a lot of customers, like I mean, I was talking to um an enterprise customer, I can't say the name, but they basically said, yeah, my my executive team wants us to move our our data lakes, you know, that are currently in Snowflake or in the AWS, but used mainly by Snowflake. 00:20:46.079 --> 00:20:51.920 They want to move those back on premises, but we don't have the the people to manage it. 00:20:52.000 --> 00:20:55.759 We actually don't manage on-prem storage or on-prem compute. 00:20:55.920 --> 00:21:04.000 And so that's where NeoCloud is perfect, but you know, they're they would be interested in having a sovereign kind of deployment. 00:21:04.079 --> 00:21:12.799 So you have a fully separated tenant, fully isolated, just for them, but managed in a colour that's hosted by this NeoCloud. 00:21:13.119 --> 00:21:20.000 And um, then of course, it's important that you have uh integrated your your key manager. 00:21:20.079 --> 00:21:28.480 So you have your your own key manager uh that allows you to control your keys so that you're in control over uh the

encryption keys. 00:21:28.559 --> 00:21:29.599 That's super important. 00:21:30.000 --> 00:21:33.519 Uh, second is yeah, the confidential compute could help with that. 00:21:33.599 --> 00:21:46.720 If you're really worried about um still there being a potential trust issue with the new cloud provider, you could go down the path of looking for a new cloud provider that offers confidential compute, although that's still early days. 00:21:46.960 --> 00:21:52.240 So most customers are really looking for a separate tenant, fully isolated. 00:21:52.319 --> 00:21:54.400 Um, they call those AI factories, right? 00:21:54.480 --> 00:21:57.920 So it's completely custom kill it for the customer. 00:21:58.240 --> 00:22:08.240 But if you do need extra Compute power and you need a temporary, like you basically need to like to expand on the compute side, then you can still use those compute marketplaces. 00:22:08.400 --> 00:22:21.279 What we're seeing there is that some of these AI engineers that are doing tests that are still sort of like in the development phase of like testing training, not training, but building their fine-tuning models. 00:22:21.359 --> 00:22:34.880 Um, they're they're doing the tests, some of these tests on these compute marketplaces that give you sort of spot prices that allow you to like more easily get access to GPUs on a short notice. 00:22:35.039 --> 00:23:00.079 But there, it's important that you create snapshots of your environment so that you don't have to rebuild from scratch and you have disaster recovery or some type of recovery copy that you can restore from uh whenever you need to, because if something gets corrupted, and you know, uh there's multiple ways that things can get corrupted in a cluster. 00:23:00.319 --> 00:23:15.119 Some uh engineer last week was telling me that even the phys, uh, the basically the network connectors um could overload and that could slow down performance and could intact eventually crash. 00:23:15.279 --> 00:23:25.920 And so there's a lot of uh reasons why, not only software, but also hardware, obviously, wise, uh that could yeah, where you could actually lose, you know, multiple days of work. 00:23:26.160 --> 00:23:33.519 Like a partner told me that a snapshot is worth minimum 300 hours of engineering hours minimum. 00:23:33.680 --> 00:23:42.480 So you can calculate how much that costs to you and the time it would take to rebuild your cluster before you can actually start using your GPUs. 00:23:42.559 --> 00:23:51.519 So, you know, like the time between your purchasing access to the GPUs and the time that you actually can use it, that's extremely expensive, right? 00:23:51.599 --> 00:23:58.079 So if you can reduce that to minutes instead of days, yeah, you're saving a lot of money. 00:23:58.240 --> 00:24:00.160 So it's not just uh the time. 00:24:00.559 --> 00:24:11.039 This is not just it's not just the ability to be able to recover, but also the time it takes for you to actually spin up those uh GPUs and utilize them. 00:24:11.440 --> 00:24:12.319 Absolutely. 00:24:12.559 --> 00:24:18.240 Well, if there was one key takeaway you could leave our audience with today, what would that be? 00:24:19.759 --> 00:24:23.759 Uh one key takeaway, uh yeah, don't don't wait. 00:24:24.000 --> 00:24:37.200 I've never seen such high demand for infrastructure that it we really need to start making decisions on whether you're long-term uh committed to AI. 00:24:37.359 --> 00:24:43.359 Because if you are, act quickly, mostly because at this moment September, right? 00:24:43.519 --> 00:24:44.880 September is 2026. 00:24:44.960 --> 00:24:46.480 We'll see what next year brings. 00:24:46.640 --> 00:24:53.440 But at this point, it's getting harder and harder to get access to to capital, to to the equipment. 00:24:53.759 --> 00:24:55.920 And so, you know, start now. 00:24:56.000 --> 00:24:56.880 That's what I would say. 00:24:57.039 --> 00:25:06.799 Even if it's like your decision is I'm gonna wait another year before I'm gonna actually deploy some of these pipelines, but um a lot of customers are not in that position. 00:25:06.960 --> 00:25:14.640 A lot of customers are actually in the in a position where they're trying to roll out proof of concepts and yeah pipelines. 00:25:14.880 --> 00:25:24.960 So start to start to start to lock in your your hardware or at least your rental agreements, like to to uh support your business strategy. 00:25:25.359 --> 00:25:26.319 Most definitely. 00:25:26.559 --> 00:25:26.960 All right. 00:25:27.039 --> 00:25:30.960 Well, thank you so much for coming on the show and sharing your insights with us today. 00:25:31.279 --> 00:25:32.400 Okay, thank you, Anna. 00:25:32.559 --> 00:25:33.279 Thanks for having me. 00:25:33.599 --> 00:25:34.559 Thank you to our audience. 00:25:34.640 --> 00:25:39.279 If you have any questions or comments, leave those below, and I'll try to respond back as soon as possible.