TRANSCRIPT 🔗
WEBVTT
NOTE
Transcription provided by Podhome.fm
Created: 01/26/2026 23:55:45
Duration: 2761.825
Channels: 1
1
00:00:05.120 --> 00:00:08.000
Hello, and welcome to the AI Engineering Podcast,
2
00:00:08.000 --> 00:00:12.160
your guide to the fast moving world of building scalable and maintainable AI systems.
3
00:00:19.685 --> 00:00:24.485
Unlock the full potential of your AI workloads with a seamless and composable data infrastructure.
4
00:00:25.060 --> 00:00:32.739
Bruin is an open source framework that streamlines integration from the command line, allowing you to focus on what matters most, building intelligent systems.
5
00:00:33.300 --> 00:00:41.255
Write Python code for your business logic and let Bruin handle the heavy lifting of data movement, lineage tracking, data quality monitoring, and governance enforcement.
6
00:00:41.735 --> 00:00:48.935
With native support for ML and AI workloads, Bruin empowers data teams to deliver faster, more reliable, and scalable AI solutions.
7
00:00:49.495 --> 00:00:55.255
Harness Bruin's connectors for hundreds of platforms, including popular machine learning frameworks like TensorFlow and PyTorch.
8
00:00:55.910 --> 00:01:00.230
Build end to end AI workflows that integrate seamlessly with your existing tech stack.
9
00:01:00.550 --> 00:01:05.510
Join the ranks of forward thinking organizations that are revolutionizing their data engineering with Bruin.
10
00:01:05.750 --> 00:01:09.575
Get started today at aiengineeringpodcast.com/bruin.
11
00:01:09.575 --> 00:01:14.135
And for dbt cloud customers, enjoy a $1,000 credit to migrate to Bruin Cloud.
12
00:01:15.095 --> 00:01:23.175
Your host is Tobias Maci, and today I'm interviewing Hugo Shi about the strategic realities of sourcing GPUs in the cloud for your training and inference workloads.
13
00:01:23.500 --> 00:01:25.420
Hugo, can you start by introducing yourself?
14
00:01:25.659 --> 00:01:48.235
Yeah. My name is Hugo Shi. I'm the cofounder and CTO of Saturn Cloud. I've been working on data science and machine learning for a long time. And the reason why I'm interested in this topic is because we are like a platform layer that sort of lets AI and ML engineers self-service the compute that they need, and we install into GPU clouds. And so like, that's why we've been doing a lot of research and work on understanding the GPU cloud market.
15
00:01:49.115 --> 00:01:52.235
And do you remember how you first got started working in the ML and AI space?
16
00:01:53.620 --> 00:01:57.140
So this was a long time ago, actually. This was in 2009.
17
00:01:57.220 --> 00:02:14.325
Canvas was not really a thing yet, or it had maybe had just been released. I was primarily using NumPy, and the history was that I was actually like a MATLAB jockey because my PhD program was all MATLAB code. And then the first job that I took out of grad school was a desk quant job at a
18
00:02:14.565 --> 00:02:48.505
options market making firm. And so for there, was doing mostly MATLAB and Java, and then I wanted to get some sort of scientific compute environment that I could run at home open source. And so being a MATLAB user, I naturally tried Octave. I wasn't super happy. And then my friends told us, Hey, you you should check this Python stuff. And I was like, Oh, that doesn't sound right. It's like some dumb thing that web developers use. And And then I started messing around and turned out it was pretty cool. And it could unify like general purpose compute along with scientific computing and mathematical compute, and I fell in love. And so as you mentioned, you're now responsible
19
00:02:48.505 --> 00:02:49.545
for a business.
20
00:02:49.970 --> 00:02:56.690
A big portion of your client base is very interested in being able to execute workloads against GPUs.
21
00:02:56.690 --> 00:03:00.610
And I'm wondering if you can just start by giving your summary of your
22
00:03:00.770 --> 00:03:06.695
understanding and view of the current market of cloud GPUs and some of the different players?
23
00:03:06.935 --> 00:03:11.575
I would say the the most common source of GPUs are the hyperscalers. So these are AWS,
24
00:03:11.575 --> 00:03:32.590
GCP, and Azure. And then I guess Oracle is like the up and coming hyperscaler even though they're a huge company. So I I would group them in there. And the hyperscalers tend to have higher prices for GPUs, for example, like h one hundreds are generally around $10 an hour on the hyperscalers. Then there are GPU clouds and GPU clouds, they specialize in providing access to GPUs. They also have CPU instances many times, but their pricing
25
00:03:33.085 --> 00:03:36.045
tends to be much better, somewhere between 1.5
26
00:03:36.045 --> 00:03:46.765
to $3 to $4 an hour for an h100. And then I would say that within the GPU cloud market, it gets a little it's pretty, it gets pretty interesting. So you have full service clouds, and I would classify those as
27
00:03:47.340 --> 00:03:51.100
clouds where they often have additional managed services, not
28
00:03:51.580 --> 00:03:59.020
to the same depth as the hyperscalers, but they also have like usually managed Kubernetes, managed Slurm. Oftentimes they'll have VPCs,
29
00:03:59.180 --> 00:04:13.825
load balancers, things like that. These are clouds like Lambda Labs, CoreWeave, Nebius, Crusoe. And then there's also bare metal clouds, which I would, or I would say they're more concierge clouds. And that is where maybe I'll talk to sales and I say, Hey, I need 64 H100s
30
00:04:13.825 --> 00:05:09.575
for the next six months. And they say, okay, and I wire transfer some money and then they provision the machines and they email me like SSH keys and IP addresses or something like that, and then I have access. And then finally, I think something that's also interesting is, there's a GPU aggregator market that I would say is a subset of the GPU cloud market. And these are companies like Shadeform, Vast AI, RunPod, FluidStack, SF Compute, RevDev. There
Load full transcript
are more. And then also, I would point out that a lot of times these business models get mixed. So for example, RunPod and FluidStack, they do aggregate GPUs, they also have their own hardware, whereas SFMpute primarily aggregates GPUs and does not have their own hardware. And so GPU aggregators is just, you know, you go to one place and you can access GPUs from a bunch of different providers. It's something that I never thought would work just because that's, like, very counter to how most people have been using cloud computing. Like, most people try to stay in one cloud only. But because of GPU scarcity and GPU pricing, the aggregator model is actually taking off. And I think that's probably
31
00:05:10.135 --> 00:05:14.970
a good we can dig into more, but that's that's a that's a high level summary of the market.
32
00:05:16.330 --> 00:05:20.330
And another interesting layer on top of that is
33
00:05:20.650 --> 00:05:25.130
there are businesses such as yours where one of their major
34
00:05:25.685 --> 00:05:35.525
value add is the fact that they will manage orchestration of your workload across these various GPU providers and abstract some of that complexity. And I'm wondering how you see
35
00:05:35.845 --> 00:05:38.405
those players in this broader ecosystem.
36
00:05:39.980 --> 00:06:17.180
Well, there are not as many of us. So and I would say and and actually the other the most recent, the big one, Lightning AI, just acquired or merged with Voltage Park, and so there's even fewer now. I would say that in the GPU cloud market, there's still not a lot of products that are geared towards it because the market's a lot smaller. There's a lot of open source stuff, right? Because most of the GPU clouds have either a managed Kubernetes service or you can set up your own Kubernetes service, and then any other platform that works on Kubernetes like Kubeflow, you can install and run, and I think that's what most people do. But there are, I think, a growing number of companies that are doing what we're doing. I guess the ones that I'm most aware of are like DSTACK and Metaflow,
37
00:06:17.980 --> 00:06:33.634
and also Flight. And I would say that these products all have like significant overlap, but they're also quite different. So, you know, they tackle different parts of the stack. But the point is that there's a lot of There's a growing amount of products that are focused on helping people run different workloads on GP clouds.
38
00:06:34.810 --> 00:06:44.810
Another one that I'm aware of just because I had them on my podcast somewhat recently is a company, I think they're called ARIA AI. But, again, with many of these providers,
39
00:06:44.889 --> 00:06:48.485
they're not necessarily concerned with managing
40
00:06:48.485 --> 00:06:55.445
the access to GPUs as the core primitive that they're exposing. They're interested in getting you onboarded to
41
00:06:55.925 --> 00:06:58.085
their model of how to actually
42
00:06:58.405 --> 00:07:05.500
create and manage those workflows, and so you're going to be abstracting on top of whatever APIs or
43
00:07:05.660 --> 00:07:08.940
primitives that they're going to expose versus
44
00:07:09.180 --> 00:07:11.260
just saying, here's some raw compute,
45
00:07:11.340 --> 00:07:18.074
have at it. Yes. Exactly. And then I guess the other thing that I would add is that some of these additional services blend
46
00:07:18.074 --> 00:07:21.835
like a managed AI service with aggregator models.
47
00:07:22.235 --> 00:07:31.770
So for example, well, Lightning did this. Like, you could access Lightning, then from Lightning, you could dispatch to a bunch of different clouds. And we do some of that as well, but,
48
00:07:32.490 --> 00:07:35.770
I think that is distinct from a model where
49
00:07:35.930 --> 00:07:54.155
the application actually runs in your cloud account, which for a lot of enterprises, that becomes more and more important because you get better security and tighter integration with, your existing services. And so I think a lot of these products that function as aggregators can also be deployed in this other model. I know, like, Daxter can take advantage of compute in your environment. Flight can also,
50
00:07:54.635 --> 00:07:56.315
and Metaflow can as well, I think.
51
00:07:58.330 --> 00:08:00.330
And for people who
52
00:08:00.490 --> 00:08:27.514
their primary concern is just I need to be able to access these high powered GPUs either because I'm trying to do my own training run of building my own foundation models or fine tuning or because they need a certain amount of capacity to handle whatever scale of inference they're trying to deal with or maybe because they're running some fluid simulation model, they don't even actually care about any of this generative AI stuff. I'm just wondering how you see the general calculus
53
00:08:27.750 --> 00:08:29.830
deciding which category
54
00:08:29.910 --> 00:08:34.710
of provider to work with. And then once they have made that determination,
55
00:08:34.870 --> 00:08:37.510
how to actually go through the process of identifying
56
00:08:37.750 --> 00:08:40.630
which providers is going to offer the things that they actually need.
57
00:08:41.505 --> 00:08:46.065
Yeah. That's a great question. I would first start by assessing
58
00:08:46.065 --> 00:09:05.380
how much security you need and also how many managed services you think you're gonna need. So if you're just spinning up, you just need, like you just want SSH access to a machine, like, lot of providers can get you that. If you need, you know, managed Kubernetes and be to be able to set up VPCs and load balancers, then that puts you into a slightly different category of providers. And then if your
59
00:09:05.620 --> 00:09:07.300
security needs are,
60
00:09:07.540 --> 00:09:31.660
lighter if you have lighter security needs, then the aggregators are pretty compelling. Right? I would say the aggregators tend to have the best prices, but the aggregators tend to have less managed services because they're dispatching to all these different clouds. Right? And so they sort of only have access to the least common denominator of all the clouds they support or they expose different services depending on which cloud you're dispatching to. So I would say assess if you need a lot of managed services
61
00:09:31.980 --> 00:09:41.435
and assess if you need high security. If you need a lot of managed services and high security, I would go with one of the more full featured GPU clouds. So CoreWeave, Nebius, Crusoe, Vultr. If you
62
00:09:41.915 --> 00:09:45.675
don't need as many managed services or your security posture is lighter, I would go with,
63
00:09:45.995 --> 00:10:09.350
an aggregator like, SF Compute, vast.ai. And I'm not saying that the aggregators are not secure because they're good engineers. They've done everything probably correctly, but it's just that, you know, if all you're doing is working with Nebius, then you just have to be concerned about Nebius' security. If you're working with the aggregator, then there's just more links. And so, you know, the weakest link in the chain, there's just more links. So it's probably easier to have a weakest link in there somewhere or weaker link in there somewhere.
64
00:10:11.165 --> 00:10:14.125
To the point of the managed services
65
00:10:14.125 --> 00:10:16.525
and the amount of
66
00:10:16.765 --> 00:10:18.285
feature set available,
67
00:10:18.685 --> 00:10:20.685
that can definitely be a very
68
00:10:20.925 --> 00:10:21.645
strong
69
00:10:21.805 --> 00:10:22.685
differentiator
70
00:10:22.685 --> 00:10:36.070
for some of these different clouds where AWS and GCP, obviously, they've been around for a long time. They've established a lot of reputation. They've invested substantial capital and engineering hours in actually building out these various components
71
00:10:36.070 --> 00:11:11.204
that are superimposed on top of the core primitives of I just need a way to, you know, execute an instruction set on a CPU and have some means of storage and random access. And for some of these more bare bones providers where it's just, hey. I've got a server rack with dozens of h one hundreds have at it. I'm just wondering how that factors into some of the effort involved in being able to actually take advantage of the compute and, like, how much the feature set trade off versus I can save 50 or a 150%
72
00:11:11.204 --> 00:11:18.805
on the actual compute costs? Like, what do you see as being some of the maybe organizational factors that play into
73
00:11:19.400 --> 00:11:23.720
going through that calculus and determining, and decision structure?
74
00:11:24.040 --> 00:11:41.465
Yeah. I would say I mean, ultimately, it's just gonna come down to how many GPUs you need, how much you think you're gonna save, and then whether that's worth it in terms of the additional work that you're gonna have. But I do think it's interesting to talk about the
different services that GPU clouds provide because they tend to be different than what the hyperscalers provide. So at its core, every GPU cloud provides machines
75
00:11:41.705 --> 00:11:54.860
with GPUs, right? That's their job. I would say the next tier of service is that most of them, many of them will have a way to give you a Kubernetes cluster or a Slurm cluster. And then I would say on top of that, the next tier is storage.
76
00:11:55.100 --> 00:12:00.620
So, you know, not all GPU clouds have storage, or they may just have storage on the machine,
77
00:12:00.940 --> 00:12:30.600
but it's just on the machine. Like, you can't attach it to other machines. Like, it's just it's just the ephemeral storage that's on on the node that you're on. But a lot of GPU cloud providers are have their own, like, block storage service. They have their own NFS or shared file system storage, and they have their own object storage. I think a growing trend is that a lot of these clouds are using providers like Vast Data and Weka. And I just want to clarify because there's Vast Data, which is a data platform, and then there's vast.ai, which is a GPU aggregator. So different companies do different things, similar
78
00:12:31.415 --> 00:12:36.375
But Weka and vast data have gotten really good at providing high performance,
79
00:12:36.615 --> 00:12:46.490
local and shared storage that drastically outperforms the NFS and give you really fast access to your training data. And so, yeah, so there's that. And then I guess on top of storage,
80
00:12:46.730 --> 00:13:25.365
the next tier I would say is networking services, and not everyone has these, right? So load balancers, not not every GPU cloud has load balancers, you sort of take them for granted on the hyperscalers, but that's not always there. A lot of them do, but not all of them do. And then VPCs, private subnets, things like that, right? A bare metal cloud that's just saying, here's an SSH key and a public IP address. There's no, you know, everything's publicly everything's all your machines are exposed to the Internet, right? And then if you go to a different, you know, a that has more features that supports a VPC that has public and private subnets, you can deploy your nodes into the private subnets, that gives you much more control over networking. So that's how loosely I think about it. Would say raw compute, Kubernetes and Slurm, storage,
81
00:13:25.365 --> 00:13:32.005
networking services, and then finally the last year after that is managed services. So these are like managed ML flow, managed Postgres.
82
00:13:33.125 --> 00:13:42.080
Those, there tends to be the fewest amount of that, but that's growing because people are starting to expect more and more. But I definitely think like Kubernetes, Slurm, storage,
83
00:13:42.320 --> 00:13:50.400
those are networking. Those are like generally the those are like the key features that you would need. You assess which ones do you need and you make determinations based on that. And like the managed service stuff is sort of nice to have.
84
00:13:51.280 --> 00:13:53.200
One of the other interesting
85
00:13:53.200 --> 00:14:02.225
aspects of where we are now with this growth of newer cloud providers that are very focused on a specifically GPU access
86
00:14:02.305 --> 00:14:12.830
in terms of the overall evolution of the technology sector, the past decade plus has been very focused on the idea of cloud native computing.
87
00:14:12.830 --> 00:14:15.230
There's the cloud native computing foundation.
88
00:14:15.470 --> 00:14:18.910
The substrate of all of that is largely Kubernetes.
89
00:14:18.910 --> 00:14:36.330
And one of the key promises of that overall effort was this idea of workload portability of it doesn't matter what cloud you're in or whether you're on a cloud at all. You can just run it on your hardware. You just as long as you have Kubernetes, you can do whatever you want. And the reality is that there are it is a leaky abstraction,
90
00:14:36.730 --> 00:14:43.290
and every Kubernetes cluster has its own set of plug ins and assumptions made and underlying
91
00:14:43.290 --> 00:15:07.240
feature sets that are being connected into. And the cloud providers have also managed to find their way into your Kubernetes cluster by adding some of these different operators to say, hey. We'll automatically provision your load balancer and your storage layer. And I'm just wondering what you're seeing as some of the realities that teams are coming up against as they say, well, I'm on Kubernetes. So, of course, I could just use this other Kubernetes cluster over there on
92
00:15:07.399 --> 00:15:27.285
Lambda or whatever it might be, and then they start trying to actually deploy their workloads and run into some of those sharp edges. I'm just wondering how teams are starting to come to grips with some of those realities and either say, oh, well, I'm only going to run this one specific workload and then tailor may you know, tailor their deployments to that reality
93
00:15:27.285 --> 00:15:40.710
or just some of the ways that teams are thinking about what workloads to move and how to think about making them actually portable, particularly if they're just chasing the bottom dollar or the cheapest cost for any given provider?
94
00:15:41.590 --> 00:15:45.270
Okay. So first, I think the short version is if you're deploying microservices
95
00:15:45.270 --> 00:15:49.635
and most of your things are containerized, I would classify that as cloud native. And however,
96
00:15:49.875 --> 00:15:55.235
I would say, I think we should be making a decision between Kubernetes native and cloud native because
97
00:15:55.475 --> 00:16:09.980
there are lot of people who have cloud native deployments, but they're just like, it's like Terraform orchestrating ECS, right? And so if if you you got Terraform orchestrating ECS, okay, want to move to GCP, fine, move your containers over, but none of your Terraform
code's gonna work, right? Because those APIs are all different between AWS and GCP.
98
00:16:10.220 --> 00:16:36.420
So I would say, I understand the promise of cloud native being very portable, I don't I think if you're just, like, using Terraform to use ECS, that's it's not as portable. Like in some ways, just deploying to an EC two instance is more portable because you can just, you know, deploy the same instance on GCP. Kubernetes native, however, I think that's very interesting. Right? So if you're doing Kubernetes native stuff and then the question is like, do your containers make use of cloud APIs or not? Right? So if you're using ECR and if you're using S3, then you have to make sure that when you go to
99
00:16:36.740 --> 00:16:42.855
another cloud, you can either deploy those services or they have them out of the box. And so but I think from a Kubernetes
100
00:16:43.014 --> 00:17:00.930
from a Katz native perspective, this is much easier, right? Because at least then your code isn't changed. The the YAML LFS and the Helm charts that you're deploying will still work on all the clouds that you're deploying to. In practice, because we've been focused on deploying the same application, Saturn Cloud, on different clouds, we've had to deal with this problem, right? And so typically, the way we do it is
101
00:17:01.250 --> 00:17:01.970
we
102
00:17:02.450 --> 00:17:15.995
deploy a bunch of Helm charts. The Helm charts expect there to be block storage, network storage, object storage, and container registry. And so on clouds that don't have that, the nice thing is that we can deploy open source Kubernetes versions of those services
103
00:17:16.235 --> 00:17:55.030
into the cluster and then just configure your services to talk to those endpoints instead. And so so I think that's I think for Kubernetes native people, it's much easier. You just deploy the same Helm charts and then you figure out which services you need to patch. That, but I think, I mean that addresses the question like very mechanically. I think the other part is that, is data gravity, right? Because that's the other issue is that if all of your data is in AWS, then accessing it from Nebius might be kind of tricky. And so for there, I think it's I don't think there's any magic bullet. You either decide to copy your data over to the GPU cloud once, and then you deal with the egress cost once, or you set up some smart caching layer, and so then you deal with egress cost constantly, but you're trying to minimize it. And I don't think there's a better solution to that. Ironically,
104
00:17:55.030 --> 00:18:08.994
most of the GPU clouds actually have free egress. And so what you could do is you could store your data in the GPU cloud and then access it from the hyperscaler, but most people don't do that because they view their hyperscaler as their home base, Right? Because most of their other services that are on GPU focused run there.
105
00:18:09.955 --> 00:18:12.355
And on that
question of data gravity,
106
00:18:12.434 --> 00:18:16.274
how do those realities change when you're moving from the
107
00:18:16.355 --> 00:18:19.475
very high throughput, high bandwidth needs
108
00:18:19.800 --> 00:18:22.680
of running a training workflow
109
00:18:22.920 --> 00:18:23.720
versus
110
00:18:23.880 --> 00:18:24.600
the
111
00:18:24.760 --> 00:18:26.840
still high throughput potentially,
112
00:18:26.840 --> 00:18:37.945
but much lower, you know, orders of magnitude lower than a training run when you're just running inference. And you say, well, I need to be able to act assess my context corpus for this either chat or
113
00:18:38.184 --> 00:18:56.609
agentic use case, but I'm not talking about terabytes or petabytes of data that I need to be able to have random access to for being able to do this training run. Right. So that actually, that that brings up another feasible pattern, which is, I think what you're alluding to is like, could well, let me take this back. So so training workloads are generally
114
00:18:56.770 --> 00:19:06.635
much more integrated into your company, as in they're going to touch more pieces of data, they're going to call more of your APIs, they're going to access a lot more stuff that is sensitive, right? Whereas the inference,
115
00:19:06.875 --> 00:19:45.419
can just deploy a public endpoint, maybe it has to call out to some of your API services to get some additional context, but you're generally okay with those things running on the private internet. So another strategy is like you keep your training workloads in the hyperscaler, and then you can move your inference workloads to GPU clouds. Of course, that depends on where most of your spend is, right? So and I don't think it's always I think okay. Training workloads are more expensive, but you don't do as many of them. Inference workloads are cheaper, but you do a lot of them. So, you know, that trade off depends on your company. I think most companies spend more on inference now. And I think that's mirrored by the fact that many of the GPU clouds also deploy managed inference services, right, because they see that trend, they they know that it's easier easier to move inference workloads to them. So like Crusoe and Nebius
116
00:19:46.780 --> 00:19:50.940
and many other providers have their own dedicated inference services now where you can deploy your models.
117
00:19:52.140 --> 00:19:54.620
In terms of that data gravity question,
118
00:19:55.100 --> 00:19:58.404
that is not anything new. It's just being
119
00:19:58.885 --> 00:20:02.085
accentuated and Yes. I guess, revisited
120
00:20:02.085 --> 00:20:07.765
because of these change in workload patterns. But there were a set of technologies that were introduced
121
00:20:08.110 --> 00:20:22.830
a decade or so ago as an outgrowth of the Hadoop ecosystem and the just store all of your data, and someday it will be useful. One of the ones that I'm most familiar with is Aluxio that is effectively a
caching layer explicitly built for these
122
00:20:23.295 --> 00:20:26.095
high data requirement use cases
123
00:20:26.415 --> 00:20:39.100
as compared to other layers of caching that are maybe more application or content centric. And I'm just wondering what you're seeing as some of the trends as far as people may be reconsidering use of some of that data virtualization
124
00:20:39.260 --> 00:20:43.419
pattern, either specific technologies or just the the overall concept.
125
00:20:44.539 --> 00:21:09.315
I'm trying to think if I am aware of any useful patterns. We see a lot of stuff, but I don't know if there's any clear patterns on that side. Yeah. I don't think I have anything to say about, like, useful common patterns. I think there's a lot of different use cases, but I don't think the patterns have really emerged yet other than either you copy once or you continuously copy. So other than that, I haven't seen anything. More sophisticated, that's becoming a clear pattern. Right? Other people are definitely influencing more sophisticated things, but those are usually tailored to their specific use cases. I'm not seeing more general trends yet.
126
00:21:10.990 --> 00:21:23.309
Yeah. Interestingly, I haven't visited the Alexio page for some time, but when I look at it right now, one of their first pieces of content that they're surfacing is make multi GPU cloud AI a reality. Yes.
127
00:21:23.950 --> 00:21:29.505
It makes sense. It makes sense. And the other aspect of the overall economy
128
00:21:29.505 --> 00:21:32.465
of GPUs is that particularly
129
00:21:32.785 --> 00:21:33.505
in
130
00:21:33.585 --> 00:21:35.345
the 2020 through
131
00:21:35.585 --> 00:21:37.825
maybe 2023 or 2024
132
00:21:37.825 --> 00:21:38.465
time frame,
133
00:21:38.890 --> 00:21:43.690
there was a dramatic shortage of availability of these GPUs
134
00:21:43.690 --> 00:21:47.610
as an outgrowth of the supply chain impacts from COVID, etcetera,
135
00:21:47.929 --> 00:21:55.404
as well as the rampant growth of AI use cases and all of the different cloud providers trying to buy up all of the inventory.
136
00:21:55.565 --> 00:22:01.965
And I'm just curious what you're seeing now both as some of those supply chain restrictions
137
00:22:02.125 --> 00:22:04.044
have eased as well as
138
00:22:04.365 --> 00:22:12.480
successive generations of sufficiently powerful GPUs have been released and just some of the ways that that changes some of the economics
139
00:22:12.480 --> 00:22:21.279
as well as some of the demand for either a specific GPU model versus just something that has at least this type of capacity.
140
00:22:23.195 --> 00:22:27.355
Okay. So the GPU scarcity is still real.
141
00:22:27.435 --> 00:22:48.580
It is less severe than it was before, but we still have capacity issues. On the hyperscalers, it's still hard to get the top tier GPUs. Even on the GPU cloud, sometimes they don't have on demand capacity depending on what region you're trying to deploy into.
I think the interesting trend around GPU clouds is that most of the demand or most of the reserved demand, and I'll talk about what I mean by that, is
142
00:22:48.875 --> 00:22:49.595
gets
143
00:22:49.755 --> 00:23:01.275
funneled towards the top tier GPUs. And so, for example, like when g b 300 is widely available, everyone wants g b 300 because it's the latest and greatest. And what that means is that when people's contracts expire,
144
00:23:01.355 --> 00:23:12.379
they want to, if possible, roll over to the latest generation of GPUs, which means that suddenly you have much more on demand capacity for the previous generations of GPUs. And so that's starting to happen.
145
00:23:12.620 --> 00:23:22.655
There's much more on demand access for h one hundreds now than there were before. It's still not h one hundreds are still not, like, super easy to get, but they're a lot easier now. And so I think that is going to become
146
00:23:22.815 --> 00:23:30.575
interesting as there's more and more on demand capacity because then the tricks that the hyperscalers started using, spot instances and capacity reservations,
147
00:23:30.750 --> 00:23:32.590
can also be applied
148
00:23:32.590 --> 00:23:34.350
to help GPU clouds,
149
00:23:34.590 --> 00:23:36.669
optimize the use of their capacity.
150
00:23:37.870 --> 00:23:40.590
The other aspect that I think is
151
00:23:40.830 --> 00:23:44.830
factoring into some of the demand side question is
152
00:23:44.910 --> 00:23:47.545
the growth of competition
153
00:23:47.705 --> 00:24:02.809
both from AMD as a major player with companies like Meta being one of their major providers as well as their continued investment in the Rockm stack to act as a competition against the lock in of CUDA, but also
154
00:24:03.130 --> 00:24:07.210
looking at some of the ASICs and the specific
155
00:24:07.290 --> 00:24:09.930
inference chips such as AWS Trainium,
156
00:24:09.930 --> 00:24:17.605
the tensor cores that Google is investing in and just it makes it a much more complicated market as far as differentiating
157
00:24:17.605 --> 00:24:19.205
between GPUs
158
00:24:19.205 --> 00:24:29.365
versus inference compute. And there's some overlap, but they're not necessarily exactly the same. And I'm just wondering how you're seeing people try to navigate some of those questions as well.
159
00:24:31.179 --> 00:24:33.739
Okay. So I would say in terms of GPU
160
00:24:33.980 --> 00:24:35.580
or accelerator diversity,
161
00:24:35.580 --> 00:24:53.244
I think the most interesting players there are AMD. So a I mean, AMD is making great progress on that. I know when we spoke to a lot of GPU clouds before about the AMD GPUs, there were a lot of, like, ease of use reliability and hardware issues that they were dealing with, but that's gotten better. And so I think the GPU clouds are having an easier time getting access
162
00:24:53.404 --> 00:24:55.724
and standing up and making reliable
163
00:24:55.725 --> 00:25:21.134
AMD GPUs.
At the same time, the software stack for AMD GPUs is getting better, and so that's really interesting to observe. It would be great if there were multiple players in the accelerator market rather than just NVIDIA being the the dominant monopoly player. I think it's also interesting that I think AMD is doing a lot better with Rockham and PyTorch now than they were doing this kind of stuff before. I was at AMD AI Developer Day, and I met someone on the AMD team that was working on PyTorch.
164
00:25:21.295 --> 00:25:39.980
I don't remember his name, but one thing he told me was that he'd been he said that he'd been working on AMD open source stuff for, six years. And I was like, oh, that's really interesting I thought the Rock and PyTorch stuff was more recent than that. He said, Yeah, actually, he'd been working on their TensorFlow implementation. And so the way that they handled that project was like, they like forked TensorFlow and they made these changes and like, it wasn't compatible
165
00:25:40.300 --> 00:25:56.265
with the upstream, so it never got merged. So like, they had this own monstrosity that they had to maintain and deliver to their customers. So I think that's the wrong way to do this kind of thing. What they're doing now with PyTorch seems to be much better, right? They're, like, committing things upstream. It's in the PyTorch open source project. So I think I am encouraged that
166
00:25:56.870 --> 00:26:35.620
things there seem to be moving in the right direction. On the subject of hyperscaler chips, I don't really have a lot of data on this, but my personal bias is that I'm not optimistic about those just because I think people are people are already, like, hesitant about cloud vendor lock in. If you're using their silicon, like, okay, like, you just that's that's way far. And then also, like, just if you look at how much more AMD has to do to get all the open source stuff working well, hyperscalers could definitely do that, but, like, I just don't see them putting in that level of work yet to get all of that stuff running smoothly. But we don't support, like, TPUs, so I this is, like, stuff that I've heard about, but I I don't have hands on experience with TPUs.
167
00:26:36.895 --> 00:27:01.850
Yeah. That's definitely something that I have the vague sense of as well that all the cloud providers are trying to push their specific silicon to say, hey. It's great. We've got these things. They're easy to get to. But to your point, I haven't seen any real noise about people actually using them, and I think that they're largely just there for being the substrate for those cloud providers dedicated services like AWS Bedrock or Google's Vertex AI.
168
00:27:02.395 --> 00:27:05.515
NVIDIA has invested so much in the software ecosystem.
169
00:27:05.515 --> 00:27:44.855
Like, if you look at the stuff that I don't know exactly who's working on it, but I know Andy Terrell is. If you look at, like, the stuff that they're doing around CUDA
Python, like, that's really exciting, right? To be able to program all the all layers of the CUDA stack with directly in Python, like, that's that's that's pretty amazing. And it's going to take a lot of work for anyone else to catch up to that kind of stuff, right? I think AMD is not going catch up completely, right? But I'm optimistic that they'll get their stack to be good enough such that, you you can at least, you know, be successful with PyTorch, TensorFlow, JAX workloads on their chips. But but I think yeah. I just I just think the level of investment, like, I see NVIDIA investing most, AMD's close second. I don't see the hyperscalers investing nearly as much in this kind of stuff. But maybe maybe I'm just not seeing it because I don't pay attention too much to the TPU and Ferencia, like that kind of market.
170
00:27:46.400 --> 00:27:51.440
The name of the project is escaping me right now, but I know that there are also efforts.
171
00:27:51.440 --> 00:28:04.554
There's a I think it's a new language runtime that is a superset of Python, but their purpose is to make it easier to build GPU kernels to, again, help with easing some of that lock in of CUDA specifically
172
00:28:04.554 --> 00:28:06.234
and expand the
173
00:28:06.395 --> 00:28:08.794
ability to run across different
174
00:28:08.875 --> 00:28:09.914
accelerators.
175
00:28:09.914 --> 00:28:18.489
And I'm just wondering what you see as the the potential impact of work like that having on this broader ecosystem of training and inference.
176
00:28:19.770 --> 00:28:30.105
I don't know and see much about those types of projects these days. So I'm not saying I'm just saying that, like, I haven't dug deep into that space, so I not present myself as an expert in those type of libraries.
177
00:28:30.105 --> 00:28:33.065
But I would say, like, if you observe what happened to OpenCL,
178
00:28:33.065 --> 00:28:49.060
I'm not optimistic that that kind of approach is gonna be that successful just because it it just shifts the problem from the hardware vendors to, like, this open source project, which I would say, like, has less resource to devote to getting things running well on all these different ships. Maybe I think about it wrong. I just I that seems like it's gonna be really hard to get broad support.
179
00:28:50.740 --> 00:28:54.420
Yeah. So I just found it. Mojo is the name of the project.
180
00:28:55.934 --> 00:29:03.134
Okay. I mean, I wish them all this like, that would be fantastic. I wish them all the success. Like, that would be that would be great, but it does seem like a very hard problem.
181
00:29:03.455 --> 00:29:06.414
Absolutely. And then the other impact
182
00:29:06.414 --> 00:29:10.095
that I'm curious about is this trend towards
183
00:29:10.279 --> 00:29:11.159
smaller
184
00:29:11.240 --> 00:29:13.720
or more efficient model architectures
185
00:29:13.720 --> 00:29:17.719
and some of the ways that that is shifting some of the demand
186
00:29:17.960 --> 00:29:18.759
or
187
00:29:18.760 --> 00:29:20.919
requirements around specific
188
00:29:20.919 --> 00:29:22.519
GPU capacities
189
00:29:22.519 --> 00:29:35.565
and also the tendency towards trying to push some of these inference workloads to more edge compute locations or lower resourced compute locations and how that maybe shifts some of the eventual demand for GPUs.
190
00:29:35.804 --> 00:30:02.985
Okay. I will also disclaim that the edge stuff is not something that we deal with a lot because we we just focus on the cloud cloud market. But I will say, I think it's fantastic that people are making the code more efficient. Right? I think it's fantastic that they're finding smaller models or they're optimizing the architecture. But there are two sides. Right? So, like, you've got one side optimizing and making things more efficient, and so you're reducing demand. But then the other side is, the insatiable demand just keeps growing. But we need both. Right? We need we need to both make things more efficient, but we also need to grow capacity.
191
00:30:03.865 --> 00:30:05.144
Another interesting
192
00:30:05.225 --> 00:30:07.544
aspect of all of this is that
193
00:30:07.625 --> 00:30:08.985
for a lot of people,
194
00:30:09.465 --> 00:30:20.720
their first pass of saying, oh, well, this AI use case is really great. I'm just going to go and use the OpenAI's and Anthropix and Googles of the world and whatever API endpoint they're exposing.
195
00:30:21.039 --> 00:30:24.304
And then eventually, they gain sophistication
196
00:30:24.465 --> 00:30:27.585
in terms of being able to do things like evaluations
197
00:30:27.585 --> 00:30:31.025
or doing a better job of managing
198
00:30:31.345 --> 00:30:35.905
knowledge corpora to provide context to the model so that they can operate more efficiently,
199
00:30:36.065 --> 00:30:43.669
adding in things like caching, etcetera, Maybe they then migrate to some of these open weights models that they are self hosting.
200
00:30:43.830 --> 00:30:55.744
And I'm just curious what you see as maybe some of the overall industry trend or maybe some of the aspects of tooling and frameworks that allow people to
201
00:30:55.985 --> 00:30:58.065
maybe accelerate their overall
202
00:30:58.065 --> 00:30:59.184
sophistication,
203
00:30:59.345 --> 00:31:10.080
their overall kind of growth in terms of sophistication of being able to move to that. I'm going to run my own compute and my own models and maybe just some of the economic realities
204
00:31:10.080 --> 00:31:19.280
that are hard to gauge or understand when you're on one side of the fence or the other as far as maybe it's easier to just use the AI provider versus
205
00:31:19.360 --> 00:31:30.105
I'm going to actually run my own GPUs and compute and maybe optimize some things, but now I'm paying more for operational overhead or for headcount because I need more people who have expertise.
206
00:31:30.825 --> 00:31:33.544
I have an opinion. There's no magic answer
here. I think
207
00:31:33.945 --> 00:31:49.940
the first thing that you have to decide is whether the quality of your model is your competitive advantage or not. Right? So if you're just if your primary value add is that you're, you know, helping people aggregate their data, you're helping people do really good context engineering for models, like maybe you don't need to build your own models. But if your
208
00:31:50.100 --> 00:32:03.705
primary value add is that, like, you're a really good coding assistant or whatever, the quality of your model is one of your competitive advantages, then you absolutely have to invest in building your own models because otherwise you'll have no competitive advantage rate. Like your model quality is just as good as OpenAI,
209
00:32:03.705 --> 00:32:13.610
which is, it's going to be really good, but everyone else has access to that same level of model quality. So that's one thing. The second thing is that the tooling for the things that people want to do with LLMs,
210
00:32:13.690 --> 00:32:15.289
Serving, fine tuning,
211
00:32:15.450 --> 00:32:16.650
even training
212
00:32:16.730 --> 00:32:32.304
is getting better. And I think something interesting in that space is that there's the rise of, well, I'm gonna call them no code approaches or low code approaches, but that's not really that accurate. But basically, things where you can just sort of upload data and then it will handle the fine tuning for you as opposed to you writing code.
213
00:32:32.544 --> 00:32:44.590
And so I think that's an interesting trend. I'm biased because our platform is a code first platform. But the other reason why I'm biased is because we've seen this cycle play out before where people are trying to build, drag and drop data science environments.
214
00:32:44.669 --> 00:32:47.630
And I think when you're dealing with something as messy as data,
215
00:32:48.205 --> 00:32:58.445
UIs are not going to be the right way to program that. I don't think. Right. And so that's why I think I still think that the code first approaches are going to be what's winning at least for a long time.
216
00:32:59.565 --> 00:33:08.610
That idea of upload to fine tune your model also brings back some of the five or ten years ago trends of AutoML
217
00:33:08.610 --> 00:33:17.264
of being able to say, give me your spreadsheet. I will generate your predictive model for you using random forest or linear regression or whichever
218
00:33:17.985 --> 00:33:26.700
model of choice you want to use. And also, the the name of the project is escaping me, but one of the projects that came out of Uber that was very focused on that AutoML,
219
00:33:26.700 --> 00:33:30.059
but for deep learning of being able to try different
220
00:33:30.220 --> 00:33:31.420
neural architectures
221
00:33:31.420 --> 00:33:35.020
and then be able to generate a model for you based on your datasets.
222
00:33:35.180 --> 00:33:41.965
And I'm just wondering what you're seeing particularly as
somebody who is in that provider path.
223
00:33:42.285 --> 00:33:46.285
What do you see as the predominant use of these accelerators?
224
00:33:46.285 --> 00:34:09.905
Is it generative AI training and inference because that's what everybody's talking about right now, or are the realities much more nuanced and people are still relying very much on those random forest linear regression as well as being able to build their own deep learning models for certain predictive use cases and that it's actually a a very much a long tail where generative AI and large language models are just stealing the oxygen from the room.
225
00:34:10.545 --> 00:34:23.810
Oh, so right. So first of all, I do not think that traditional machine learning is dead. Like people are still using XGBoost random forest. Those are still better at doing what they're doing than large language models would be because XGBoost
226
00:34:23.810 --> 00:34:27.730
will generate a prediction in way lower latency with way
227
00:34:28.450 --> 00:34:40.065
way fewer computational resources than a large language model, and also probably be more accurate for the use case that you're using XGBoost for. So it doesn't make sense for those things to go away. They're still gonna be done. Maybe they're not as interesting now because
228
00:34:40.224 --> 00:34:53.800
if you're thinking about compute spend, way more compute spend is going to generative AI because generative AI workloads are way more expensive than traditional machine learning workloads. But yeah, I do think that AI is sucking the oxygen out of the room a little bit, right? Like AI makes
229
00:34:53.960 --> 00:34:59.960
everyone think that traditional machine learning isn't being done anymore and everyone's just building language models to do everything. That's not true.
230
00:35:01.095 --> 00:35:02.775
Yeah. Just wondering,
231
00:35:02.855 --> 00:35:16.210
you largely answered the question, which was basically just, is it actually all AI because that's a lot of what he's talking about, or is the reality a little bit more messy and nuanced than it's just that AI is having an outsized impact on the conversation.
232
00:35:16.770 --> 00:35:27.010
To your point, the actual compute spend because it's more expensive versus the level of value and utility that is being produced across the board from these different techniques.
233
00:35:27.725 --> 00:35:41.300
Yeah. Okay. One other thing I will add is that even amongst the AILM use cases, the usage patterns are not I think this is good example of how people who design products or why code first stuff is important and how you can't really scheme it.
234
00:35:42.100 --> 00:35:52.100
In this world, it's really hard to know exactly what your user is going to need to do and then build a UI around it. And so the example I'll give is that we worked with a biotech customer. That biotech customer is using protein
235
00:35:52.180 --> 00:36:04.965
or biological large language models, but their usage pattern is very different than most other companies, right? Most other companies, they'll just like, they'll fine tune and they'll deploy the model and then the model will be served by like VLM or something, and then they'll hit it from some more web application.
236
00:36:04.965 --> 00:36:21.745
These biotech firms, they may never deploy the model, right? What they do is they train the model and then they shove all their data through it, and then they check to see if anything interesting came out for the drug that they're trying to discover. And so there are people doing things with large language models that are never deploying inference services. If
237
00:36:21.985 --> 00:36:26.545
you just looked at the product landscape to understand usage, you would assume that everyone's
238
00:36:26.545 --> 00:36:32.225
uploading a bunch of JSON files and fine tuning this model and then clicking a button and it's getting magically deployed to some cluster.
239
00:36:32.465 --> 00:36:42.440
Yes, I'm sure that's happening in some places, but also people are just writing scripts and multi processing to just run a bunch of jobs that are each consuming a single GPU to run some language models to get some
240
00:36:42.839 --> 00:36:49.079
insights, which is not much different than like old deep learning workflows, right? It's just, it's using largely language models instead of a deep learning model.
241
00:36:50.465 --> 00:36:52.465
As you have been working
242
00:36:53.265 --> 00:36:55.665
in this space, working with your customers,
243
00:36:55.825 --> 00:37:01.985
what are some of the most interesting or innovative or unexpected ways that you're seeing teams be able to capitalize
244
00:37:01.985 --> 00:37:02.625
on
245
00:37:02.810 --> 00:37:04.410
GPU availability
246
00:37:04.410 --> 00:37:07.369
and economics across this growing
247
00:37:07.369 --> 00:37:08.810
number of providers?
248
00:37:09.690 --> 00:37:11.210
I would say not necessarily
249
00:37:11.210 --> 00:37:53.975
something that I've seen our customers do, but I think one of the more interesting things out there right now is SF Compute. So SF Compute is a GPU aggregator, but what's interesting about them is that it operates as a kind of like a financial marketplace. Like, so people who have GPUs that that meet their security criteria and have deployed their software can register their GPUs for sale on this marketplace, and they can set their prices. People who want GPUs can say how much they wanna buy them for. And if, you know, there's overlap, then then you buy your GPUs. And what's cool about this model is that it's not just actually have like a, like a forward curve of GPU prices. So it's not like, it's not just that you buy them. It's like, you can say, well, my next training run is going to be in three months, it's going to be from April 1 to April 14. Can I reserve 64 H100s
250
00:37:53.975 --> 00:37:58.300
at that time? And then the providers will see, well, this is what our
forecast of demand is.
251
00:37:59.100 --> 00:38:03.740
We're going have this capacity, so our price is this. And then, you know, the price gets set automatically.
252
00:38:03.740 --> 00:38:13.275
And I'm describing this process that like happens in a second, right? It's just, it's all automated. But yeah, you can just buy cheap capacity in the future when you know you're going to need it. And so that model, I think, is very interesting.
253
00:38:14.315 --> 00:38:18.234
And as you have been exploring and
254
00:38:18.395 --> 00:38:19.674
gaining understanding
255
00:38:19.674 --> 00:38:20.714
of this overall
256
00:38:21.150 --> 00:38:21.950
sector?
257
00:38:21.950 --> 00:38:28.510
What are some of the most interesting or unexpected or challenging lessons that you've learned in the process about working in this space?
258
00:38:29.630 --> 00:38:48.395
I don't know if this is for me, I think the most surprising thing was how much variance there was in GPU providers. Because if you're used to like, if you're coming from a world where you're used to working with AWS, AWS, GCP and Azure, it's like, okay, AWS, GCP and Azure, they're they're pretty different, right? AWS is much harder to use. GCP tends people think is the easiest to use, it's got the nicest UI,
259
00:38:48.555 --> 00:39:01.330
but they all pretty much have the same services. So like, whether the managed service I need exists or not is not something that you ever think about, because whatever hyperscale you move to, they all have the same managed services. They all have like a analytics database, a BigQuery, Redshift, or Postgres managed.
260
00:39:01.490 --> 00:39:56.760
You know When you go to a GPU cloud, like, you have to check some basic stuff, like, do they have block storage? Can you take snapshots? Are there backups? Right? Like there's, it's just, so I was surprised by how much variance there was. Like some GPU clouds out there are still like handling orders manually, right? You tell me what you want when you want it. And then I reserve it in our system. And then I manually provision the nodes and I send you SSH keys and IP addresses. Contrast that with like a service like Nebius that has a full blown managed Kubernetes service, you can also click on managed MLflow and it'll just deploy that on the chart to your Kubernetes cluster, right? There's a lot of variance in terms of software capabilities between the different clouds, much more so than exists in the hyperscale market. Think that was the most surprising thing for me. I guess the other surprising thing is that you forget about this, but when you work with the GPU clouds, because they have to do a lot less, they're actually way easier to use, right? So like they don't have to have very complex I'm permission setups because they don't do that many things. For them, not a question of like, do you need you don't have to opt into InfiniBand,
261
00:39:56.760 --> 00:40:02.225
you just get InfiniBand because everyone who uses it is probably going to use InfiniBand, right? And every node is deployed to an
262
00:40:02.545 --> 00:40:05.425
InfiniBand fabric. So I think the specialization
263
00:40:05.425 --> 00:40:06.225
definitely
264
00:40:06.465 --> 00:40:20.040
makes ease of use much better. Like we were deploying a lot to EKS and we had to manage CSI drivers for EBS and we had to manage, you know, the NVIDIA device plugin that did all these things that we had to do to make sure GPUs work properly. We go to Nebius Creusot, you just manage Kubernetes.
265
00:40:20.040 --> 00:40:41.755
All the operators that you need to get GPUs working are already pre configured. They all work correctly. There's no tweaking. It just all works. And so, and, you know, they can do that because that's their primary use case, right? So anyone who's using that cluster is going to want to use those services. Whereas if you're using EKS, like you may never need GPUs. So I think that's something that I understood intuitively, like that the GPU clouds would be able to make things easier,
266
00:40:41.915 --> 00:40:46.635
but like actually experiencing it was very surprising. I was surprised by how much less work there was.
267
00:40:48.089 --> 00:40:54.730
As you continue to work in this space and keep an eye on the market and work with your customers, what are some of the
268
00:40:55.130 --> 00:40:59.369
predictions or hopes that you have for the future evolution
269
00:40:59.369 --> 00:41:01.289
of the availability
270
00:41:01.289 --> 00:41:04.005
and economics of GPUs
271
00:41:04.005 --> 00:41:07.925
and some of the predictions that you have for the, I guess, need
272
00:41:08.165 --> 00:41:09.285
and evolution
273
00:41:09.285 --> 00:41:11.445
of GPU as a service providers?
274
00:41:12.724 --> 00:41:28.000
I definitely think there's gonna be more on demand capacity. Right? The thing that I said about h one hundreds more available on demand because everyone's rolling over to like GB300s or whatever. That's definitely a thing. And so I think it'll be very interesting. I'm excited to see how that plays out in terms of the availability of capacity reservations and spot prices
275
00:41:28.240 --> 00:41:39.285
on more of these GPU clouds. That's the first thing. The second thing I would say is in terms of trends, I think more and more GPU clouds are gonna become more and more full service. They're gonna look more and more like the hyperscalers.
276
00:41:39.365 --> 00:41:40.005
And so
277
00:41:40.485 --> 00:41:43.845
CoreWeave acquired Weights and Biases. DigitalOcean acquired Paperspace.
278
00:41:44.210 --> 00:41:45.170
And recently,
279
00:41:45.329 --> 00:41:48.450
Lightning AI just merged with Voltage Park. We're seeing this because
280
00:41:48.769 --> 00:42:20.360
I think the GPU clouds recognize that they have to move up the stack and not just provide raw compute, they have to provide a good developer experience for using the GPUs. And so I think that trend is going to
continue. And then the last thing that I would say is that there's going to be consolidation. Right? Because there's, I don't know, how many GPU class there's, like, 20 or 30 or a 100, depending on how you count them. And so it just doesn't make sense to have that many. Right? And so there's going be mergers and consolidations. Some of them are going to go out of business. But that also means that the ones that survive are gonna have more flush out ecosystems and are gonna be more competitive with hyperscalers.
281
00:42:20.360 --> 00:42:28.935
So I'm very optimistic about that. And then the last thing that I'm optimistic about for the future is just AMD and having multiple providers,
282
00:42:28.935 --> 00:42:32.295
hardware providers in the market that I'm optimistic and excited about.
283
00:42:33.255 --> 00:42:36.695
Are there any other aspects of this overall
284
00:42:36.855 --> 00:42:39.095
market and ecosystem
285
00:42:39.095 --> 00:42:39.895
for
286
00:42:39.895 --> 00:42:47.100
GPU as a service or your overall experience of working in the space that we didn't discuss yet that you'd like to cover before we close out the show?
287
00:42:48.140 --> 00:43:11.745
I think most people operate in a world where I think this is true, but basically, like onboarding a new vendor is difficult. Onboarding a new GPU cloud or new cloud period is even more difficult because people understand that clouds once you like a cloud is like a very long term relationship. Right? Like, you start to use their APIs, it it gets very tied in with your company. And so there's a lot of onboarding a new cloud's not an easy thing. I will say, though, that as the GPU clouds get more and more mature,
288
00:43:12.300 --> 00:43:16.620
that's becoming easier and easier. Right? A lot of them now have SOC two,
289
00:43:16.940 --> 00:43:36.465
ISO twenty seven zero zero one. I get I get the number slightly wrong each time I say that. And many of them even have HIPAA. So, you know, it used to be that getting security review for GPU cloud, that was now it's reasonable. Like, you could actually make the case at your company. Not saying that IT sec is gonna be like, oh, that's fantastic. Let's onboard a new cloud, but but you can at least make the case and you can get there.
290
00:43:37.185 --> 00:43:53.960
Alright. Well, for anybody who wants to get in touch with you and follow along with the work that you and your team are doing, I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get your perspective on what you see as being the biggest gaps in the tooling technology or human training that's available for AI systems today.
291
00:43:54.680 --> 00:44:11.645
Okay. So my viewpoint's gonna be biased because I'm so much on the infrastructure side, right? I'm not saying that there's other things like on the PyTorch side, or maybe JAX is better than PyTorch, like that kind of stuff. Model architecture stuff, I think there's tons of stuff there, but I'm much more exposed to the infrastructure side. I would say there's a lot of interesting work being done now around
292
00:44:12.280 --> 00:44:13.640
infrastructure reliability,
293
00:44:13.720 --> 00:44:19.480
because generally if you have a cluster of CPU machines, you're running them hard, but when you're running a GPU cluster, you're running them really
294
00:44:19.800 --> 00:44:21.960
hard. They're operating at super high temperatures,
295
00:44:23.080 --> 00:44:29.255
you've optimized to max them out to 100% whenever you can. You're optimizing your data loading patterns so that they're always
296
00:44:29.655 --> 00:44:38.535
that they're always saturated, they're always computing all the time, and as a result, GPUs fail a lot more, and so there are some projects out there that are focused on automatically
297
00:44:38.535 --> 00:44:40.055
relocating workloads
298
00:44:40.410 --> 00:45:02.865
when hardware fails. There's another project that I've heard of from a company called SystemStack that they actually have an application you can deploy that will sort of monitor node health, and instead of just migrating workload, it will actually try to repair the node for the most common failure conditions that it's aware of. And so I think that kind of stuff, stuff to make these long runs more reliable, I think that's the trend. That's
299
00:45:03.025 --> 00:45:05.025
interesting trend that I'm seeing right now.
300
00:45:05.425 --> 00:45:11.105
All right. Well, thank you very much for taking the time today and shedding some light on this overall
301
00:45:11.640 --> 00:45:12.680
ecosystem
302
00:45:12.680 --> 00:45:30.555
and the economics and realities of it. It's definitely something that I have been tangentially aware of, but certainly haven't explored to the depth that you have. So I appreciate you taking the time to share some of your lessons learned and insights from that, and I hope you enjoy the rest of your day. Oh, yeah. Thank you so much, and, thanks for having me. It was really fun conversation.
303
00:45:32.075 --> 00:45:40.349
Thank you for listening. Don't forget to check out our other shows. The Data Engineering Podcast covers the latest on modern data management, and podcast.init
304
00:45:40.349 --> 00:45:44.510
covers the Python language, its community, and the innovative ways it is being used.
305
00:45:44.750 --> 00:45:55.349
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a project from the show, then tell us about it. Email hosts@aiengineeringpodcast.com
306
00:45:55.349 --> 00:45:56.069
with your story.