প্রতিলিপি 🔗
WEBVTT
NOTE
Transcription provided by Podhome.fm
Created: 09/28/2025 23:45:41
Duration: 3391.384
Channels: 1
1
00:00:00.080 --> 00:00:09.380
Hope you enjoy this crossover episode of the AI Engineering podcast, which is another show that I run to act as your guide to the fast moving world of building scalable and maintainable AI systems.
2
00:00:10.240 --> 00:00:18.685
As generative AI models have grown more powerful and are being applied to a broader range of use cases, the lines between data and AI engineering are becoming increasingly blurry.
3
00:00:19.305 --> 00:00:28.204
The responsibilities of data teams are being extended into the realm of context engineering as well as designing and supporting new infrastructure elements that serve the needs of agentic applications.
4
00:00:29.220 --> 00:00:34.040
This episode is an example of the types of work that are not easily categorized into one or the other camp.
5
00:00:45.275 --> 00:00:49.455
Hello, and welcome to the Data Engineering podcast, the show about modern data management.
6
00:00:50.075 --> 00:00:58.655
Data teams everywhere face the same problem. They're forcing ML models, streaming data, and real time processing through orchestration tools built for simple ETL.
7
00:00:59.300 --> 00:01:03.320
The result, inflexible infrastructure that can't adapt to different workloads.
8
00:01:04.100 --> 00:01:06.760
That's why Cash App and Cisco rely on Prefect.
9
00:01:07.300 --> 00:01:15.885
Cash App's fraud detection team got what they needed, flexible compute options, isolated environments for custom packages, and seamless data exchange between workflows.
10
00:01:16.585 --> 00:01:21.485
Each model runs on the right infrastructure, whether that's high memory machines or distributed compute.
11
00:01:22.265 --> 00:01:26.525
Orchestration is the foundation that determines whether your data team ships or struggles.
12
00:01:27.270 --> 00:01:34.490
ETL, ML model training, AI engineering, streaming, Prefect runs it all from ingestion to activation in one platform.
13
00:01:35.270 --> 00:01:38.650
Whoop and 1Password also trust Prefect for their data operations.
14
00:01:39.395 --> 00:01:46.215
If these industry leaders use Prefect for critical workloads, see what it can do for you at dataengineeringpodcast.com/prefect.
15
00:01:46.674 --> 00:01:55.050
Are you tired of data migrations that drag on for months or even years? What if I told you there's a way to cut that timeline by up to a factor of six while guaranteeing accuracy?
16
00:01:55.930 --> 00:02:05.310
Datafolds migration agent is the only AI powered solution that doesn't just translate your code. It validates every single data point to ensure a perfect parity between your old and new systems.
17
00:02:06.090 --> 00:02:16.575
Whether you're moving from Oracle to Snowflake, migrating stored procedures to DBT, or handling complex multisystem migrations, they deliver production ready code with a guaranteed timeline and fixed price.
18
00:02:17.275 --> 00:02:22.655
Stop burning budget on endless consulting hours. Visit dataengineeringpodcast.com/datafold
19
00:02:22.715 --> 00:02:38.120
to book a demo and see how they turn months long migration nightmares into week long success stories. Your host is Tobias Macy, and today I'm interviewing Brijesh Tripathi about Flex AI, a platform offering a service oriented abstraction for AI workloads. So, Brijesh, can you start by introducing yourself?
20
00:02:38.694 --> 00:02:39.275
Yeah. Hi.
21
00:02:39.735 --> 00:02:44.555
My name is Brijesh Topati. I'm the CEO of Flex AI. I grew up in India.
22
00:02:45.175 --> 00:02:46.795
Did did my undergrad
23
00:02:47.255 --> 00:02:51.515
at IIT Kanpur and then moved to US, entered my master's at Stanford.
24
00:02:52.100 --> 00:03:00.120
And from there on, I've been lucky to be part of some of the most amazing companies in in the Bay Area. I worked at Nvidia, Apple,
25
00:03:00.420 --> 00:03:10.855
Tesla, and Intel. And what I realized during my career is that I really enjoy bringing people and technologies together, and I really like solving tough problems.
26
00:03:11.235 --> 00:03:19.495
So couple of years ago, I was working on an interesting project, which was one of the biggest, largest supercomputer of the world at that time, Aurora,
27
00:03:19.860 --> 00:03:24.760
that was handed over to Argonne National Lab. I realized that there is something that is
28
00:03:25.220 --> 00:03:26.599
missing in the current
29
00:03:26.900 --> 00:03:29.879
offerings at the infrastructure level, and that is
30
00:03:30.260 --> 00:03:31.400
the ease of access.
31
00:03:32.095 --> 00:03:34.275
We got the the systems installed,
32
00:03:34.575 --> 00:03:37.395
and it took us a very long time to
33
00:03:37.855 --> 00:03:40.675
get that handed over to some of our developers,
34
00:03:41.135 --> 00:03:41.635
scientists
35
00:03:41.935 --> 00:03:52.000
who were actually solving real world problems like climate change and medicine research. And from there, I got the idea that there is a layer missing in the infrastructure management
36
00:03:52.860 --> 00:03:53.600
that simplifies
37
00:03:53.980 --> 00:03:56.960
access to compute. So that's what Flex AI is all about.
38
00:03:57.420 --> 00:04:01.360
And do you remember how you first got started working in the ML and AI space?
39
00:04:02.265 --> 00:04:33.565
Yeah. I think it's it's a long story again, but it goes all the way back to really dealing with the complexities of AI. But before that, I was, you know, I was assigned a role of chief architect for AI and HPC architecture at Intel. The first task I had was to figure out what is the right architecture for AI and ML workloads. And I observed that a lot of the new architectures that are coming out now had one deficiency, and that was they all focused on the highest performance,
40
00:04:34.104 --> 00:04:36.845
yet they were all becoming very inefficient.
41
00:04:37.384 --> 00:04:43.965
And while I was trying to
Load full transcript
figure all of that out, I was also realizing that we build this massive amount of infrastructure,
42
00:04:44.599 --> 00:05:08.824
and a lot of that actually sits idle. So combine the two things, I've realized that I think we need to change the fundamental architecture of how we design these things and focus on efficiency, energy efficiency, and then change the way we deliver these compute to our end users. And that led to an infrastructure pallet platform management layer, which is which is what we are building at Flex AI.
43
00:05:09.604 --> 00:05:21.230
And so digging a bit more into Flex AI and what you're offering, I'm also interested in some of the ways that you're seeing those infrastructure challenges contribute to the friction
44
00:05:21.690 --> 00:05:23.710
that's involved in developing
45
00:05:24.010 --> 00:05:29.389
and operating these very compute heavy AI applications that have become
46
00:05:29.905 --> 00:05:33.285
very popular and prolific over the past couple of years?
47
00:05:33.825 --> 00:05:36.805
The chat GPT era basically changed
48
00:05:37.105 --> 00:05:44.005
a lot of the thought process on how small teams can actually be massively powerful and very efficient.
49
00:05:44.560 --> 00:05:47.380
So now you're starting to see one or two people companies
50
00:05:47.840 --> 00:06:05.835
that are actually building real good products, solving real problems. They actually are onto something. They have access to powerful models, dealing with very smart people, and they have this urgency to move fast. So they trade fast. They want to trade fast. But the challenge is that they all run into
51
00:06:06.375 --> 00:06:08.955
setting up infrastructure, maintaining infrastructure,
52
00:06:09.655 --> 00:06:16.889
and dealing with failures that happen left and right. So instead of solving real problems, these teams actually then become DevOps experts
53
00:06:17.190 --> 00:06:19.370
and and start dealing with infrastructure.
54
00:06:19.750 --> 00:06:29.385
That is slowing them down, and that's that's actually one of the challenges that we are trying to solve here. Now the other thing that that's actually also interesting is when you're dealing with the current infrastructure
55
00:06:29.845 --> 00:06:36.905
and your access is through renting GPUs from either hyperscalers or these large, you know, new clouds like Colby,
56
00:06:37.205 --> 00:06:40.185
you have to now deal with unpredictable cost.
57
00:06:40.530 --> 00:07:00.595
And if you're trying to solve a problem that requires you to have a GPU for, let's say, a few hours, but there is no access for a few hours for a GPU, you have to now go block it for a week or a month, and that literally spikes the bill that raises the bill to a point where you can run out of money while just renting GPUs
58
00:07:01.055 --> 00:07:03.875
before you even finish building what you're building. And finally,
59
00:07:04.335 --> 00:07:08.139
as people rent GPUs, they be they go and use infrastructure.
60
00:07:08.840 --> 00:07:09.819
Different places
61
00:07:10.280 --> 00:07:14.060
will end up requiring different setup. So if you go,
62
00:07:14.520 --> 00:07:29.955
compute from get compute from Azure, it will have a very different setup. If you get it from Amazon, it will be a very different setup. Google, same thing. And then these new clouds will offer various different settings like virtual machines or bare metal. All of that requires
63
00:07:30.335 --> 00:07:30.835
these
64
00:07:31.295 --> 00:07:34.195
very small teams who are actually not infrastructure
65
00:07:34.575 --> 00:07:50.590
experts to now have to go build the infrastructure layer on top of what they're getting. That takes a toll on these people. They they basically are spending less time building and solving their real problems, but they end up spending a lot of time just maintaining and managing these infrastructures.
66
00:07:51.125 --> 00:08:12.360
So, you know, as a founder, I actually feel for them. And, you know, I want to have a very small team, a scrappy team, and I empathize with these people. So sometimes when I deal with with these small groups of people, small teams, small start ups, and they realize the value of what Flex AI is doing for them, that makes my heart happy. So I'm I'm I'm really excited about
67
00:08:12.900 --> 00:08:15.240
helping some of these, you know, founders,
68
00:08:15.620 --> 00:08:21.745
my peers, who are trying to solve some real problems, get rid of the infrastructure problem they face every day.
69
00:08:22.205 --> 00:08:25.585
Particularly with a focus on those founders and
70
00:08:25.965 --> 00:08:32.065
projects that are in the early stages of proof of concept or trying to figure out the appropriate
71
00:08:32.605 --> 00:08:37.610
product market fit, what are some of the ways that that friction and the cost unpredictability
72
00:08:38.070 --> 00:08:42.410
at the infrastructure level can prevent them from being able
73
00:08:43.110 --> 00:08:43.850
to scale
74
00:08:44.150 --> 00:08:48.890
at the pace that they would otherwise be able to if that cost was not a factor?
75
00:08:49.745 --> 00:08:51.365
Yeah. I mean, if look. Iteration
76
00:08:51.745 --> 00:09:00.725
loop, the number of times you can actually go experiment with your model, improve your model, that's the key there. And if you're dealing with the infrastructure challenges,
77
00:09:01.105 --> 00:09:21.985
that means you're not iterating fast enough. That slows you down. If you slow down that pace that someone else can catch up and actually do something better than you. So losing your edge by just reducing the number of iterations or the making the loop longer is how they are struggling. And one of the things that Flex AI does is it takes the friction away from these,
78
00:09:22.365 --> 00:09:28.625
startups. You are just focused on your model iteration or your application iteration or your agent iteration,
79
00:09:29.160 --> 00:09:35.180
And rest
of it is taken care of by Flex AI, and that means you're going much faster than what you would do without us.
80
00:09:36.040 --> 00:09:48.345
And to be a little bit hyperbolic on the infrastructure piece, isn't that what Kubernetes was supposed to solve for us? You could just write your code and deploy it into Kubernetes, and it would run everywhere that you had a Kubernetes cluster. Right?
81
00:09:48.885 --> 00:09:52.265
So that's ideally true. If it actually worked periodically,
82
00:09:52.725 --> 00:09:54.905
that was the promise made. But, unfortunately,
83
00:09:55.779 --> 00:10:03.240
the dependencies, the libraries, the overall complexity of actually starting from x number of n number of Kubernetes
84
00:10:03.620 --> 00:10:07.080
implementation. Not everybody offers the exact same abstraction.
85
00:10:07.700 --> 00:10:18.814
It just doesn't work the way we we we expect people expected. And what in fact, that is actually one of the premises we have, which is we start from a Kubernetes layer that is now consistent.
86
00:10:19.194 --> 00:10:23.535
So our developers don't have to deal with the differences and the complexities of missing
87
00:10:23.850 --> 00:10:37.230
libraries or missing dependencies. It just works. But, fundamentally, you're right. It was supposed to solve all of this. It doesn't. In fact, one of the things that we are doing to bias is where in near future, we are going to offer our customers
88
00:10:37.535 --> 00:10:45.075
an ability to bring their own containers. And those containers are then okay to run on multiple places. That becomes
89
00:10:45.615 --> 00:10:54.460
a way for us to handle custom flows and flows that are not currently supported by our platform. However, it still requires a very stable
90
00:10:54.920 --> 00:11:02.300
underlying management layer, underlying orchestration layer so we can get better utilization. You know, if you get Kubernetes to work,
91
00:11:02.600 --> 00:11:03.900
it doesn't mean that
92
00:11:04.360 --> 00:11:06.460
you are now accessing resources
93
00:11:07.225 --> 00:11:17.165
that are dedicated to you. It means that you have to carve out those things out, and that means it's still dealing with a bunch of idle cycles and wasted resources.
94
00:11:18.185 --> 00:11:27.030
One of the aspects of running, in particular, these very compute heavy and GPU heavy workloads on a Kubernetes environment
95
00:11:27.410 --> 00:11:29.190
is that with CPUs,
96
00:11:30.050 --> 00:11:34.070
as long as they're the same architecture, it doesn't matter if you're running Intel or AMD.
97
00:11:34.610 --> 00:11:36.225
But if you are
98
00:11:36.845 --> 00:11:39.985
then in the GPU space and you're running between
99
00:11:40.445 --> 00:11:41.665
an NVIDIA GPU
100
00:11:42.285 --> 00:11:43.905
or an AMD GPU
101
00:11:44.285 --> 00:11:48.305
or a tensor processing unit, you need to have different drivers in place,
102
00:11:48.620 --> 00:11:56.160
which then also brings in questions around the software support in the code that you're writing, whether you're using CUDA or RockM or
103
00:11:56.540 --> 00:12:01.680
some other library that is able to take advantage of that compute accelerator.
104
00:12:02.334 --> 00:12:04.675
And that also brings in another layer of complexity
105
00:12:05.135 --> 00:12:06.915
and maintenance and management
106
00:12:07.295 --> 00:12:14.035
at the software level that is directly tied to the underlying hardware that you're working against. And I'm wondering how you're seeing
107
00:12:14.459 --> 00:12:30.725
teams address some of that complexity and some of the ways that you're thinking about that problem and maybe offering some sort of, you know, hardware, software abstraction layer to be able to paper over some of those or and then also that brings up the question of potential loss of efficiency
108
00:12:31.425 --> 00:12:38.084
when you do have that translation layer because you don't have just a direct throughput to the underlying hardware because you have to do that translation.
109
00:12:38.945 --> 00:12:44.839
Yeah. Yeah. Yeah. Otherwise, that that's an amazing by the way, that's a very long question, so the answer is going to be long. But that's
110
00:12:45.380 --> 00:12:46.680
really the crux of
111
00:12:47.139 --> 00:12:49.319
what holds back a lot of
112
00:12:49.699 --> 00:12:54.519
alternates compared to what NVIDIA has to offer. So, you know, if if you are
113
00:12:54.899 --> 00:12:58.120
writing a model, a lot of times, you will end up having
114
00:12:58.805 --> 00:13:07.144
very explicit CUDA calls in there that that now go target a specific NVIDIA architecture. And in fact, the way it is implemented,
115
00:13:07.524 --> 00:13:13.464
you know, if if you're running on a 100 and you move to h 100, there is no automatic
116
00:13:14.470 --> 00:13:20.650
gains that come from a new architecture even on NVIDIA. So it's so specific to a specific
117
00:13:21.110 --> 00:13:24.010
hardware model number that, you know, ignoring
118
00:13:24.630 --> 00:13:25.450
some small,
119
00:13:25.910 --> 00:13:28.010
updates, every generation requires
120
00:13:28.385 --> 00:14:02.635
a bunch of rewrite and a bunch of rework of the models, which is within the same architecture. So NVIDIA and AMD definitely doesn't work the same way. We have to deal with RockEm versus CUDA. What we are trying to do is actually take away that complexity from from developers. And and the market is actually kind of helping in in that sense. We are focusing on PyTorch. So PyTorch is an abstraction layer above CUDA where a lot you know, you you've seen a Triton layer being introduced in the Python's work, which which is trying to create a unified
121
00:14:03.255 --> 00:14:17.100
model or programming model for, multiple GPUs and and both AMD and NVIDIA supported in there. But at our level, we are actually staying with CUDA on NVIDIA, but we we have a code analyzer
122
00:14:17.560 --> 00:14:23.339
that will look at what your
CUDA calls are and suggest replacing it with a more
123
00:14:23.720 --> 00:14:27.115
universal code that can also then go run on,
124
00:14:27.735 --> 00:14:31.834
AMD. In some cases, we will even offer our developers, our customers
125
00:14:32.135 --> 00:14:32.635
support
126
00:14:33.014 --> 00:14:35.355
to port their code to different architectures.
127
00:14:35.894 --> 00:14:39.995
Now these are things that that are quite relevant for training. And,
128
00:14:40.470 --> 00:14:47.850
you know, as the market is moving more towards inference, we have made a decision that all of our training currently runs on NVIDIA.
129
00:14:48.230 --> 00:14:52.410
And so we focus on NVIDIA only with the training. However,
130
00:14:52.949 --> 00:14:54.649
as we move towards inference,
131
00:14:55.055 --> 00:14:57.634
we are moving away from specific,
132
00:14:58.735 --> 00:15:07.315
architecture optimization, and most of these inference solutions endpoints are being called with a standard open API OpenAI's API.
133
00:15:08.000 --> 00:15:10.100
Once we go into that level of abstraction,
134
00:15:10.480 --> 00:15:11.139
the implementation
135
00:15:11.440 --> 00:15:11.940
details
136
00:15:12.320 --> 00:15:13.139
per architecture
137
00:15:13.440 --> 00:15:27.925
is actually controlled by us. The developer or the user doesn't have to know which architecture it's going to run on as long as we deliver the same API and the same results, And that's helping a lot. In the next couple of years, we believe that, you know, a pretty big majority
138
00:15:28.305 --> 00:15:37.685
of workload AI workloads is going to be inference. And, hence, we are seeing a lot more multiple architectures being at play here. In fact, we are already now deploying
139
00:15:38.130 --> 00:15:46.950
AMD for inference and 10 store for inference. And in future, we are also considering looking at Trainium and a couple of other new architectures for delivering inference.
140
00:15:47.970 --> 00:15:49.110
The workload
141
00:15:49.730 --> 00:15:57.485
distribution was the next thing I was going to ask about, and you preempted my question a little by stating that you're focusing on inference. But to your point,
142
00:15:57.945 --> 00:16:04.205
inference is not the entirety of AI. It's just the thing that is gaining all of the attention right now. And so,
143
00:16:04.585 --> 00:16:05.645
obviously, the
144
00:16:06.110 --> 00:16:15.170
serving of a given model, whether that's a generative AI model that's focused on language outputs or a generative model that's focused on vision outputs
145
00:16:15.630 --> 00:16:16.930
that has a certain architecture,
146
00:16:17.550 --> 00:16:24.404
although even that is in flux right now. But regardless of whether it's a generative model or
147
00:16:24.865 --> 00:16:34.145
a predictive model that's using something like a random forest or an x g boost, the training time compared to the serving time are going to be,
148
00:16:34.700 --> 00:16:36.320
orders
of magnitude difference.
149
00:16:36.860 --> 00:16:38.240
And so I'm curious
150
00:16:38.779 --> 00:16:41.680
how you think about that aspect of the
151
00:16:42.140 --> 00:16:45.680
platform that you're offering in terms of being able to optimize for
152
00:16:46.060 --> 00:16:51.255
training time and iteration speed for being able to manage the experimentation
153
00:16:51.715 --> 00:16:53.255
loop and then the
154
00:16:53.555 --> 00:16:57.415
deployment and serving of those models where even if you are using
155
00:16:57.875 --> 00:17:10.080
a foundation model off the shelf, there are often cases where you want to be able to do some sort of post training and fine tuning of that model before you actually start serving it. And so I'm just wondering if you could talk to some of the ways that you're thinking about the overall
156
00:17:10.460 --> 00:17:14.320
AI life cycle as well as the expansion beyond the
157
00:17:14.715 --> 00:17:24.495
transformer attention based large language models that have gained all the popularity, and especially as we start to look into things like, diffusion based models and mamba architectures, etcetera?
158
00:17:25.115 --> 00:17:31.660
Yeah. That that that's, again, a really good relevant question for the time, Wes. Couple of things to start with. Number one,
159
00:17:32.200 --> 00:17:35.420
I believe that the world is going to be
160
00:17:35.720 --> 00:17:37.020
a heterogeneous compute
161
00:17:37.560 --> 00:17:50.115
world in the future instead of a homogeneous single architecture. And the and the way I maybe it's a simplifying thing for myself and and our implementation right now. But the way I think about this is every stage of this
162
00:17:50.575 --> 00:17:51.075
development
163
00:17:51.535 --> 00:17:52.755
starting from pre
164
00:17:53.135 --> 00:18:02.990
training to training to fine tuning to, you know, implementing a a deployment model, whether it's a rack thing or, you know, you know, some other ways of MCPs.
165
00:18:03.930 --> 00:18:08.350
Every stage is going to need or benefit from having
166
00:18:08.730 --> 00:18:09.790
somewhat customized
167
00:18:10.090 --> 00:18:11.790
and specialized architecture.
168
00:18:12.375 --> 00:18:21.595
And to us, to to Flex AI, our offering is going to make sure that the workflow doesn't really have to fundamentally change
169
00:18:21.975 --> 00:18:23.195
because we are switching
170
00:18:23.575 --> 00:18:26.075
which architecture is being at which stage.
171
00:18:26.400 --> 00:18:29.220
So what we are doing right now is for pretraining
172
00:18:29.600 --> 00:18:30.340
and training,
173
00:18:30.800 --> 00:18:42.815
and in fact, even fine tuning for some at some level depending on what the size of the cluster that we need to put there in, is we focus on NVIDIA because, you know, a lot of these models
174
00:18:43.115 --> 00:18:52.495
have been developed and are being developed with NVIDIA. So it's literally just simply too hard to actually start porting all of those models and giving a new architecture,
175
00:18:53.210 --> 00:19:10.235
you know, multiple years of, you know, time to develop and optimize and become competitive compared to what NVIDIA has to offer. So we kind of just say, to simplify the world, we have we have multiple things that we do on the pretraining and training side that still improve access
176
00:19:10.535 --> 00:19:12.235
and quality of training.
177
00:19:12.695 --> 00:19:40.635
We take care of failures automatically. We call it self healing. We take care of replacements automatically. So if you one node fails, we basically automatically replace it with another. We take care of scale. We take care of, checkpointing seamless checkpointing in the background without having to worry about wasting cycles. All of those things are actually already optimizing the run time or or the quality of the model and the iteration cycles that that are being developed. But as we move towards
178
00:19:40.935 --> 00:19:41.595
the deployment
179
00:19:41.975 --> 00:19:46.875
and serving these models, we start getting into multiple architectures and multiple
180
00:19:47.255 --> 00:19:47.755
options,
181
00:19:48.190 --> 00:19:53.730
which we believe some of them are better than what if we stayed with just one architecture or one,
182
00:19:54.430 --> 00:19:54.930
company
183
00:19:55.310 --> 00:20:01.250
are are possible. So we just we just take a look at every stage of the problem, every stage of the workflow,
184
00:20:01.790 --> 00:20:02.290
and
185
00:20:02.955 --> 00:20:05.534
pick the right answer for the right workload.
186
00:20:05.835 --> 00:20:18.810
And the end result is that you get the best of all words, not just two words. We we are even able to put ASICs in there for certain use cases. So there are some customers who are looking for edge deployment,
187
00:20:19.190 --> 00:20:47.710
and the cost will not allow them to put this on NVIDIA or AMD. On the other hand, if you are deploying it on edge, the the performance and the power is actually so limiting that you have to now serve it on a much smaller much smaller SoC or an ASIC, and we are actually even enabling that in our future. So, fundamentally, we believe that you should be able to select the right architecture for the right workload. And even within a workflow, you should be able to do that without worrying about having homogeneity across the board.
188
00:20:48.169 --> 00:21:09.455
You mentioned that workload as a service, which is something that is core to your offering and the the framing that you're presenting as far as the overall capability. And I'm wondering if you can dig into that abstraction and how that simplifies some of the approach of going from I have an idea to I have deployed an operating system
189
00:21:09.835 --> 00:21:12.815
that the workload as a service helps in
190
00:21:13.115 --> 00:21:15.215
accelerating that path to deployment
191
00:21:15.840 --> 00:21:17.940
and reduces some of these
infrastructure
192
00:21:18.640 --> 00:21:19.539
concerns and
193
00:21:19.919 --> 00:21:24.020
the variance and rapid evolution of the ecosystem that we've been discussing.
194
00:21:25.120 --> 00:21:26.260
Absolutely. Yes.
195
00:21:26.640 --> 00:21:30.020
Look. We are evolving from platform as a service
196
00:21:30.935 --> 00:21:35.515
to GPU as a service. And the last couple of years, this was the buzzword,
197
00:21:36.055 --> 00:21:38.795
GPU as a service. What it was was
198
00:21:39.255 --> 00:21:52.700
plain and simple, renting GPUs that only you have access to. You start renting with a renting a GPU. You have to now go build a stack on top of it to make sure that it can run whether you're trying to run training on it,
199
00:21:53.240 --> 00:22:10.455
pre train, fine tuning, or you are trying to deploy an inference server on it. And everything will require a very different setup. Everything will require you to go get the right libraries, get the right dependencies sorted out. At the end, generally speaking, it used to take
200
00:22:10.800 --> 00:22:14.180
teams days to weeks to set up a new infrastructure
201
00:22:14.560 --> 00:22:59.725
to start a new workload and then tear it all down and then put a new configuration on it to then run new set of workloads. And that was massively time consuming. So while you need to do that to be able to actually build what you are building so you can deploy some of your interesting ideas that you come up with, you were wasting a ton of time just setting it all up. So what we are trying to do with workload as a service is you worry about coming up with interesting ideas and deploying them, and we take care of the rest of it in the back end infrastructure management. So all you have to think about is what your idea is and what problems you're solving and not worry about the maintenance and management and, you know, keeping track of all of the heterogeneous
202
00:23:00.025 --> 00:23:03.164
compute that might be actually useful or part of your workflow.
203
00:23:03.945 --> 00:23:10.044
To your point of renting a GPU being the default method of gaining access to these
204
00:23:10.440 --> 00:23:11.419
compute accelerators,
205
00:23:12.279 --> 00:23:43.230
you then have to figure out, okay. Either I find the smallest GPU I can get away with for my model, and then you have a very fixed ceiling in terms of what you're able to do, or you say, I don't know exactly what I'm going to do. I'm going to find the biggest GPU or cluster of GPUs that I can find, and then I have to figure out how I'm actually going to maximize usage of it where maybe I have a fleet of LLMs that I'm serving for a chat based application, or maybe I'm building an agentic compute system.
206
00:23:43.610 --> 00:23:51.710
Or maybe I say, forget about renting GPUs. That's too much upfront capital. I'm just going to go the API approach and use a metered
207
00:23:52.035 --> 00:24:20.270
inference endpoint, and that has its own unpredictability and cost, potentially even more so. At least with the GPU, you know what the per hour rate is, and you you can understand that, but maybe you're not going to have the flexibility to manage the capacity that you actually need. And I'm wondering if you could talk to some of the ways that you've developed your platform and your architecture to be able to help smooth over some of those load patterns and get the sort of economies of scale as well as some of the ways that the unpredictability
208
00:24:20.890 --> 00:24:22.510
of these AI workloads
209
00:24:23.275 --> 00:24:28.495
complicates the work that you are doing to be able to manage that overall compute infrastructure
210
00:24:29.355 --> 00:24:34.655
and smooth out the those load patterns across the fleet of GPUs that you are providing.
211
00:24:36.090 --> 00:24:39.630
Again, really good question. And and, Tobias, we see this
212
00:24:39.930 --> 00:24:47.470
topic come up in lot more customer conversations these days than actual access to compute. You know?
213
00:24:47.850 --> 00:24:52.725
A lot of people are able to get compute these days, but their cost is becoming
214
00:24:53.025 --> 00:24:57.125
a real sore point. Recently, one of the, you know, favorite companies,
215
00:24:57.665 --> 00:24:58.725
Lovable apparently,
216
00:24:59.184 --> 00:25:00.965
showed up with $100,000,000
217
00:25:01.184 --> 00:25:04.465
ARR, but the cost of delivering that $100,000,000
218
00:25:04.465 --> 00:25:05.985
was almost $87,000,000
219
00:25:05.985 --> 00:25:12.440
or something like that. Like, it is just gigantic compared to what they are actually able to make out of these,
220
00:25:12.840 --> 00:25:15.419
GPU. So we definitely are
221
00:25:15.720 --> 00:25:30.725
building an architecture that allows people to smoothen their, you know, peaks and balance it with with a little more averaged, you know, overall capacity required. However, we also have this ability to give you
222
00:25:31.105 --> 00:25:32.085
more compute
223
00:25:32.690 --> 00:25:34.070
for lower cost
224
00:25:34.450 --> 00:25:38.790
when there is less demand. So it's it's almost like you know, think about how Uber
225
00:25:39.250 --> 00:25:41.030
would do, you know, price,
226
00:25:41.650 --> 00:25:42.690
surge surges
227
00:25:43.010 --> 00:26:02.960
surge pricing based on what the demand is. But if you go middle of the night, you know, there are no drivers. You just have to pay high. If you go, you know, peak hours where there's too much demand, even though there are drivers, you pay higher prices, and then there are the average times. For companies that are want that are trying to get a sort of an SLA for their customers,
228
00:26:03.500 --> 00:26:04.640
they end up choosing
229
00:26:04.940 --> 00:26:08.800
the highest level that they will ever expect to hit, which means
230
00:26:09.180 --> 00:26:10.720
most of the time, they're sitting,
231
00:26:11.340 --> 00:26:12.640
with idle infrastructure,
232
00:26:13.285 --> 00:26:18.105
and that's a huge cost. For companies that are trying to optimize for cost, they run into
233
00:26:18.405 --> 00:26:27.545
quality of service issues. We know at the peak hours, you know, your responses for your chatbot or or your queries is is gonna be hitting super high latencies.
234
00:26:28.110 --> 00:26:30.530
What we provide is a combination
235
00:26:30.990 --> 00:26:31.890
of access
236
00:26:32.670 --> 00:26:34.530
across multiple clouds
237
00:26:34.990 --> 00:26:36.610
and multiple architectures.
238
00:26:37.230 --> 00:26:39.885
So let let's just take a case of
239
00:26:40.285 --> 00:26:52.065
enterprise who's trying to serve, you know, a chatbot or, you know, support system using some model. And they see these variations in in their usage and their throughput,
240
00:26:52.440 --> 00:27:08.975
and they put some compute capacity on prem because that's a little more in control, and they they control the cost. But it is below the peak that they can actually hit. So what we do is if we bring their compute under our platform, we also connect a whole bunch of capacity
241
00:27:09.515 --> 00:27:11.055
outside of their infrastructure
242
00:27:11.435 --> 00:27:16.975
that is now available to them instantly. So the moment they get out of their own capacity,
243
00:27:17.355 --> 00:27:19.855
we have agreements with cloud providers
244
00:27:20.155 --> 00:27:20.975
with new clouds
245
00:27:21.340 --> 00:27:25.520
that we we can take their capacity and start running the workload from the customer.
246
00:27:25.820 --> 00:27:32.400
And there is a small delta between what the cost is going to be because it's an on demand capacity, but it's still lower than
247
00:27:32.700 --> 00:27:39.675
that company putting capacity that was actually going to meet that peak. On the other hand, somebody puts a capacity that is significantly
248
00:27:40.054 --> 00:27:42.075
higher than what they need normally.
249
00:27:42.615 --> 00:27:45.675
What we then do is we actually orchestrate
250
00:27:46.054 --> 00:27:46.554
various
251
00:27:46.855 --> 00:27:48.235
workloads using multitenancy
252
00:27:48.615 --> 00:27:58.690
and shared GPU concepts such that more than one customer, more than one user can use the same infrastructure at the same time. And that's another thing that, you know,
253
00:27:59.390 --> 00:28:20.780
CPUs have been doing this for a while. Right? I mean, virtual machines multiple virtual machines can run on a single machine until you run out of capacity, then you auto scale a new new node, a new server. GPUs weren't doing that, and we just started enabling that in our platform where you can run side by side a training workload and an inference workload on the same infrastructure.
254
00:28:21.160 --> 00:29:00.260
So given where the demand is from inference, the training can actually auto scale to a different size if there is, you know, a a QoS there that allows it to be downgraded a bit. Right? If you're doing, you know, fine tuning and you were letting it run for a few days or a, or a week or so, you know, slowing down during peak hours is not going to kill it. It's not going to affect your business too much. And that's a beautiful way of actually maximizing utilization of your existing capacity. So these are things that we are actually enabling right now. On the other hand, there's this last thing that we are adding, which is actually very exciting for us, which is we were able to put capacity,
255
00:29:00.560 --> 00:29:08.445
you know, let's say, an on prem capacity on NVIDIA GPUs, but there is inference that we can then launch on a different architecture.
256
00:29:08.905 --> 00:29:24.929
When you run out of your own capacity, we can just move that to a different architecture and scale it over there while minimizing your cost. These are things that were not possible before, but with our platform, we just seamlessly connect the users, the customer, and their workload scaling
257
00:29:25.230 --> 00:29:27.330
to multiple clouds, multiple architecture
258
00:29:27.870 --> 00:29:28.770
at any location.
259
00:29:29.710 --> 00:29:31.410
That speaks to the overall
260
00:29:31.870 --> 00:29:33.169
requirement for
261
00:29:33.615 --> 00:29:34.675
effective orchestration
262
00:29:35.215 --> 00:29:42.435
in any AI application, whether it's just being able to orchestrate the deployment of the model to the GPU
263
00:29:43.135 --> 00:29:47.875
or being able to optimize for cost because maybe the workload isn't
264
00:29:48.430 --> 00:29:56.770
interactive and user facing. It's a more long running agentic workflow, and so you can schedule the execution of that agent to those
265
00:29:57.150 --> 00:30:03.835
cheaper times, lower traffic times. And curious if you can talk to some of the ways that your workload as a service architecture
266
00:30:04.215 --> 00:30:05.035
is incorporating
267
00:30:05.335 --> 00:30:10.715
these orchestration concepts to be able to effectively manage the load distribution
268
00:30:11.735 --> 00:30:15.675
and identify workloads that are time sensitive or
269
00:30:15.975 --> 00:30:16.475
interactive
270
00:30:17.389 --> 00:30:17.889
required
271
00:30:18.269 --> 00:30:24.130
versus the ones that can be more asynchronous or are long running and able to take advantage of these
272
00:30:24.750 --> 00:30:26.130
distributions of load?
273
00:30:26.750 --> 00:30:50.670
Yeah. Absolutely. And and, you know, the concept is not new, Tobias. You know, if you have studied computer architecture, you know there is a whole concept of scheduling that the entire purpose there is to make sure every cycle of the CPU is busy. Why is that? Because that's the most expensive resource you have in the system. So your entire job in scheduling is to make sure every cycle is being used and used for the right purposes.
274
00:30:51.050 --> 00:30:56.190
But there's always a priority in there. One, the number one priority is based on the the actual,
275
00:30:56.570 --> 00:31:01.075
sequence in the queue. Right? You you pick the first one first. However, there are priorities
276
00:31:01.455 --> 00:31:14.019
attached to interrupts and high priority tasks, real time tasks that get picked out of the queue and executed much faster ahead of something else. So we apply very similar concept in our workload orchestration,
277
00:31:14.559 --> 00:31:29.515
and we let our customers define their priorities for various workloads. So they can say this is real time, and that means no one can interrupt them, and it will actually get ahead of the queue as long as there is no other real time thing ahead of the them themselves.
278
00:31:29.975 --> 00:31:33.355
They can set the priority to be, give me the best effort.
279
00:31:33.735 --> 00:32:01.555
Best effort means I will find the cheapest and the most inexpensive resource and put it in the queue behind everything else. It will run with every other real time and high priority task is completed. There are some that actually are not real time, but they still have a priority on them because it affects the business objectives. For example, if you have a training that runs in one day and if it ran over two days, you might actually have business losses.
280
00:32:01.855 --> 00:32:04.595
So in those cases, they don't get preempted,
281
00:32:05.135 --> 00:32:06.035
but they actually
282
00:32:06.415 --> 00:32:06.915
are
283
00:32:07.375 --> 00:32:26.029
getting similar priorities as, you know, real time, except the real time will always trumpet anything else because that is the priority zero. Then there's priority one, then there's this priority two. Depending on what the business objectives are, you can continue to set those things up there. Now here's here's an interesting thing that we discovered and then implemented
284
00:32:26.330 --> 00:32:26.830
recently.
285
00:32:28.154 --> 00:32:32.575
As most training jobs go, there's always a checkpoint happening. And
286
00:32:32.955 --> 00:32:34.255
when we were checkpointing,
287
00:32:34.875 --> 00:32:36.815
we all also got an opportunity
288
00:32:37.274 --> 00:32:56.915
to move that workload on the side if there is something else to preempt it. Interestingly, the cost of checkpointing has significantly gone down in the last one year. So now we do it seamlessly without waiting, without pausing the workload. It happens in the background pretty much zero cost to the workload. Every checkpoint, we get an opportunity to see if there is
289
00:32:57.215 --> 00:33:10.100
a there's a need to preempt it, go get a more high priority real time workload executed, get out of the way, and then go back and start running the same workload that is a long running workload that we previously checkpointed.
290
00:33:10.400 --> 00:33:12.340
So these things are actually now becoming
291
00:33:12.640 --> 00:33:24.195
very effective in maximizing the utilization of the existing infrastructure and resources we have access to. And that's something that I'm actually seeing a lot of people actually talking about, oh, we used to have 30%
292
00:33:24.195 --> 00:33:28.775
utilization to now. We are pushing our utilization to almost 80%,
293
00:33:28.835 --> 00:33:31.815
which is which is pretty good. One anecdote on this is
294
00:33:32.275 --> 00:33:36.215
recently, a few weeks ago, we had, you know, I call this the magical
295
00:33:36.515 --> 00:33:38.055
point, which was a proof
296
00:33:38.400 --> 00:33:44.100
of all of the things that we have been talking about. We had multiple customers on our platform.
297
00:33:44.400 --> 00:33:52.005
We had multiple compute clusters from various providers on our platform, and we have multiple workloads on our platform.
298
00:33:55.205 --> 00:33:55.705
Different
299
00:33:56.165 --> 00:33:56.665
GPUs
300
00:33:57.125 --> 00:34:03.385
consuming 100% of it. Everybody happily running for moving forward, and everybody combined
301
00:34:03.685 --> 00:34:10.830
using 100% of the compute capacity available. That was beautiful. That was beautiful. I posted this picture on LinkedIn just because I was so happy about it.
302
00:34:12.490 --> 00:34:13.950
And as you have been
303
00:34:14.410 --> 00:34:18.430
developing and iterating on this platform and the workload
304
00:34:20.145 --> 00:34:23.605
onboarding customers, what are some of the ways that the overall scope and implementation
305
00:34:24.065 --> 00:34:30.005
of your system have evolved from when you first started thinking about it and building the initial versions?
306
00:34:31.025 --> 00:34:31.525
Yeah.
307
00:34:31.980 --> 00:34:32.480
Good
308
00:34:32.860 --> 00:34:39.119
good, really good question. I I think when we started on this journey, you know, towards the beginning of 2024,
309
00:34:39.260 --> 00:34:46.865
large language models were still being built. There were many, many models being built at that time, But the pace has slowed down, and
310
00:34:47.485 --> 00:34:48.625
I'm gonna expect
311
00:34:49.085 --> 00:34:50.385
a a bit of consolidation
312
00:34:50.685 --> 00:34:51.825
as we move forward.
313
00:34:52.125 --> 00:34:54.785
That size of the models is growing pretty
314
00:34:55.165 --> 00:35:00.340
rapidly. So there are going to be people who just fall out and not be in the race anymore.
315
00:35:00.720 --> 00:35:03.540
But what it what is happening because of that is
316
00:35:03.840 --> 00:35:04.340
the
317
00:35:04.720 --> 00:35:05.220
efficiency
318
00:35:06.160 --> 00:35:08.260
for large language model training
319
00:35:08.720 --> 00:35:13.220
is constantly improving because now it's consolidated in one or two places
320
00:35:13.635 --> 00:35:18.215
or maybe five companies working on very large language models, but they all spend
321
00:35:18.755 --> 00:35:20.775
enormous amount of time optimizing
322
00:35:21.555 --> 00:35:23.175
infrastructure, the model architecture,
323
00:35:23.715 --> 00:35:24.375
the data
324
00:35:24.675 --> 00:35:33.870
flow, failures. Everything is actually now under scrutiny, and everybody is looking for even the half percent improvement in utilization and and and speed.
325
00:35:34.250 --> 00:35:39.610
We didn't have that when we started. So early twenty four, we were still seeing 30%
326
00:35:39.610 --> 00:35:40.110
utilization
327
00:35:40.570 --> 00:35:42.430
of very, very expensive resources.
328
00:35:42.984 --> 00:36:04.780
I'm already starting to see even outside of, like, say, I some improvements on that utilization factor. However, when we started, we were thinking training is actually going to be a a lot more ubiquitous thing than than what in the it ended up being. So we didn't really spend a lot of time on inference and, you know, model serving. But as as we spend time with our customers, we realize that
329
00:36:05.160 --> 00:36:08.780
model serving and inference is actually running through the same inefficiency.
330
00:36:09.285 --> 00:36:12.964
You know, as we were talking about people either book capacity to reach
331
00:36:13.605 --> 00:36:24.470
to hit the peak, and then they sit idle on that, or they're struggling with the response times during peak hours. And we realized that this concept of capacity versus
332
00:36:24.770 --> 00:36:25.270
utilization
333
00:36:25.730 --> 00:36:34.950
is very similar for even inference. And that's one of the things that we have evolved from the time we started to now. We now pay a lot more attention to
334
00:36:35.250 --> 00:36:36.150
smaller workloads,
335
00:36:36.450 --> 00:36:36.950
but
336
00:36:37.375 --> 00:36:52.780
the spikiness is pretty high. And we want we we divide the workloads in a way and run the agents in an in a way that it auto scales to make sure that we are maximizing the due utilization of the capacity that we have. There's an there's something else that that we also realized,
337
00:36:53.880 --> 00:36:56.540
which is the the focus on
338
00:36:57.160 --> 00:37:06.575
multiple architectures wasn't there when it was just the training that people cared about. When they were all building models, they all raised millions and millions and billions and billions of dollars
339
00:37:06.955 --> 00:37:11.215
and just, you know, focused on NVIDIA. But as as they are moving towards
340
00:37:11.595 --> 00:37:12.095
inference,
341
00:37:12.395 --> 00:37:13.295
we are seeing
342
00:37:13.675 --> 00:37:17.135
lot more opportunities of offering heterogeneous compute.
343
00:37:17.780 --> 00:37:20.840
And now it's not just about offering an API
344
00:37:21.540 --> 00:37:30.520
on, you know, a grok or a Cerberus, but and and, again, if you stay in that paradigm, then you deal with all sorts of complexities with the actual
345
00:37:30.945 --> 00:37:38.805
platform. You know, you will be on one architecture for training, then you move somewhere else, then you move somewhere else. For Flex AI, we just remain
346
00:37:39.185 --> 00:37:56.680
consistent with the platform. So your data doesn't have
to be copied over. You don't have to move anywhere to a different platform. Your experience remains the same. You start on one architecture. You end on another, but you feel that experience is one platform. And that's actually one of the advantages that we are bringing, which is one are one platform,
347
00:37:57.060 --> 00:38:00.280
multiple workloads, multiple clouds, multiple location,
348
00:38:00.845 --> 00:38:01.744
tailored and
349
00:38:02.204 --> 00:38:03.904
optimized for your needs.
350
00:38:04.924 --> 00:38:13.744
And so for people who are onboarding onto the Flex AI platform, I'm curious if you can talk to some of the interfaces
351
00:38:14.045 --> 00:38:14.865
and abstractions
352
00:38:15.540 --> 00:38:19.480
and workflow that they should be aware of and just the overall process
353
00:38:19.860 --> 00:38:30.040
of using Flex AI for going from, I have an idea that I want to experiment with to I am now serving this at production load in an auto scaled capacity.
354
00:38:31.395 --> 00:38:31.895
Absolutely.
355
00:38:32.835 --> 00:38:37.175
The first thing is, you know, I my initial thought was it should be
356
00:38:37.715 --> 00:38:42.935
as simple as, you know, one click to two clicks. Right? Make it a user interface,
357
00:38:43.475 --> 00:38:45.975
a GUI that is super clean. And
358
00:38:46.430 --> 00:38:50.210
anyone, even without an AI or ML background or
359
00:38:50.590 --> 00:38:57.490
infrastructure understanding, can can come in and deploy something. Obviously, we work with a very, you know, technical
360
00:38:57.790 --> 00:39:07.555
group of people. So, you know, a lot of them actually already have workflows and they run, you know, their own containers, their own, you know, virtual machines. We enable them through
361
00:39:07.855 --> 00:39:11.555
a CLI, command line interface. So we allow people to launch
362
00:39:11.855 --> 00:39:15.315
their workloads whether it's training, fine tuning, or inference
363
00:39:15.730 --> 00:39:32.655
using a simple command line for any of the models you can pick from the open source world, point the platform to your data, and we take care of the rest. None of that, setting up the infrastructure, defining it, you know, how to set up the networking and defining where to set up that storage and s three buckets.
364
00:39:33.355 --> 00:39:42.210
But we also simplified things for people who do not have that level of experience or have built AI applications before. We actually started something
365
00:39:42.590 --> 00:39:43.730
called blueprints recently,
366
00:39:44.110 --> 00:39:48.450
where if you are interested in building something like a smart search, voice transcription,
367
00:39:48.910 --> 00:39:49.890
multi cloud migration,
368
00:39:50.430 --> 00:40:00.865
media image playground, we just enable that by giving people a blueprint of how to build these things. So we pick some standard models, open source models from Hugging Face.
369
00:40:01.245 --> 00:40:15.750
We have some
dummy data or some reference data that that we have used from open source world, and we build some of these applications for them to use for their own use case. So now they can pick a different model instead of the four models that we we picked.
370
00:40:16.290 --> 00:40:23.990
They can bring their own data, which they can use for fine tuning on or rack. And then depending on what their applications is, they can actually have multimodal
371
00:40:24.365 --> 00:40:27.905
input and output. And all of that is driven by a single platform
372
00:40:28.285 --> 00:40:34.785
where these blueprints are already available. You can just go in and click click and change some things and get get exactly what you wanted.
373
00:40:35.645 --> 00:40:36.625
One of the
374
00:40:36.970 --> 00:40:43.710
perennial challenges for anybody who wants to build a platform as a service and provide a unified experience
375
00:40:44.010 --> 00:40:54.545
for their customers is that there are always going to be some edge cases or some assumptions baked into the platform that are contrary to the assumptions of the team who are building the solution.
376
00:40:55.085 --> 00:40:55.585
And
377
00:40:55.965 --> 00:40:57.665
so finding the appropriate
378
00:40:58.045 --> 00:41:09.230
abstractions and interfaces as well as providing escape hatches for those edge cases is a path towards success. And I'm wondering if you can just talk to some of the ways that you're thinking about addressing
379
00:41:09.530 --> 00:41:10.030
those
380
00:41:10.490 --> 00:41:10.990
affordances
381
00:41:11.450 --> 00:41:12.589
and escape hatches
382
00:41:12.970 --> 00:41:15.230
and the aspects of the overall
383
00:41:15.609 --> 00:41:16.109
AI
384
00:41:16.570 --> 00:41:23.445
deployment and development ecosystem that you're explicitly leaving out of scope in order to be able to focus on your core competency
385
00:41:24.145 --> 00:41:25.525
of infrastructure abstraction?
386
00:41:26.945 --> 00:41:30.545
Absolutely, Tobias. That's that's a great question. And if we realize that,
387
00:41:31.105 --> 00:41:34.650
very early on that the world is not going to be homogeneous
388
00:41:35.190 --> 00:41:37.290
even from a philosophy perspective.
389
00:41:37.670 --> 00:41:46.570
Every customer that we talked to initially had a very different view on what their workflow would be, and we had to make some decisions because, you know, you can't serve
390
00:41:46.905 --> 00:42:00.205
infinite number of combinations and permutations. So so we had to make some choices. We did some make some choices, and the objective was that we serve a pretty large majority of the market. And one segment that we explicitly
391
00:42:01.050 --> 00:42:04.590
said we we cannot or we shouldn't be targeting is
392
00:42:04.890 --> 00:42:07.070
someone who was already deeply expert
393
00:42:07.370 --> 00:42:19.085
in infrastructure management. And they were just about turning to become model people and and trying to build some models, but their background and and history was infrastructure management. The amount of custom
394
00:42:19.545 --> 00:42:21.165
code and micro optimizations
395
00:42:21.545 --> 00:42:34.309
they put in their flows is just impossible to manage and and support because these are sometimes their proprietary flows, and, you know, we we don't want to get in the middle of their business. Having said that, yeah, we we have now
396
00:42:34.609 --> 00:42:42.105
dealt with about fifty, sixty high quality users who, you know, all are looking to build models,
397
00:42:42.485 --> 00:42:45.465
train, fine tune, build multimodal outputs.
398
00:42:45.765 --> 00:42:59.410
And we have we can very proudly say that we now serve most of them, and we have road maps to serve the remaining ones with some of the new features that we are going to add. So you you talked about edge cases.
399
00:42:59.790 --> 00:43:04.690
We said there is going to be a catch all solution, and that's going to be containers.
400
00:43:05.070 --> 00:43:18.805
So our solution is you wanna run fine tuning or training or or inference. There is a very simple CLI. There is a very simple GUI web interface that you can go launch things with, and that should cover 90% of the world. But the other 10%,
401
00:43:18.945 --> 00:43:52.860
if they have some custom flows that we maybe, if it becomes important enough, we might actually make it part of our platform managed services offering. But for now, you can pack we can either help you package it in a container, and then now that becomes the workload. The container becomes a workload on our platform, and it asks for the resources it needs. It it basically runs in the right priority order and then gets out of the queue when it's done. But that means that anybody who's not fitting in the way we run our platform or provide the service, they can bring their own containers and make it part of our,
402
00:43:53.240 --> 00:43:55.260
scheduling system. It still
403
00:43:55.560 --> 00:44:12.474
can leave gaps and holes, and we are claiming that we are gonna leave that point 1% of the market because that's probably out of scope for us for now. I believe pretty much every platform has to make a choice like that, and no one can guarantee 100% coverage, and that's the choice we are making.
404
00:44:13.174 --> 00:44:14.635
And in your experience
405
00:44:15.015 --> 00:44:18.270
of building Flex AI, getting it in front of customers,
406
00:44:18.570 --> 00:44:25.950
helping them use it to its fullest effect, what are some of the most interesting or innovative or unexpected ways that you've seen the platform applied?
407
00:44:26.490 --> 00:44:29.390
Yeah. Yeah. So this there were many, many learnings.
408
00:44:29.770 --> 00:44:46.750
One of them was one of our customers was trying to actually get their model trained and get ready for a YC demo. And they were struggling to just get their infrastructure up and running and get the training going. They somehow
heard about us, and they reached out to us. And we got them up and running within
409
00:44:47.050 --> 00:44:51.870
a couple of hours, and they were able to actually set this up and and go do their demos.
410
00:44:52.490 --> 00:44:58.904
There were a couple of other examples where we didn't realize that we can actually solve a specific customer's
411
00:44:59.204 --> 00:44:59.704
issue
412
00:45:00.085 --> 00:45:06.025
by so this this is an example where somebody had a ton of data, one of the hyperscalers.
413
00:45:06.724 --> 00:45:08.585
And they were looking for compute
414
00:45:08.964 --> 00:45:25.470
in, you know, a cheaper location with with a new cloud. And we all were connected to the the same people, and and I knew where the pricing was going to be. It was, like, a dollar 50 compared to $6 on an hyperscaler. So the math was obvious. But the data movement
415
00:45:25.845 --> 00:45:34.105
meant that every time they had to pay the egress fees, which was, like, adding up to tens of thousands of dollars, the amount of data was large. Our architecture,
416
00:45:34.405 --> 00:45:37.545
even though we were designing it not for this case, our architecture
417
00:45:38.244 --> 00:45:40.825
allowed them to save the egress fees
418
00:45:41.190 --> 00:45:41.690
for
419
00:45:42.150 --> 00:45:42.810
any iterations.
420
00:45:43.270 --> 00:45:45.750
The first time we we put this in a cache,
421
00:45:46.230 --> 00:45:47.930
and the caching was done between,
422
00:45:48.390 --> 00:45:48.890
cloud,
423
00:45:49.430 --> 00:45:50.570
storage and,
424
00:45:51.190 --> 00:45:53.849
on node storage. But once the first
425
00:45:54.869 --> 00:46:20.260
step was done, there was no more egress fees. And that saved a ton of money for them, and and they were like, we would actually go with this solution because now we have access to all the compute in the world that is the right size, that is the right price, and that is the right capacity for us, and we don't have to worry about data movement or egress fees. There are many other examples that I'm definitely forgetting, but it just makes me happy to see some of the customers'
426
00:46:20.640 --> 00:46:31.055
use cases getting solved by the right thinking on our platform and the right solutions that we are building. And this is the reason why we started the company, which is pick a good problem to solve and solve it in
427
00:46:31.355 --> 00:46:33.535
a reasonable way, in a fast way.
428
00:46:34.234 --> 00:46:37.220
And in your experience of building the business,
429
00:46:37.760 --> 00:46:39.700
investing in the technology
430
00:46:40.320 --> 00:46:41.700
and the abstractions
431
00:46:42.400 --> 00:46:48.340
and building the business around it, what are some of the most interesting or unexpected or challenging lessons that you've learned personally?
432
00:46:49.255 --> 00:47:10.360
Interesting is when we first thought I I talked about this a little bit. When we first thought that we should build a managed service platform where, you know, we can, you know, reduce the number of permutation combination people deal with in terms of doing training or fine tuning. It was very clear that even if we were right, word was actually going to be a pretty
433
00:47:10.740 --> 00:47:11.240
fragmented
434
00:47:11.780 --> 00:47:56.815
word in terms of how many things people do, how people run their workflows, and and training. So we had to actually adapt very quickly on supporting more than what we were initially planning to support because it was more important for us to meet the customers where they were rather than the idea that we had initially. That was that was a pretty big learning for us, and that that happened very quickly. And I'm really, really glad that we were working with some of the, you know, best developers and and users who were very friendly and gave us support and helped us guide and nudge us in the right direction. That was very helpful. The second thing was, as we started deploying our first iteration of the product, we realized that people are not really excited about switching platforms.
435
00:47:57.275 --> 00:48:49.245
They they don't like being on AWS for one thing, on GCP for another thing, and new clouds and and other small cloud providers for some other stuff, even if there is a cost advantage. And it is basically our concept too, which is, you know, everybody needs a different setup, and nobody really has the patience or time to set up all these different environments and manage multi cloud and multi provider. So that translated for us as we cannot offer just one piece of the equation of their workflows and say, use it. This is the best you can get. And then have them go somewhere else to finish their task. It needed to be an end to end workflow support, and that's something that we are actually now enabling and getting tons of good feedback on it. So you don't have to get out of our platform as long as you're not developing some of the largest, largest models in the world. For example,
436
00:48:49.545 --> 00:48:55.725
x a I, you are our prime target. If you develop models, if you fine tune models, if you deploy
437
00:48:56.025 --> 00:48:59.085
agents, or if you have endpoints that you're deploying,
438
00:48:59.610 --> 00:49:02.350
Everything is served in just one unified platform.
439
00:49:02.970 --> 00:49:03.870
And you mentioned
440
00:49:04.250 --> 00:49:11.150
that on the high end, there's a certain level of scale at which it might not make sense to use Flex AI. I'm curious if there is a corresponding
441
00:49:12.075 --> 00:49:29.059
cohort on the other side of the spectrum where maybe there's a a point at which you're too small for Flex AI to be beneficial. I'm just wondering if you can talk to some of the ways that you think about sort of your ideal customer and give some guidance for people who are evaluating Flex AI for their own use cases?
442
00:49:29.920 --> 00:49:33.059
Actually, that the other end is
zero. We are
443
00:49:33.440 --> 00:49:35.779
so supportive of small developers
444
00:49:36.160 --> 00:49:40.955
and startups that, you know, we have no minimum required for on our platform.
445
00:49:41.335 --> 00:50:11.925
We offer on demand pricing. We offer credits. We offer ways to minimize your cost. So if you're looking for one GPU equivalent compute or even fractional GPU equivalent compute because we now support fractional GPUs on our platform, This is the right platform for you. So, yes, you know, let's say beyond 10,000 GPUs, you probably shouldn't be using a platform because you'd be trying to get every single ounce of this, you know, very expensive cluster and running a managed service on the top of that is probably not the best use of
446
00:50:12.805 --> 00:50:20.345
the resources. And, you know, when you are at that capacity, you're probably also focused on a single workload. You're not doing multiple things simultaneously
447
00:50:20.645 --> 00:50:29.369
because when people build models, they just, you know, build models for months. So below that, I I will say Flex AI covers everything
448
00:50:29.750 --> 00:50:37.875
under the sky. Once you get on the platform, you don't have to leave it, whether it's building it, optimizing it, tuning it, or deploying it.
449
00:50:38.595 --> 00:50:49.335
And as you continue to build and iterate on the Flex AI platform, what are some of the things you have planned for the near to medium term or any particular projects or problem areas you're excited to explore?
450
00:50:50.769 --> 00:51:20.215
Absolutely. I you know, some some of the things that we are very excited about is bringing new architectures on the platform. So, you know, we are working with TPUs. We are working with AMD. We're working with dense torrent. We will continue to work with more, you know, silicon providers. One of the exciting things that we are testing right now in our internal development is inference auto scaler. So we have a tool that we have developed that that is a simulator for the size or the capacity of the cluster needed for a specific throughput
451
00:51:20.630 --> 00:51:31.850
required for a given model. We're now applying that to start auto scaling the capacity for a given user or a given workload. And that's going to be the most cost
452
00:51:32.150 --> 00:51:38.475
optimized solution for people who have spiky workloads or spiky inference needs. It's going to deploy
453
00:51:38.775 --> 00:51:40.715
new nodes and new GPUs
454
00:51:41.095 --> 00:51:41.995
within seconds
455
00:51:42.295 --> 00:51:50.315
of seeing extra traffic or higher traffic come through. As we were talking about before, I'm also very excited about deploying some blueprints.
456
00:51:50.750 --> 00:51:52.530
The ability to actually
457
00:51:53.069 --> 00:52:01.490
translate your vision and idea into a real product. You know, I'm I'm super impressed by Lovable. You have an idea for a for an app or a website.
458
00:52:01.790 --> 00:52:08.285
I'm able to now implement that in less than an hour or a day. That's exactly what I want to do
459
00:52:08.665 --> 00:52:11.885
with AI applications. You have an idea or a concept.
460
00:52:12.265 --> 00:52:17.725
You come on our platform, and you should be able to just use our widgets and and platform
461
00:52:18.170 --> 00:52:28.270
and just deploy it within hours or a day. That's actually very, very, very exciting. Now all of that can be summarized into one thing. I would like my fellow
462
00:52:28.650 --> 00:52:29.550
start up founders
463
00:52:29.930 --> 00:52:35.555
to focus on what they started the company for and leave the DevOps and the infrastructure
464
00:52:35.855 --> 00:52:45.555
management to us so they can get the most value out of their their teams and the money they raised. We can help them with providing a seamless interface
465
00:52:45.855 --> 00:52:46.915
to a clean infrastructure.
466
00:52:48.000 --> 00:52:52.900
Are there any other aspects of the work that you're doing at Flex AI or the overall
467
00:52:53.280 --> 00:52:53.780
complexities
468
00:52:54.080 --> 00:52:57.460
and points of friction related to the infrastructure
469
00:52:57.760 --> 00:53:02.415
for training and inference that we didn't discuss yet that you'd like to cover before we close out the show?
470
00:53:02.815 --> 00:53:07.715
I think we covered pretty much everything. My summary of what we are doing is
471
00:53:08.095 --> 00:53:16.915
I I wanna make sure that people who are starting to develop new applications come up with new ideas on using AI to solve some real world problems.
472
00:53:17.420 --> 00:53:22.160
Actually, focus on that, and let us take care of the the back end of it, the infrastructures
473
00:53:22.460 --> 00:53:29.600
side of it. The the customers are recognizing our value. They are already, you know, using our platform and saving time.
474
00:53:30.055 --> 00:53:34.475
In my mind, the success metric for us is going to be when our users
475
00:53:35.015 --> 00:53:44.670
can claim that we haven't dealt with infrastructure issues in the last so many months that we have been working with Flexiad. And that's going to be a success for us.
476
00:53:45.150 --> 00:54:00.645
Alright. Well, for anybody who wants to get in touch with you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get your perspective on what you see as being the biggest gaps in the tooling technology or human training that's available for AI systems
477
00:54:01.365 --> 00:54:09.705
today? I think I think this this this is a complex question with a very simple answer. My take is if you are not using Flex AI,
478
00:54:10.005 --> 00:54:11.305
then you're actually
479
00:54:11.845 --> 00:54:16.570
building a team of very smart DevOps people, very smart DevOps engineers.
480
00:54:16.950 --> 00:54:25.210
And if that was the purpose of starting your business, starting your company, then you're doing the right thing. However, if you want to focus on the workloads
481
00:54:25.685 --> 00:54:35.065
and the problems that you're really solving, then your answer is going to be Flex AI. And the long answer of that is every single setup, every single infrastructure
482
00:54:35.765 --> 00:54:36.665
setup requires
483
00:54:37.045 --> 00:54:38.744
a very different set of expertise,
484
00:54:39.045 --> 00:54:39.545
knowledge,
485
00:54:39.910 --> 00:54:41.290
and pinpoint resolution.
486
00:54:41.590 --> 00:55:18.490
A lot of people deal with just setting up the infrastructure, maintaining it, keeping it up to date. Right? I mean, which version of PyTorch and which version of CUDA should I actually use for this application? What is the model optimized for versus what is the quantization level that I have to set? Those are two different problems. If you're focused on quantization, just focus on that instead of dealing with which PyTorch version we should be dealing with. And that's actually lacking in the industry. Most people actually are learning on the job, and I would do I we just want to stop doing that. We want people to spend time on developing their applications and let us take care of the infrastructure piece.
487
00:55:19.190 --> 00:55:31.155
Alright. Well, thank you very much for taking the time today to join me and share the work that you and your team are doing on Flex AI and giving some insights into the complexities of dealing with hardware and optimization
488
00:55:31.615 --> 00:55:44.145
of throughput for managing these AI applications. I appreciate all the time and energy that you folks are putting into making that a solved problem for the rest of us, and I hope you enjoy the rest of your day. You do the best. Thank you.
489
00:55:51.425 --> 00:55:55.745
Thank you for listening, and don't forget to check out our other shows. Podcast.net
490
00:55:55.745 --> 00:56:04.630
covers the Python language, its community, and the innovative ways it is being used. And the AI Engineering podcast is your guide to the fast moving world of building AI systems.
491
00:56:05.330 --> 00:56:15.575
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a project from the show, then tell us about it. Email hosts@dataengineeringpodcast.com
492
00:56:15.575 --> 00:56:16.315
with your story.
493
00:56:16.615 --> 00:56:21.755
Just to help other people find the show, please leave a review on Apple Podcasts and tell your friends and colleagues.