TRANSCRIPTIE 🔗
1
00:00:04,673 --> 00:00:07,823
Kubernetes Pod Live Migration.
2
00:00:08,063 --> 00:00:12,353
That's what Cast AI calls it when they
dynamically move your pod to a different
3
00:00:12,353 --> 00:00:14,873
node without downtime or data loss.
4
00:00:15,293 --> 00:00:19,193
They've built a Kubernetes controller
that works with CSI storage and CNI
5
00:00:19,193 --> 00:00:24,173
network plugins to copy your running
pod data, the memory, the IP address,
6
00:00:24,173 --> 00:00:28,553
and the TCP connections from one
node to another in real time Welcome
7
00:00:28,553 --> 00:00:32,093
to DevOps and Docker talk, and I'm
your solo host today, Bret Fisher.
8
00:00:32,453 --> 00:00:36,143
This YouTube live stream was a fun one
because I got to nerd out with engineers
9
00:00:36,413 --> 00:00:42,143
Philip Andrews and Dan Muret from Cast
AI on Kubernetes pod live migrations
10
00:00:42,203 --> 00:00:43,913
and how it works under the hood.
11
00:00:44,363 --> 00:00:47,573
We talk about use cases for
this feature, including hardware
12
00:00:47,573 --> 00:00:49,403
or OS maintenance on a node.
13
00:00:49,823 --> 00:00:55,403
Maybe for right sizing or bin packing your
pods for cost savings or moving off of a
14
00:00:55,403 --> 00:00:57,068
spot instance that's about to shut down.
15
00:00:57,848 --> 00:01:01,808
Or really anytime you need to move a
daemon set that would cause an outage if
16
00:01:01,808 --> 00:01:04,568
the pod had to restart or be redeployed.
17
00:01:05,068 --> 00:01:10,558
I don't know of any other turnkey way
to do this on Kubernetes today, but
18
00:01:10,558 --> 00:01:13,588
I've got a feeling that Cast AI has got
a winning feature on their hands, and
19
00:01:13,588 --> 00:01:15,658
I'm glad that we got to dig into it.
20
00:01:16,108 --> 00:01:20,458
Over 20 years ago, the virtual machine
vendors built this live migration feature
21
00:01:20,458 --> 00:01:27,088
into their products, and finally, in 2025,
we're now able to do that in Kubernetes.
22
00:01:27,588 --> 00:01:28,998
Let's get into the episode.
23
00:01:31,023 --> 00:01:31,743
Welcome to the show.
24
00:01:32,024 --> 00:01:35,134
All right, Both of these
gentlemen are from Cast AI.
25
00:01:35,344 --> 00:01:38,464
Philip is the global
field, CTO at Cast AI.
26
00:01:38,514 --> 00:01:40,524
What exactly does Global CTO do?
27
00:01:40,962 --> 00:01:42,762
handling, a lot of our large customers.
28
00:01:42,762 --> 00:01:46,052
a lot of the strategic partnerships,
usually new technologies, so working
29
00:01:46,052 --> 00:01:49,822
with a lot of our, you know, customers on
showing them new technologies, helping in,
30
00:01:49,912 --> 00:01:51,472
proof of concepts with new technologies.
31
00:01:51,932 --> 00:01:53,192
it's been kind of a cool role.
32
00:01:53,472 --> 00:01:56,262
basically on the technical
customer facing side.
33
00:01:56,572 --> 00:01:58,762
I get to do a lot with our
largest customers in solving
34
00:01:58,762 -->
Load full transcript
00:02:00,172
some of the the hardest problems.
35
00:02:00,672 --> 00:02:01,212
Nice.
36
00:02:01,722 --> 00:02:03,852
when you don't know what the
title is, it makes it sound like
37
00:02:03,852 --> 00:02:04,992
you just live in an airplane.
38
00:02:05,182 --> 00:02:06,022
it sounds impressive.
39
00:02:06,122 --> 00:02:08,692
and we've got Dan Muret or Muret.
40
00:02:09,022 --> 00:02:12,372
I really don't know my French, so that's
probably, a horrible pronunciation.
41
00:02:12,872 --> 00:02:15,962
Dan is here, he's the senior sales
engineer with CAST AI or one of them.
42
00:02:15,962 --> 00:02:20,162
I'm gonna make you the sales senior sales
engineer so that you sound very elite.
43
00:02:20,222 --> 00:02:20,832
Uh, welcome Dan.
44
00:02:21,332 --> 00:02:24,962
who could tell me about what's the
elevator pitch for Cast AI because
45
00:02:24,962 --> 00:02:26,312
I've known about you all for years.
46
00:02:26,612 --> 00:02:30,482
I've visited your booth at KubeCon at
least a half a dozen times over the years.
47
00:02:30,732 --> 00:02:32,982
what's the main benefit of Cast AI?
48
00:02:33,212 --> 00:02:34,112
Sure I can grab that one.
49
00:02:34,162 --> 00:02:37,342
when Cast AI was founded, it was to
solve a problem that our founders
50
00:02:37,342 --> 00:02:41,692
had in their last startup years ago,
which was every month their AWS bill
51
00:02:41,692 --> 00:02:45,662
was going up 10% regardless of what
they did the month before to try to.
52
00:02:45,922 --> 00:02:48,682
Mitigate and manage costs
across the in infrastructure.
53
00:02:49,232 --> 00:02:52,452
they sold that startup to Oracle, when
they finished that Oracle time, they went
54
00:02:52,452 --> 00:02:54,252
and figured out how to solve this problem.
55
00:02:54,252 --> 00:02:57,272
They said, well, Kubernetes is
gonna be the future platform.
56
00:02:57,332 --> 00:02:58,892
We're gonna make our bet on Kubernetes.
57
00:02:59,192 --> 00:03:01,862
And the only way to solve this
problem is through automation.
58
00:03:01,922 --> 00:03:04,712
Because doing things manually
every month, one, it's tedious
59
00:03:04,712 --> 00:03:05,882
and takes up a lot of time.
60
00:03:06,002 --> 00:03:08,242
And two, it's just, not helpful, right?
61
00:03:08,242 --> 00:03:10,762
You save a little bit every,
you know, it's two steps back
62
00:03:10,762 --> 00:03:11,962
for every step forward you make.
63
00:03:12,462 --> 00:03:15,852
With Cast AI, it's fully
automation first, right?
64
00:03:15,852 --> 00:03:18,942
We made the effort that everything was
going to be automated from the start.
65
00:03:19,282 --> 00:03:22,492
when it comes to node auto scaling,
node selection, node, right sizing,
66
00:03:22,492 --> 00:03:25,552
workload, right sizing, everything
to do with Kubernetes, everything
67
00:03:25,552 --> 00:03:27,472
we implement is automated.
68
00:03:27,922 --> 00:03:31,312
And that's where the live migration
piece came in being able to automatically
69
00:03:31,312 --> 00:03:35,392
move applications around within the
cluster without having downtime.
70
00:03:35,442 --> 00:03:37,782
and that's where application
performance automation comes
71
00:03:37,782 --> 00:03:41,572
in, Moving from this application
performance monitoring mindset.
72
00:03:42,361 --> 00:03:43,831
Datadog has made a lot of money on that.
73
00:03:44,331 --> 00:03:45,291
I dunno if I'm allowed to say that.
74
00:03:45,601 --> 00:03:48,191
but they've done very well
and have a fantastic platform.
75
00:03:48,221 --> 00:03:51,891
we love Datadog, but
you get data overload.
76
00:03:52,011 --> 00:03:55,701
You end up with metric overload and
actioning on those is very hard.
77
00:03:56,201 --> 00:03:59,321
Where we need to go from here,
especially with the AI mindset that
78
00:03:59,321 --> 00:04:02,801
we're moving into, is automation
of that application performance.
79
00:04:02,801 --> 00:04:04,361
And that's what Cast AI
is leading the way in.
80
00:04:04,861 --> 00:04:08,931
Nice, when you all reached out and I
learned about the fact that you now have
81
00:04:08,991 --> 00:04:13,701
live migration, it took me back to almost
25 years ago when that was first invented
82
00:04:13,701 --> 00:04:17,151
for VMs at the time it felt like magic.
83
00:04:17,181 --> 00:04:18,651
It did not seem real.
84
00:04:18,981 --> 00:04:22,971
We all had to try it to believe
it because it seemed impossible.
85
00:04:23,376 --> 00:04:27,226
To, move from one host to
another, maintain the ip,
86
00:04:27,226 --> 00:04:28,816
maintain the TCP connections.
87
00:04:28,946 --> 00:04:31,706
surely I'm gonna freeze up and
it's gonna be like a frozen screen.
88
00:04:31,836 --> 00:04:32,916
we all just assumed that.
89
00:04:32,916 --> 00:04:35,796
eventually, and maybe at first it
was a little hiccupy, I think, if I
90
00:04:35,796 --> 00:04:38,046
remember correctly, like 2003, 2005.
91
00:04:38,046 --> 00:04:40,926
It was one of those where it wasn't
quite live and they didn't, it was
92
00:04:40,926 --> 00:04:42,426
like very short amounts of gaps.
93
00:04:42,426 --> 00:04:44,196
then eventually it got good
enough that it was live.
94
00:04:44,596 --> 00:04:47,276
I was running data centers, for
local governments at the time.
95
00:04:47,276 --> 00:04:48,536
So I was very interested in this.
96
00:04:48,536 --> 00:04:50,696
'cause we were running
both ESX and HyperV.
97
00:04:50,696 --> 00:04:53,776
So I was heavily invested in
that feature and functionality.
98
00:04:54,076 --> 00:04:57,366
So when I saw that you were doing
it to Kubernetes, my first thought
99
00:04:57,366 --> 00:04:59,196
was, why did this take so long?
100
00:04:59,696 --> 00:05:01,676
why don't we have this yet on everything?
101
00:05:02,066 --> 00:05:04,706
Because it's clearly possible,
it's technically possible.
102
00:05:05,066 --> 00:05:08,696
Obviously, it's not super easy and
requires a lot of low level tooling
103
00:05:08,696 --> 00:05:13,456
that has to understand networking
and memory and you know, dis disc
104
00:05:13,456 --> 00:05:15,256
rights and all that kind of stuff.
105
00:05:15,256 --> 00:05:18,946
So I'm excited for us to get
into exactly how this sort of
106
00:05:19,036 --> 00:05:20,956
operates for a Kubernetes admin.
107
00:05:20,956 --> 00:05:24,406
And I really feel like this show's gonna
be great for anyone learning Kubernetes or
108
00:05:24,436 --> 00:05:29,196
Kubernetes admins, because to talk about
the stateful set, the daemon set problem
109
00:05:29,256 --> 00:05:33,026
of, we've got stateful work, everyone's
got, I mean, everyone I know almost
110
00:05:33,026 --> 00:05:34,526
has stateful workflows in Kubernetes.
111
00:05:34,526 --> 00:05:37,256
I would say, I don't know about you
all's experience, but to me it's
112
00:05:37,256 --> 00:05:40,196
an exception when everything is
stateless in Kubernetes nowadays.
113
00:05:40,196 --> 00:05:41,526
Do you find that to be the case
114
00:05:42,066 --> 00:05:43,476
we talk to customers all the time, right.
115
00:05:43,476 --> 00:05:44,316
And yeah.
116
00:05:44,316 --> 00:05:47,436
Used to, you know, a few, just
a couple years ago, right.
117
00:05:47,436 --> 00:05:49,326
It was a lot more stateless.
118
00:05:49,326 --> 00:05:50,646
Web servers, whatever.
119
00:05:51,016 --> 00:05:54,616
now we're definitely seeing a shift
to, more stateful workloads whether
120
00:05:54,616 --> 00:05:58,326
it's, legacy applications being
forced into Kubernetes as part of a
121
00:05:58,326 --> 00:06:00,006
modernization project or whatever.
122
00:06:00,006 --> 00:06:03,576
We're seeing a lot more stateful
workloads in Kubernetes for sure.
123
00:06:03,819 --> 00:06:06,622
Particularly amongst the Fortune one
hundreds, fortune five hundreds, right?
124
00:06:06,622 --> 00:06:10,712
'cause you've got this modernization,
and I put it in quotes, where
125
00:06:10,712 --> 00:06:14,762
modernization means taking some
crusty old 15-year-old application,
126
00:06:15,182 --> 00:06:18,752
containerizing it and shoving it in
Kubernetes and calling it cloud native,
127
00:06:18,802 --> 00:06:20,632
the flawed approach of lift and shift.
128
00:06:21,112 --> 00:06:23,962
and you end up with a lot of
applications that are in Kubernetes.
129
00:06:23,962 --> 00:06:27,862
They're listed as a deployment, but
you can't restart them without your
130
00:06:27,862 --> 00:06:29,422
customer having a significant outage.
131
00:06:30,092 --> 00:06:32,342
it goes against everything
Kubernetes was built on.
132
00:06:32,342 --> 00:06:33,922
But that's the world we live in today.
133
00:06:33,922 --> 00:06:36,712
when we first launched Live migration,
and I posted about it on LinkedIn,
134
00:06:37,212 --> 00:06:39,552
some of the first questions I
got was, why is this even needed?
135
00:06:39,612 --> 00:06:43,362
If you're doing Kubernetes correctly, live
migration shouldn't even be
a real thing.
136
00:06:43,412 --> 00:06:46,832
yes, but 95% of the customers I deal with.
137
00:06:47,177 --> 00:06:49,127
Don't do Kubernetes, the right way.
138
00:06:49,627 --> 00:06:53,787
Well, yeah, I honestly think we
could argue that's a great point.
139
00:06:53,787 --> 00:06:56,247
when Docker and Kubernetes were
both created, it was like stateless.
140
00:06:56,367 --> 00:06:58,157
'cause it's easy, you know,
move everything around.
141
00:06:58,157 --> 00:06:58,757
It's wonderful.
142
00:06:59,057 --> 00:07:03,427
But I think that to, I mean, this channel,
if there's anything consistent about this
143
00:07:03,427 --> 00:07:07,507
channel over the almost decade that has
existed, it's that I, it's all containers.
144
00:07:07,507 --> 00:07:10,357
Like I don't care what the tool
is, we're doing it in containers.
145
00:07:10,607 --> 00:07:14,987
the large success of containers is
because we could put everything in it.
146
00:07:14,987 --> 00:07:15,557
so many.
147
00:07:16,007 --> 00:07:19,997
Evolutions or attempted evolutions
in tech have been, well, you're
148
00:07:19,997 --> 00:07:23,357
gonna have to rewrite to, you know,
functionless, you know, or functions.
149
00:07:23,387 --> 00:07:26,057
you're serverless, you're gonna have to
write functions now or you're gonna have
150
00:07:26,057 --> 00:07:27,977
to rewrite this language or whatever.
151
00:07:28,237 --> 00:07:31,817
and that's, I think that's like the
secret sauce of containers was that we
152
00:07:31,817 --> 00:07:33,257
could literally shove everything in it.
153
00:07:33,347 --> 00:07:34,667
It's also the negative.
154
00:07:34,697 --> 00:07:37,077
And so there's, there's a thing that,
I don't know if I learned it from a
155
00:07:37,077 --> 00:07:41,097
therapist or whatever, but often our
weaknesses are just overdone strengths.
156
00:07:41,097 --> 00:07:43,947
And I feel like the strength of containers
is that you can do everything with it.
157
00:07:43,947 --> 00:07:46,797
you can put every known
app on the planet in it.
158
00:07:47,067 --> 00:07:48,987
They will eventually work
if you figure it out.
159
00:07:49,347 --> 00:07:52,647
the overdone weakness is that we're
putting everything in there which makes
160
00:07:53,007 --> 00:07:54,967
managing these infrastructures very hard.
161
00:07:54,997 --> 00:07:58,117
you have to assume everything's
fragile Until you are sure
162
00:07:58,117 --> 00:07:59,797
that it's truly stateless.
163
00:07:59,927 --> 00:08:04,247
even stateless, people say stateless
and what they really mean is it's, it
164
00:08:04,247 --> 00:08:08,587
doesn't care about disc, but it definitely
cares about connections, which is when
165
00:08:08,587 --> 00:08:11,197
we're trying to talk about stateless,
that's not technically accurate.
166
00:08:11,197 --> 00:08:13,627
Like when we say stateless,
we should probably mean also
167
00:08:13,627 --> 00:08:15,277
doesn't care about connections.
168
00:08:15,617
--> 00:08:17,177
at least once the connections are drained.
169
00:08:17,207 --> 00:08:19,937
it's an interesting dilemma we all have
in the infrastructure of we have the
170
00:08:19,937 --> 00:08:23,297
power to be able to move everything and
do everything, but also everything's
171
00:08:23,297 --> 00:08:26,507
super fragile that we're running at the
same time, so how do we even manage that?
172
00:08:26,507 --> 00:08:29,567
We've encountered a lot of teams
that, swore up and down they were
173
00:08:29,567 --> 00:08:32,257
stateless, right up until you
started bin packing their cluster.
174
00:08:32,757 --> 00:08:33,597
They said, wait, wait, wait.
175
00:08:33,647 --> 00:08:35,177
why are we having all
these restarted pods?
176
00:08:35,237 --> 00:08:37,907
We're like, because we're bin
packing and we're moving things, and
177
00:08:37,907 --> 00:08:39,287
we're getting better, optimization.
178
00:08:39,287 --> 00:08:41,147
They're like, but my container restarted.
179
00:08:41,207 --> 00:08:43,727
Well, yes, that's what
containers do in Kubernetes.
180
00:08:45,296 --> 00:08:45,626
Right.
181
00:08:45,626 --> 00:08:49,476
And for those that are maybe just getting
into Kubernetes or haven't dealt with
182
00:08:49,476 --> 00:08:53,181
large enterprise forever workloads
where they just can't be touched.
183
00:08:53,181 --> 00:08:55,491
I've had 30 years in tech
of don't touch that server.
184
00:08:55,491 --> 00:08:56,511
don't touch that workload.
185
00:08:56,511 --> 00:08:59,121
It's fragile, it's precious, but
it's also probably on the oldest
186
00:08:59,121 --> 00:09:00,501
hardware in the least maintained.
187
00:09:00,951 --> 00:09:05,451
so the idea of, one of the performance
measures that any significant size
188
00:09:05,451 --> 00:09:07,821
Kubernetes team is dealing with
is the cost of infrastructure.
189
00:09:08,091 --> 00:09:10,971
And then we keep getting told, I think
this was just in the last year at
190
00:09:10,971 --> 00:09:17,391
KubeCon, that even on top of Kubernetes,
we're still only averaging like 10% CPU
191
00:09:17,391 --> 00:09:19,401
on average utilization across nodes.
192
00:09:19,611 --> 00:09:23,661
Like we, we still are struggling with
the same infrastructure problems that
193
00:09:23,661 --> 00:09:28,161
we were dealing with the last 30 years,
even before VMs, before virtualization.
194
00:09:28,521 --> 00:09:31,521
That was the same problem we had then
because everybody would want their
195
00:09:31,521 --> 00:09:35,811
own server and they always had to plan
for the worst busiest day of the year.
196
00:09:35,811 --> 00:09:38,731
So they would buy huge servers,
put 'em in, and they'd sit idle
197
00:09:38,731 --> 00:09:41,951
almost all the time because
they barely got 5%, utilization.
198
00:09:42,251 --> 00:09:47,421
So I can see where like one of the core
premises of something like a application
199
00:09:47,421 --> 00:09:52,701
performance tool is that we're gonna
save tons of money by bin packing.
200
00:09:52,701 --> 00:09:55,481
can you explain the bin packing process?
201
00:09:55,481 --> 00:09:56,081
what does that look like?
202
00:09:56,591 --> 00:10:01,261
so one of the big things with Kubernetes
is the scheduler will typically
203
00:10:01,261 --> 00:10:03,211
round robin, assign pods to nodes.
204
00:10:03,281 --> 00:10:06,941
if you have 10 nodes in a cluster,
your pods will more or less get evenly
205
00:10:06,941 --> 00:10:08,501
distributed to those nodes in the cluster.
206
00:10:08,501 --> 00:10:11,261
You can manage that with certain
different scheduler hints and
207
00:10:11,261 --> 00:10:15,021
certain scheduler, suggestions to
steer that towards, you know, most
208
00:10:15,021 --> 00:10:16,431
utilized, least utilized, et cetera.
209
00:10:16,431 --> 00:10:18,531
But at the end of the day,
you're gonna have spread out
210
00:10:18,531 --> 00:10:19,671
workloads across your nodes.
211
00:10:20,171 --> 00:10:24,491
Bin packing is basically the
defragmentation of Kubernetes right back.
212
00:10:24,571 --> 00:10:27,041
You know, Bret, when you were first
starting out, when I was first starting
213
00:10:27,041 --> 00:10:29,771
out, and you could actually defragment
a hard drive and you get to move the
214
00:10:29,771 --> 00:10:31,421
little Tetris blocks around the screen,
215
00:10:31,621 --> 00:10:31,831
days.
216
00:10:32,177 --> 00:10:35,687
being able to do that in a Kubernetes
cluster can mean massive, massive
217
00:10:35,687 --> 00:10:39,167
savings on the actual utilization of that
cluster because now you free up a bunch
218
00:10:39,167 --> 00:10:42,707
of workloads, a bunch of nodes in the
cluster that are no longer necessary,
219
00:10:42,707 --> 00:10:45,617
you can delete those off and when you
need them, you just add them back.
220
00:10:45,617 --> 00:10:49,027
that's the joy of being in a cloud
environment, you can use the least amount
221
00:10:49,027 --> 00:10:50,287
of resources when you don't need 'em.
222
00:10:50,347 --> 00:10:53,257
So in, for instance, at, you know,
your off busy hours, your nighttime
223
00:10:53,257 --> 00:10:57,127
hours, and then when you start needing
'em again, you spin 'em up, you add
224
00:10:57,127 --> 00:11:01,507
more to it, you scale up during the
day, and being able to do that process
225
00:11:01,507 --> 00:11:05,377
over and over again every day is how
you can optimize your cloud resources.
226
00:11:05,877 --> 00:11:06,867
What we see is that.
227
00:11:07,367 --> 00:11:11,827
People have so many stateful workloads
whether it's stateful in real state or
228
00:11:11,827 --> 00:11:15,697
stateful in, this is a really poorly
architected application, or stateful in
229
00:11:15,697 --> 00:11:20,107
this application takes 15 minutes to start
up and I, it's a monolithic and I can only
230
00:11:20,107 --> 00:11:23,167
run one copy of it so I can't move it.
231
00:11:23,497 --> 00:11:27,427
All of those
things causes, so you
can't bin pack a cluster, right?
232
00:11:27,427 --> 00:11:29,077
You can't move those things around.
233
00:11:29,167 --> 00:11:33,967
So what ends up happening is people just
end up with these stateful workloads
234
00:11:33,967 --> 00:11:35,767
scattered throughout all 10 nodes.
235
00:11:36,187 --> 00:11:39,877
And even if the 10 nodes are only
60% utilized, you can't get rid
236
00:11:39,877 --> 00:11:42,577
of any of them because it'll cause
some kind of service interruption.
237
00:11:43,077 --> 00:11:45,867
And that's where live migration
allows you to move those
238
00:11:45,867 --> 00:11:47,487
stateful sensitive workloads.
239
00:11:47,717 --> 00:11:52,327
So now those 10 nodes can go down
to six or seven nodes without having
240
00:11:52,327 --> 00:11:56,377
a service interruption, even if
there's less than ideal workloads
241
00:11:56,377 --> 00:11:57,547
scattered throughout the cluster.
242
00:11:58,057 --> 00:12:05,357
stateful versus stateless versus like,
where's the scenario for where we need
243
00:12:05,357 --> 00:12:11,497
a live pod migration and like to those
that are perfect in all their software
244
00:12:11,497 --> 00:12:15,007
and they run and they control all
the software that runs on Kubernetes.
245
00:12:15,187 --> 00:12:17,587
I don't know who those people are,
but let's just say they exist.
246
00:12:18,087 --> 00:12:19,347
Then this isn't needed.
247
00:12:19,527 --> 00:12:22,617
Every database has a replica
or database mirror, so you
248
00:12:22,617 --> 00:12:23,967
can always take a node down.
249
00:12:23,967 --> 00:12:29,897
Every pod properly has proper
shutdown for ensuring that connections
250
00:12:29,897 --> 00:12:31,937
are properly moved to a new pod.
251
00:12:32,057 --> 00:12:36,357
By the way, I used to do a whole
conference talk on TCP packets
252
00:12:36,357 --> 00:12:39,717
and, resetting the connection to
make sure it moves properly through
253
00:12:39,717 --> 00:12:42,547
the load balancer to the next one,
and having a long shutdown time
254
00:12:42,547 --> 00:12:43,897
so that you can drain connections.
255
00:12:43,947 --> 00:12:48,207
that world of shutting down a pod is so
more complicated than anyone gives it.
256
00:12:48,267 --> 00:12:50,937
everyone treats it like it's casual
and easy, and it's just not if
257
00:12:50,937 --> 00:12:53,957
you're dealing with, hundreds of
thousands or millions of connections.
258
00:12:54,227 --> 00:12:56,387
there is a lot of nuance
and detail to this.
259
00:12:56,387 --> 00:12:59,307
and I often end up with teams
where they implement Kubernetes.
260
00:12:59,667 --> 00:13:01,287
It's a sort of predictable pattern, right?
261
00:13:01,287 --> 00:13:03,317
They implement Kubernetes,
move workloads to it.
262
00:13:03,817 --> 00:13:07,297
They think Kubernetes gives their
workloads magic and then they just
263
00:13:07,297 --> 00:13:09,787
start trying moving things around
and they realize when
their customers
264
00:13:09,787 --> 00:13:14,937
complain that, The rules of TCP IPS
load balancers and connection state,
265
00:13:14,937 --> 00:13:16,407
like all these rules still apply.
266
00:13:16,687 --> 00:13:20,347
you have to understand those lower levels
and obviously disc and writing to disc
267
00:13:20,347 --> 00:13:21,637
and logs for databases and all that stuff.
268
00:13:21,637 --> 00:13:22,807
that's still there too, I think.
269
00:13:23,077 --> 00:13:26,437
I think the networking is where
I see a lot of junior engineers
270
00:13:26,837 --> 00:13:30,697
hand waving over it because quite
honestly, the cloud has made a lot
271
00:13:30,697 --> 00:13:32,017
of the networking problems go away.
272
00:13:32,017 --> 00:13:34,447
So we don't have to have
Cisco certifications just
273
00:13:34,447 --> 00:13:36,067
to run servers anymore.
274
00:13:36,067 --> 00:13:40,387
We used to, but now we can get away
with it till a certain point, in
275
00:13:40,387 --> 00:13:42,757
the career or complexity level.
276
00:13:42,847 --> 00:13:47,897
And then suddenly you're having to really
understand the difference between TCP and
277
00:13:47,897 --> 00:13:53,387
UDP and how session state long polling or
web sockets, how all these things affect.
278
00:13:53,887 --> 00:13:57,447
Whether you're going to break
customers when you decide to restart
279
00:13:57,447 --> 00:13:59,457
that pod or redeploy a new pod.
280
00:13:59,797 --> 00:14:03,247
I love that stuff because it's super
technical and you can get really into
281
00:14:03,247 --> 00:14:06,657
the weeds of it, and it's not, I wouldn't
call a solved problem for everyone.
282
00:14:06,907 --> 00:14:09,397
my understanding of something
like a live migration is,
283
00:14:09,397 --> 00:14:11,197
takes most of those concerns.
284
00:14:11,627 --> 00:14:15,377
it doesn't make them irrelevant, but
it does deal with those concerns.
285
00:14:15,377 --> 00:14:15,857
am I right?
286
00:14:15,857 --> 00:14:18,787
in terms of networking we're talking
about live migrations, having
287
00:14:18,787 --> 00:14:22,352
to be concerned with ips and,
connection state, stuff like that.
288
00:14:22,925 --> 00:14:26,585
yeah, so with being able to do the
networking move of things, to your
289
00:14:26,585 --> 00:14:29,165
point, reestablishing those sessions.
290
00:14:29,195 --> 00:14:32,075
one of the big things we
see is long running jobs.
291
00:14:32,125 --> 00:14:35,305
if you've got a job that's running for
eight hours and it gets interrupted
292
00:14:35,305 --> 00:14:36,535
at six, you've lost that job.
293
00:14:37,035 --> 00:14:41,675
Even if you try to, move it from one to,
there's some checkpointing involved, a
294
00:14:41,675 --> 00:14:45,875
lot of times, like on a spark workload,
the driver will just kill the pod
295
00:14:45,875 --> 00:14:48,725
and restart it if it senses any kind
of interruption in the networking.
296
00:14:49,225 --> 00:14:50,935
So the networking's super important there.
297
00:14:50,935 --> 00:14:51,805
Being able to maintain that.
298
00:14:52,735 --> 00:14:54,235
Long running sessions, web sockets.
299
00:14:54,235 --> 00:14:57,055
To your point, we've actually tested
this extensively with web sockets.
300
00:14:57,325 --> 00:14:58,585
Web sockets, stay connected.
301
00:14:59,085 --> 00:15:03,125
And we're still in those
early vMotion days.
302
00:15:03,175 --> 00:15:05,125
there is a slight pause
when we move things.
303
00:15:05,405 --> 00:15:09,065
it took vMotion multiple years before
they got it kind of really ironed out.
304
00:15:09,565 --> 00:15:11,395
We're moving probably faster
to them 'cause we have a lot
305
00:15:11,395 --> 00:15:14,305
of experience, you know, of the
experiences they went through.
306
00:15:14,585 --> 00:15:16,385
and the research that's
happened since then.
307
00:15:16,385 --> 00:15:18,875
So I think we're moving pretty fast
on shortening that time window.
308
00:15:19,215 --> 00:15:23,865
But what we found is you queue up all
the traffic and once the pod is live on
309
00:15:23,865 --> 00:15:27,765
the new node that traffic is replayed
and all the messages come through.
310
00:15:27,765 --> 00:15:30,765
So even on something like a web socket,
you don't actually lose messages.
311
00:15:30,765 --> 00:15:32,805
They're just held up for a few seconds.
312
00:15:33,305 --> 00:15:36,035
And that's extremely important
from maintaining that connection
313
00:15:36,035 --> 00:15:37,145
state, like you were mentioning.
314
00:15:37,555 --> 00:15:39,835
one of our customers that we're
working with this heavily on,
315
00:15:39,865 --> 00:15:41,485
they run spark streaming jobs.
316
00:15:41,985 --> 00:15:46,455
So they're 24 7, 365, pulling off a
queue, running data transformations
317
00:15:46,455 --> 00:15:49,695
and detections, and then pushing
somewhere else for alerting mechanisms.
318
00:15:50,195 --> 00:15:54,515
If they have a pod go down, it takes about
two minutes to get that process restarted,
319
00:15:54,515 --> 00:15:55,865
pull in all the data that they need.
320
00:15:55,865 --> 00:15:58,585
Again, that's two minutes of backlog.
321
00:15:58,675 --> 00:16:00,175
They have super tight SLAs.
322
00:16:00,175 --> 00:16:05,895
They have a five minute SLA from
message creation to end run through
323
00:16:05,895 --> 00:16:07,185
the entire detection pipeline.
324
00:16:07,185 --> 00:16:10,365
So if you've got a two minute
delay on that shard in your Kafka
325
00:16:10,365 --> 00:16:14,555
topic, that's a huge chunk of that
five minutes that you just ate up.
326
00:16:15,055 --> 00:16:16,945
You're talking all the other pipeline.
327
00:16:16,945 --> 00:16:19,075
It's very easy to start
missing SLAs there.
328
00:16:19,575 --> 00:16:23,475
It's, you can't take maintenance
windows if you're 24 7, 365 and
329
00:16:23,475 --> 00:16:24,765
you're doing security processing.
330
00:16:25,195 --> 00:16:25,645
you don't.
331
00:16:25,705 --> 00:16:27,865
You can't be like, well, security's
gonna be offline for 10 minutes
332
00:16:27,865 --> 00:16:28,945
while we move our pods around.
333
00:16:29,526 --> 00:16:32,316
Like, that's just, that's not
acceptable in that world mindset.
334
00:16:32,716 --> 00:16:35,956
so keeping that connectivity, keeping
the connection state, being able to
335
00:16:35,956 --> 00:16:39,386
keep everything intact, keeping the
Kafka connection, keeping the, spark
336
00:16:39,536 --> 00:16:43,406
driver connection is all super important
with being able to move that entire
337
00:16:43,406 --> 00:16:47,646
TCP/IP stack over from one node to
another, during that migration process.
338
00:16:48,146 --> 00:16:51,236
Yeah, and I mean, we're really talking
about a lot of the different kinds of
339
00:16:51,236 --> 00:16:56,886
problems that come with shifting workloads
like being able to say, you know, walking
340
00:16:56,886 --> 00:17:01,876
into an environment and sort of being
your own wrecking ball and, your own chaos
341
00:17:01,876 --> 00:17:04,996
monkey and saying, I'm gonna go over here
and push the power button on this node,
342
00:17:05,066 --> 00:17:09,966
or I'm gonna properly shut down this node,
do you have everything set up correctly so
343
00:17:09,966 --> 00:17:15,256
that connections are properly drained that
is a, such a. I would say a moving target,
344
00:17:15,256 --> 00:17:18,836
especially because every time we've had
these processes where I've had clients
345
00:17:18,836 --> 00:17:22,706
where we go through this like exercise of
we're going to do maintenance on a node
346
00:17:22,766 --> 00:17:27,116
and we're even gonna plan for it, and
then we do it, and then we fix all the
347
00:17:27,116 --> 00:17:31,896
issues of the pods and the shutdown timing
and, the Argo CD deployment settings
348
00:17:31,896 --> 00:17:33,756
that we need to massage and perfect.
349
00:17:33,876 --> 00:17:38,016
And then, you know, six months later, if
we do it again, the same thing happens
350
00:17:38,016 --> 00:17:41,256
because now there's new workloads that
weren't perfected and weren't well tested.
351
00:17:41,536 --> 00:17:44,056
if I can make a career out
of actually being like.
352
00:17:44,556 --> 00:17:49,576
a pod migration guru, like that sounds
like my kind of dream job where we crash
353
00:17:49,576 --> 00:17:53,666
and break everything and then we track
all of the potential issues of that and
354
00:17:53,666 --> 00:17:57,296
we are like a tiger team that goes pod
by pod and certifies this is like, yep,
355
00:17:57,296 --> 00:18:00,116
this pod can now move safely without risk.
356
00:18:00,336 --> 00:18:01,746
because we've got everything dialed in.
357
00:18:01,746 --> 00:18:03,036
We've got all the right settings.
358
00:18:03,396 --> 00:18:05,016
I feel like that's a workshop opportunity.
359
00:18:05,016 --> 00:18:07,876
maybe sell something on that because
there are so
many levels of complexity
360
00:18:07,876 --> 00:18:13,336
we haven't even talked about, like
database logs and database mirroring
361
00:18:13,386 --> 00:18:16,266
you can't really spin up a new node
of a database and let it sit there.
362
00:18:16,266 --> 00:18:19,176
Idle is a pod while you're waiting for
the old one to shut down, they can't
363
00:18:19,176 --> 00:18:20,736
access the same files, blah, blah, blah.
364
00:18:20,786 --> 00:18:23,686
it just depends on the workload,
on how complex this all gets.
365
00:18:23,686 --> 00:18:28,266
But I'm assuming also that when we talk
about something like live migrations,
366
00:18:28,266 --> 00:18:29,556
we're not just concerned with networking.
367
00:18:29,556 --> 00:18:31,626
We're also somehow shifting to storage.
368
00:18:31,981 --> 00:18:35,441
I'm guessing there's certain
limitations to that where you're not
369
00:18:35,441 --> 00:18:38,231
replicating on the backend volumes.
370
00:18:38,231 --> 00:18:41,651
You're, I guess you're just
using like ice zy reconnects or
371
00:18:41,811 --> 00:18:43,011
how's that, how does that work?
372
00:18:43,011 --> 00:18:45,501
we haven't really gotten into the
solution, but I know you're only on
373
00:18:45,501 --> 00:18:49,561
certain clouds right now, and I'm assuming
that's partly due to the technicals of
374
00:18:49,741 --> 00:18:51,421
the limitations of their infrastructure.
375
00:18:51,921 --> 00:18:52,671
Right, exactly.
376
00:18:52,681 --> 00:18:55,521
each cloud has different kind
of quirks, around how they
377
00:18:55,521 --> 00:18:58,041
function, what the different,
technologies look like around them.
378
00:18:58,321 --> 00:19:01,801
somebody had asked about being able
to move, you know, larger systems
379
00:19:01,801 --> 00:19:06,681
are the limits around it and their,
it depends on the use case, right?
380
00:19:06,681 --> 00:19:10,401
If you're talking spot instances,
being able to move from one spot
381
00:19:10,401 --> 00:19:13,221
instance to another spot instance
in a two minute interruption window
382
00:19:13,221 --> 00:19:15,551
on AWS, depends on how much data.
383
00:19:15,791 --> 00:19:19,511
If you're trying to move 120 gigs of
data physics is working against you.
384
00:19:19,541 --> 00:19:22,571
you don't have enough time in that two
minute window to get enough through
385
00:19:22,571 --> 00:19:24,901
the pipe over to the new, system.
386
00:19:25,401 --> 00:19:30,151
Now if you're talking small pods,
if you're talking less than 32 gig
387
00:19:30,151 --> 00:19:31,796
nodes, you can move that fast enough.
388
00:19:31,796 --> 00:19:34,046
64 gigs, maybe you're on the edge.
389
00:19:34,046 --> 00:19:36,536
Depends on how much other network
traffic is tying up the bandwidth,
390
00:19:36,876 --> 00:19:40,026
64 gigs is getting on the edge that
you can move in a two minute window.
391
00:19:40,376 --> 00:19:43,586
that other example I was talking about,
those long running spark
streaming
392
00:19:43,586 --> 00:19:48,027
jobs, if they're running on demand,
live migration, is still a massive
393
00:19:48,027 --> 00:19:51,327
benefit because now you can do server
upgrades without taking an outage.
394
00:19:51,687 --> 00:19:55,287
You can create a new node running
your new patch version of, Kubernetes,
395
00:19:55,287 --> 00:19:58,767
running your new Patched Os, and
migrate the pod from one to the other.
396
00:19:59,127 --> 00:20:02,627
Your time to replicate is less important.
397
00:20:03,127 --> 00:20:06,032
Even if it takes you three minutes
four minutes, or five minutes
398
00:20:06,032 --> 00:20:09,242
to replicate the memory from
one box to the other, who cares?
399
00:20:09,512 --> 00:20:11,942
It's not gonna be paused for that
long, because what we're doing
400
00:20:11,942 --> 00:20:13,352
is we're doing delta replication.
401
00:20:13,742 --> 00:20:16,772
So you replicate a big chunk, and then
a smaller chunk, and then a smaller
402
00:20:16,772 --> 00:20:20,252
chunk until the chunk is small enough
where you can do it in a pause window.
403
00:20:20,672 --> 00:20:23,882
And so now when you're moving a
huge service from one to the other.
404
00:20:24,182 --> 00:20:24,602
Same thing.
405
00:20:24,602 --> 00:20:28,402
If you're talking like NVME local storage,
we've got another customer we're working
406
00:20:28,402 --> 00:20:30,472
with and it's a different set of problems.
407
00:20:30,832 --> 00:20:35,812
They have a terabyte of NVME that they use
on local ephemeral disc on every node that
408
00:20:35,812 --> 00:20:37,492
needs to be replicated from node to node.
409
00:20:37,492 --> 00:20:41,072
When they do node upgrades, that
takes about 20 minutes, to replicate
410
00:20:41,072 --> 00:20:43,202
all of that from one node to
another, even on high throughput
411
00:20:43,202 --> 00:20:44,792
discs, on high throughput nodes.
412
00:20:45,292 --> 00:20:47,872
But if it's happening in the
background, while everything else
413
00:20:47,872 --> 00:20:50,512
is humming along nicely, who cares?
414
00:20:50,962 --> 00:20:52,072
Keep replicating it over.
415
00:20:52,072 --> 00:20:55,132
You keep going down to Deltas, and
then once your deltas get small enough,
416
00:20:55,132 --> 00:20:58,492
you pause for six to 10 seconds,
depending on how big the service is.
417
00:20:58,732 --> 00:20:59,842
And then you slide it over.
418
00:21:00,182 --> 00:21:01,562
a lot of these things are being solved.
419
00:21:01,562 --> 00:21:06,242
We're actively reducing these pause
times by being able to do more prep
420
00:21:06,242 --> 00:21:09,122
behind the scenes, being able to do
more processing behind the scenes.
421
00:21:09,402 --> 00:21:11,872
everything is, operating
as a containerd plugin.
422
00:21:12,272 --> 00:21:16,852
I saw somebody asked, about on-prem
we will be supporting on-prem, we
423
00:21:16,852 --> 00:21:18,592
will be supporting other solutions.
424
00:21:19,092 --> 00:21:23,962
The big catch there is everybody
has some different flavor of
425
00:21:23,962 --> 00:21:26,212
networking and different flavors
of things behind the scenes.
426
00:21:26,212 --> 00:21:31,162
The actual live migration piece right now
could apply to any Kubernetes anywhere.
427
00:21:31,222 --> 00:21:35,602
It's the IP stack that gets a little
trickier because you've got cilium
428
00:21:35,602 --> 00:21:39,912
running places, you've got Calico running
places, you've got VPN, V-P-C-C-N-I
429
00:21:40,002 --> 00:21:42,762
running places, you know, everybody
has different networking flavors.
430
00:21:43,262 --> 00:21:45,872
So being able to maintain network
connections when you do the
431
00:21:45,872 --> 00:21:50,062
migration is largely the more
difficult part of the whole process.
432
00:21:50,436 --> 00:21:50,706
Hmm.
433
00:21:50,722 --> 00:21:53,992
Being able to move the pod isn't that
bad, so if you've got workloads that
434
00:21:54,112 --> 00:21:57,562
you can reestablish connections and the
connection resetting is not a big deal,
435
00:21:57,742 --> 00:22:02,232
but you don't wanna have to restart
the pod, that's fairly straightforward.
436
00:22:02,232 --> 00:22:05,922
We could pretty much do that today across
any containerd compatible Kubernetes.
437
00:22:06,252 --> 00:22:09,102
It's specifically the networking
that causes a lot more hardships,
438
00:22:09,132 --> 00:22:11,262
because everybody has a
different flavor of networking.
439
00:22:11,672 --> 00:22:17,452
for AWS, we were able to fork
the open source AWS node CNI, and
440
00:22:17,452 --> 00:22:20,422
create our own flavor of it that
now handles the networking piece.
441
00:22:20,782 --> 00:22:24,652
So we're using the open source
AWS CI code, and we've modified
442
00:22:24,652 --> 00:22:26,212
it, and now it works just fine.
443
00:22:26,212 --> 00:22:30,422
for our purposes, we're doing something
similar on GCP, working with the gc.
444
00:22:30,472 --> 00:22:34,057
GPFI recall is using cilium under
the hood for their networking.
445
00:22:34,807 --> 00:22:38,087
So we're gonna be, building a
similar plugin for their cilium side.
446
00:22:38,587 --> 00:22:41,932
Yeah, and the nice thing is, I
guess if you build it for cilium.
447
00:22:42,432 --> 00:22:44,952
would it work universally
across any cilium deployment?
448
00:22:44,952 --> 00:22:48,632
In theory, I mean, I'm just thinking of
the most popular CNIs and if you check
449
00:22:48,632 --> 00:22:54,122
those off the list, it suddenly gives you,
you know, a lot more reach than having to
450
00:22:54,122 --> 00:22:57,002
go plowed by cloud or os by os, you know?
451
00:22:57,222 --> 00:22:57,312
and,
452
00:22:57,362 --> 00:22:57,962
Exactly.
453
00:22:58,042 --> 00:23:01,522
our first iteration of this back in
January, February, the first version
454
00:23:01,522 --> 00:23:04,312
that we demoed was actually Calico.
455
00:23:04,882 --> 00:23:07,342
A lot of people were
like, I don't
wanna have to rip out my cluster and
456
00:23:07,342 --> 00:23:09,292
rebuild it with Calico as the CNI.
457
00:23:09,292 --> 00:23:11,992
we were able to figure out
a way to work with the, VPC
458
00:23:11,992 --> 00:23:14,182
CNI, as a backing basis there.
459
00:23:14,452 --> 00:23:16,582
So Calico's pretty much already built.
460
00:23:16,922 --> 00:23:21,212
we've got AWS CNI now built,
cilium our next target, that saw
461
00:23:21,212 --> 00:23:25,022
somebody asked about Azure, Azure
is probably gonna be early 2026.
462
00:23:25,082 --> 00:23:28,692
we'll be E-K-S-G-K-E and then
we'll work on a KS, and then we'll
463
00:23:28,692 --> 00:23:30,262
work on-prem solutions after that.
464
00:23:30,352 --> 00:23:32,542
So on-prem will probably
be sometime in 2026.
465
00:23:33,092 --> 00:23:35,852
Yeah, I can remember, going
back to the two thousands.
466
00:23:35,852 --> 00:23:39,352
I can remember when we went from
delayed migrations or paused
467
00:23:39,352 --> 00:23:41,002
migrations to live migrations.
468
00:23:41,302 --> 00:23:44,632
I can remember reading the technical
papers coming out of, VMware and
469
00:23:44,632 --> 00:23:48,852
Microsoft and they were talking about
the idea of, these deltas continually
470
00:23:48,852 --> 00:23:52,402
repeating the Delta process until
you get down to zero or like you can
471
00:23:52,402 --> 00:23:55,102
fit it in a packet and then that's
the final packet kind of thing.
472
00:23:55,102 --> 00:23:57,652
I don't know why I remember that all
these years later, but I do remember
473
00:23:57,652 --> 00:24:00,932
that I thought that was some pretty cool
science, like some pretty cool, physics
474
00:24:00,932 --> 00:24:04,052
across the wire because back then we were
lucky if our servers had one gigabit,
475
00:24:04,792 --> 00:24:06,472
200 gig workloads or anything like that.
476
00:24:06,782 --> 00:24:10,322
this actually led me during my
research and, we could talk about
477
00:24:10,322 --> 00:24:15,362
the idea that there are, there are
attempts in Linux over the years
478
00:24:15,362 --> 00:24:17,222
to try to solve this universally.
479
00:24:17,222 --> 00:24:22,792
I did some research before the show and
saw some projects around ML workloads in
480
00:24:22,792 --> 00:24:27,082
particular a lot of, engineers, whether
it's platform engineering or just the ML
481
00:24:27,082 --> 00:24:32,902
engineers themselves interested in this
because of the, sort of the problems of
482
00:24:32,902 --> 00:24:36,952
large ML or AI workloads today where you
can't interrupt them if you interrupt
483
00:24:36,952 --> 00:24:38,962
them, you have to basically start over.
484
00:24:38,962 --> 00:24:41,242
it's sort of a precious workload
while it's running and it
485
00:24:41,242 --> 00:24:42,322
might be running long time.
486
00:24:42,372 --> 00:24:45,982
do you have, AI and ML workload
customers where they're.
487
00:24:46,432 --> 00:24:50,602
Are they maybe
part of the first movers
to move onto something like this?
488
00:24:50,632 --> 00:24:53,167
I'm basing it on the KubeCon talks
and things that I've seen out there,
489
00:24:53,257 --> 00:24:57,197
Large scale data analytics is
definitely, one of the big players here.
490
00:24:57,197 --> 00:25:00,317
A lot of it's spark driven data
analytics that we're seeing,
491
00:25:00,667 --> 00:25:02,257
because of exactly that problem.
492
00:25:02,587 --> 00:25:06,397
A lot of these jobs will be running
for 8, 10, 12, 14, 16 hours and.
493
00:25:07,602 --> 00:25:10,362
Running those on demand at the
scale that they're running them
494
00:25:10,362 --> 00:25:12,912
at is extraordinarily expensive.
495
00:25:12,912 --> 00:25:18,792
So the big ask is how do we get those
workloads onto spot instances where
496
00:25:18,792 --> 00:25:22,662
when we get the interruption notice we
can fall back to some type of reserved
497
00:25:22,662 --> 00:25:25,332
capacity and then fail back to spot.
498
00:25:25,422 --> 00:25:29,702
So basically the goal is to move
to this new concept where in your
499
00:25:29,702 --> 00:25:34,312
Kubernetes cluster you have some swap
space, whether that's, excess, spot
500
00:25:34,312 --> 00:25:37,642
capacity, two or three extra nodes of
spot capacity or a couple of nodes of
501
00:25:37,642 --> 00:25:42,622
on-demand capacity where if you get a
node interruption, you can quickly swap
502
00:25:42,622 --> 00:25:47,342
into those nodes, and then once you stand
your new spot instance back up, then
503
00:25:47,342 --> 00:25:48,737
you can swap back to that spot instance.
504
00:25:50,072 --> 00:25:50,762
where we're headed.
505
00:25:50,812 --> 00:25:53,812
that's what Q4 is gonna be working on
this year, is to be able to automate that
506
00:25:53,812 --> 00:25:58,492
entire process to where you can float back
and forth between reserve capacity and
507
00:25:58,492 --> 00:26:04,042
spot capacity to really save on those,
data analytics jobs, those large ML jobs.
508
00:26:04,462 --> 00:26:07,102
We're not to the GPU side of things yet.
509
00:26:07,492 --> 00:26:11,362
I'd love to get us to where we could
migrate GPU workloads 'cause that's where
510
00:26:11,362 --> 00:26:13,402
the next big bottleneck is gonna be.
511
00:26:13,882 --> 00:26:17,122
the hooks aren't there in the
Nvidia tool sets yet for the Cuda
512
00:26:17,122 --> 00:26:20,522
tool sets in a lot of places, to be
able to get what we need for data.
513
00:26:20,522 --> 00:26:21,962
we're figuring our way around that.
514
00:26:22,272 --> 00:26:27,972
they tend to be much larger, so the
time taken to move them very expensive.
515
00:26:28,252 --> 00:26:31,342
it might be 20 minutes to be able
to move a job from one to the other.
516
00:26:31,372 --> 00:26:34,012
'cause it took 20 minutes to get
a startup up in the first place.
517
00:26:34,272 --> 00:26:37,152
just due to the size of the models and
how much data you have to replicate.
518
00:26:37,652 --> 00:26:41,432
we're starting to put some POC work
into the GPU side of things while we're
519
00:26:41,432 --> 00:26:44,822
continuing on full steam with building
out the expansion of the feature set
520
00:26:44,822 --> 00:26:47,492
of the CPG, and memory based workloads.
521
00:26:48,042 --> 00:26:48,312
Alright.
522
00:26:48,312 --> 00:26:52,992
Dan, I was curious if you've seen
on, the implementation side of this.
523
00:26:53,542 --> 00:26:54,862
when we talk about.
524
00:26:55,362 --> 00:26:59,322
the need to live, migrate a pod, whether
it's for maintenance then the almost
525
00:26:59,322 --> 00:27:03,732
feel like the next level is the idea
of spot instances, I love that idea of
526
00:27:03,732 --> 00:27:07,332
my infrastructure dynamically failing
and my applications can handle it.
527
00:27:07,332 --> 00:27:11,002
is there a maturity level where
you see people start out it's hard
528
00:27:11,002 --> 00:27:13,822
for me to imagine like on day one
someone's like, yeah, let's just put
529
00:27:13,822 --> 00:27:15,562
it all on spot instances in yolo.
530
00:27:15,592 --> 00:27:17,032
let's just fi we don't care.
531
00:27:17,032 --> 00:27:17,662
It's all good.
532
00:27:17,662 --> 00:27:19,102
Mi Live migration will solve it all.
533
00:27:19,102 --> 00:27:21,962
Because obviously, there are
physics limits to the amount of
534
00:27:21,962 --> 00:27:23,702
data we can transmit over the wire.
535
00:27:23,702 --> 00:27:25,232
I'm imagining this scenario where you're.
536
00:27:25,527 --> 00:27:30,437
accrediting certain workloads,
like this replica set is good
537
00:27:30,707 --> 00:27:33,347
for spot because it's low data.
538
00:27:33,397 --> 00:27:36,667
we don't need to transfer a hundred
giga data during a two minute outage
539
00:27:36,667 --> 00:27:38,077
or a two minute notice of outage.
540
00:27:38,307 --> 00:27:41,477
do you see that as a maturity
scale where you have to
541
00:27:41,593 --> 00:27:41,803
Yeah.
542
00:27:41,803 --> 00:27:44,228
I mean, it is absolutely
a maturity scale, right?
543
00:27:44,228 --> 00:27:46,668
kind of going back to the references
we talked about, the early days of
544
00:27:46,668 --> 00:27:50,978
VMware, nobody started doing vMotion
in production, everyone started it.
545
00:27:51,478 --> 00:27:55,798
Oh, we've got this, five second
interruption, development and test
546
00:27:55,858 --> 00:27:57,688
boxes can handle that all day long.
547
00:27:57,938 --> 00:28:00,458
so it's the same concept really
that we're living in now.
548
00:28:00,458 --> 00:28:01,898
We're going through that same evolution.
549
00:28:01,898 --> 00:28:02,498
I agree with Phil.
550
00:28:02,498 --> 00:28:07,028
I think we're doing it much faster
than VMware did in 2002, 2003.
551
00:28:07,698 --> 00:28:09,228
I was around when that happened as well.
552
00:28:09,228 --> 00:28:11,873
So I remember racking and
stacking all those boxes.
553
00:28:12,213 --> 00:28:13,713
but yeah, it's very much the same thing.
554
00:28:13,713 --> 00:28:17,843
container live migration is brand new,
we've just been GA for a month with it.
555
00:28:17,843 --> 00:28:21,213
So we've had conversations at trade
shows and with customers and there's
556
00:28:21,213 --> 00:28:22,473
a lot of excitement around it.
557
00:28:22,473 --> 00:28:25,083
I think we're still trying to
figure out where it fits, what
558
00:28:25,083 --> 00:28:28,293
are the exact workloads that it
makes the most sense to do this in.
559
00:28:28,653 --> 00:28:31,378
And yeah, I think it's going
to be a process of adoption.
560
00:28:31,428 --> 00:28:33,258
there's definitely a lot of use cases.
561
00:28:33,618 --> 00:28:37,878
I think the spot is a very interesting
use case, especially the large data models
562
00:28:37,878 --> 00:28:39,678
and things that we're processing today.
563
00:28:40,018 --> 00:28:43,078
I'm working with a customer now that's
doing a lot of video processing in
564
00:28:43,078 --> 00:28:47,488
Kubernetes and that's a very, you
know, CPN memory intensive job.
565
00:28:47,488 --> 00:28:51,688
I mean, you know, we're talking a cluster
that scales from a D CPUs to 6,500
566
00:28:51,688 --> 00:28:53,368
CPUs while they're processing this.
567
00:28:53,588 --> 00:28:56,888
we're really trying to figure out
where it makes the most sense to
568
00:28:56,888 --> 00:28:58,478
apply this type of technology.
569
00:28:58,978 --> 00:29:03,708
no one wants to have that kind of
dynamic scale and then have to pay
570
00:29:03,708 --> 00:29:06,408
for reserved instances for all of
that, like, worst case scenario.
571
00:29:06,458 --> 00:29:08,018
that sounds like a billing nightmare.
572
00:29:08,068 --> 00:29:11,363
and you don't want a job that
runs for, you know, hours that
573
00:29:11,363 --> 00:29:15,833
cost you tons of money to fail 80%
through and have to restart it I
574
00:29:15,833 --> 00:29:17,603
mean, that's just not efficient.
575
00:29:17,603 --> 00:29:20,673
So, yeah, I think the ability to
really move this and allow those
576
00:29:20,673 --> 00:29:22,023
workloads to finish is gonna be.
577
00:29:22,518 --> 00:29:23,588
Huge for the market.
578
00:29:24,088 --> 00:29:25,948
Alright, so we have been
talking a lot about the problem
579
00:29:25,948 --> 00:29:27,028
and some of the solution.
580
00:29:27,088 --> 00:29:29,998
we do have some slides that give
visualizations for those on YouTube.
581
00:29:30,028 --> 00:29:31,138
this will turn into a podcast.
582
00:29:31,138 --> 00:29:34,708
So audio listeners, we will give
you the alt text, version of it
583
00:29:34,708 --> 00:29:35,518
while we're talking about it.
584
00:29:35,518 --> 00:29:40,122
But, Philip, I'm Exactly what is
happening and the process of how a live
585
00:29:40,122 --> 00:29:41,952
migration, like how does it kick off?
586
00:29:42,382 --> 00:29:45,082
what's really going on in the
background when it starts,
587
00:29:45,142 --> 00:29:45,502
Absolutely.
588
00:29:45,502 --> 00:29:48,492
and we do have some
better demos other places.
589
00:29:48,492 --> 00:29:49,302
I think on the website.
590
00:29:49,352 --> 00:29:52,442
basically what we have is a live
migration controller looking
591
00:29:52,442 --> 00:29:55,322
across all the workloads and nodes
that are lab migration enabled.
592
00:29:55,372 --> 00:29:58,012
You don't necessarily have to
turn this on for everything.
593
00:29:58,222 --> 00:30:00,832
You've got all your stateless
workloads, you don't need to live
594
00:30:00,832 --> 00:30:03,322
migration, stateless workloads,
just treat them as normal.
595
00:30:03,712 --> 00:30:06,772
You've got your stateful workloads
that you do want this to use for,
596
00:30:06,832 --> 00:30:09,802
so you could set up a specific,
you know, node group for that.
597
00:30:10,052 --> 00:30:14,012
that's gonna allow you to select what you
actually want to do live migration for.
598
00:30:14,512 --> 00:30:17,512
You could use it for everything, but
it just eats up more network bandwidth
599
00:30:17,512 --> 00:30:19,882
if you're using it for the stuff
that already tolerates being moved.
600
00:30:20,382 --> 00:30:23,532
that controller's gonna be looking for
different signals within the cluster of
601
00:30:23,532 --> 00:30:25,422
when something needs to be live, migrated.
602
00:30:25,422 --> 00:30:26,862
instance, interruption is a good one.
603
00:30:27,212 --> 00:30:30,662
being able to do bin packing, evicting
a node from the cluster because it's
604
00:30:30,662 --> 00:30:34,802
underutilized, and then migrating those
workloads to another node in the cluster.
605
00:30:34,862 --> 00:30:36,062
what we call rebalancing.
606
00:30:36,132 --> 00:30:38,952
basically, rebuilding the
cluster with a new set of nodes.
607
00:30:39,252 --> 00:30:42,012
And that could be because you're
doing a node upgrade, you're doing a
608
00:30:42,012 --> 00:30:45,152
Kubernetes upgrade, you're doing an
OS upgrade, you're just trying to.
609
00:30:45,652 --> 00:30:47,632
Get a more efficient set of nodes.
610
00:30:47,952 --> 00:30:51,672
all of those are good reasons that you
would want to do your live migration.
611
00:30:52,172 --> 00:30:56,672
So what's gonna happen in that process
is the two daemon sets on the source
612
00:30:56,672 --> 00:30:58,922
node and the destination node are
gonna start talking to each other.
613
00:30:59,422 --> 00:31:03,412
They're going to look at the pods on
the source node, start synchronizing
614
00:31:03,412 --> 00:31:04,972
them over to the destination node.
615
00:31:05,272 --> 00:31:08,902
So behind the scenes, all of that
kind of memory is being copied over
616
00:31:08,902 --> 00:31:12,532
any disc states being copied over
any TCP/IP connection statuses are
617
00:31:12,532 --> 00:31:16,552
being copied over and you're doing
all that prep work behind the scenes.
618
00:31:17,052 --> 00:31:20,982
If you have, ephemeral
storage, on
the node that'll start getting copied.
619
00:31:21,042 --> 00:31:24,252
Obviously, depending on how much, it's
gonna depend on how long it takes.
620
00:31:24,752 --> 00:31:29,572
Once the two nodes have identical
copies of the data, that's when
621
00:31:29,572 --> 00:31:33,742
the live controller will say, it's
time to cut over it will cut the.
622
00:31:34,102 --> 00:31:38,252
Connections from one, pause it and put
it into a pause state in containerd,
623
00:31:38,802 --> 00:31:40,992
then it will unpause on the new node.
624
00:31:41,052 --> 00:31:42,222
It'll come up with a new name.
625
00:31:42,402 --> 00:31:44,232
Right now we call 'em
clone one, clone two.
626
00:31:44,232 --> 00:31:45,312
We just add clone to 'em.
627
00:31:45,312 --> 00:31:47,532
So you can tell which was the
before and which was the after.
628
00:31:47,872 --> 00:31:51,702
when that clone one unpause,
then traffic will be going to it.
629
00:31:51,702 --> 00:31:55,422
It'll have the same exact IP address that
it had while it was on the previous node.
630
00:31:55,722 --> 00:31:57,852
All the traffic continues onto that node.
631
00:31:57,882 --> 00:32:02,232
It picks up exactly where it left off,
and the old pod disappears, right?
632
00:32:02,232 --> 00:32:07,302
The old pod gets shut down and torn down
if you have something like a PVC attached.
633
00:32:07,362 --> 00:32:11,102
So if you've got an E-B-S-P-V-C
attached, there is a longer pause
634
00:32:11,102 --> 00:32:13,052
because you have to do a detach reattach.
635
00:32:13,332 --> 00:32:15,252
with the API calls, it works.
636
00:32:15,312 --> 00:32:17,502
It just takes a little bit
longer for that pause state.
637
00:32:18,002 --> 00:32:20,992
That's The downfall of
having to work with, APIs.
638
00:32:21,422 --> 00:32:23,812
it takes time to do an unbind
rebind, to the new node.
639
00:32:24,212 --> 00:32:25,142
but it works today.
640
00:32:25,142 --> 00:32:28,292
If you're using NFS where you can
do a multitask, then it's instant,
641
00:32:28,292 --> 00:32:29,612
it doesn't actually add any delay.
642
00:32:29,892 --> 00:32:32,262
just NFS is a slower storage technology.
643
00:32:32,262 --> 00:32:35,162
So does that sort of make
sense from a high level?
644
00:32:35,662 --> 00:32:36,172
Yeah.
645
00:32:36,232 --> 00:32:39,882
when we talk about Cast AI as a
solution, it do live migrations
646
00:32:39,882 --> 00:32:41,052
based on certain criteria?
647
00:32:41,112 --> 00:32:46,352
Is it making decisions around, if you
say I want a bin pack all the time,
648
00:32:46,852 --> 00:32:49,702
it in the background, is it just like
doing live migrations on your behalf?
649
00:32:49,702 --> 00:32:53,812
Or is this something where you're
largely doing it with humans clicking
650
00:32:53,812 --> 00:32:56,362
buttons and controlling the chaos.
651
00:32:56,862 --> 00:32:59,322
No, this goes back to what we
had talked about at the beginning
652
00:32:59,322 --> 00:33:01,182
where, automation is key.
653
00:33:01,602 --> 00:33:05,952
when a node is underutilized,
our bin packer is, probably the
654
00:33:05,952 --> 00:33:07,302
most sophisticated on the market.
655
00:33:07,752 --> 00:33:11,682
It analyzes and runs live tests on
every node in the cluster of whether
656
00:33:11,682 --> 00:33:15,342
that node can be deleted, whether that
node doesn't need to be there anymore.
657
00:33:15,372 --> 00:33:18,942
And it'll simulate all the pods being
redistributed throughout the cluster.
658
00:33:19,342 --> 00:33:22,852
if the answer is we don't need this
node, it would automatically kick
659
00:33:22,852 --> 00:33:26,582
off a live migration of all the
pods on that node Once it's empty,
660
00:33:26,792 --> 00:33:28,052
it'll just get garbage collected.
661
00:33:28,552 --> 00:33:31,072
once it's gone, all your pods
are running on the new nodes.
662
00:33:31,072 --> 00:33:32,482
Everything's moved seamlessly.
663
00:33:32,602 --> 00:33:33,982
You haven't seen any interruption.
664
00:33:34,322 --> 00:33:36,172
the cluster keeps continuing as normal.
665
00:33:36,282 --> 00:33:39,042
Most of our customers do
scheduled rebalances, so those
666
00:33:39,042 --> 00:33:40,362
are just in the background.
667
00:33:40,362 --> 00:33:45,542
It's evaluating how efficiently
designed the nodes in the cluster are.
668
00:33:46,022 --> 00:33:47,852
If the nodes in the cluster are.
669
00:33:48,352 --> 00:33:51,382
Not as efficient as they could be, and
different shapes and sizes would be
670
00:33:51,382 --> 00:33:55,172
better for, that setup at that point
in the day, based on the mixture of
671
00:33:55,172 --> 00:34:00,002
workloads there, it'll do a blue-green
deployment, set up new nodes live,
672
00:34:00,002 --> 00:34:02,972
migrate the workloads to those new
nodes and tear down the old ones.
673
00:34:02,972 --> 00:34:05,702
So everything that we're
talking about here can either
674
00:34:05,762 --> 00:34:07,262
be scheduled or it's automatic.
675
00:34:07,262 --> 00:34:09,122
It's running every few minutes on a cycle.
676
00:34:09,372 --> 00:34:11,542
but yeah, no, it's entirely
seamless to the users.
677
00:34:12,042 --> 00:34:12,522
Nice.
678
00:34:13,022 --> 00:34:17,432
So in the technical details, we're
moving the IP address, I think you had a
679
00:34:17,432 --> 00:34:22,102
diagram, showing the pod, on the nodes,
when we get down to the nitty gritty of
680
00:34:22,102 --> 00:34:26,692
Kubernetes level stuff, pod is recreated,
but pod names have to be unique you
681
00:34:26,692 --> 00:34:29,682
can't have the IP on both nodes at once.
682
00:34:30,042 --> 00:34:33,342
And then there's the difference
between TCP and UDP and other, you
683
00:34:33,342 --> 00:34:38,282
know, IP protocols and the, there's
a lot of little devils in the
684
00:34:38,282 --> 00:34:39,662
details that I'm super interested in.
685
00:34:39,662 --> 00:34:43,082
We won't have time to go into all
of it, but I do remember you showed
686
00:34:43,132 --> 00:34:46,422
the replica, the pod that you're
creating, step one is we create a
687
00:34:46,422 --> 00:34:47,622
pod and download an image, right?
688
00:34:47,622 --> 00:34:48,952
this is all still going
through containerd.
689
00:34:48,972 --> 00:34:52,572
So it's not, there's not like voodoo
magic in the background happening
690
00:34:52,572 --> 00:34:54,392
outside of the purview of containerd.
691
00:34:54,917 --> 00:34:56,177
maybe you can talk about that for a minute
692
00:34:56,367 --> 00:34:57,087
Right, Exactly.
693
00:34:57,087 --> 00:35:01,277
So being by changing the pod name,
you now have a. Placeholder for
694
00:35:01,277 --> 00:35:03,317
your new pod information to go into.
695
00:35:03,797 --> 00:35:07,337
And it does maintain the same IP address
when it moves over from one to the other.
696
00:35:07,337 --> 00:35:11,107
So to your point, that's when that
switch has to kick in where the
697
00:35:11,107 --> 00:35:12,847
old pod definition disappears.
698
00:35:12,847 --> 00:35:16,687
And the new pod definition appears
in your, control plane with the
699
00:35:16,687 --> 00:35:18,487
API calls up to the coop API.
700
00:35:18,797 --> 00:35:22,877
that cutover is extremely important
because you can't have the same
701
00:35:22,877 --> 00:35:24,677
pod living in the same place twice.
702
00:35:25,107 --> 00:35:28,047
that's why we do have to change
a name when we switch it over.
703
00:35:28,327 --> 00:35:31,417
there's certain services that, cause
some tricks because they have an
704
00:35:31,417 --> 00:35:34,147
operator structure where they expect
there to be a certain pod name.
705
00:35:34,477 --> 00:35:37,432
So when you move it and add the clone
suffix to it, we're working on finding
706
00:35:37,432 --> 00:35:39,412
workarounds to that, in certain areas.
707
00:35:39,842 --> 00:35:42,632
that is a little bit tricky on
certain workloads because you can't
708
00:35:42,782 --> 00:35:45,242
have the same pod existing with the
same name in two different spots.
709
00:35:45,242 --> 00:35:46,232
They have to be unique.
710
00:35:46,592 --> 00:35:47,672
but yeah, definitely.
711
00:35:48,172 --> 00:35:52,842
the IP is the same, but the pod
name is gonna have a clone dash
712
00:35:52,842 --> 00:35:54,252
one or something like that on it
713
00:35:54,252 --> 00:35:54,462
Yep.
714
00:35:54,512 --> 00:35:58,672
yeah, so it starts with your pod
and then there's an event that
715
00:35:58,672 --> 00:36:04,782
happens outside of the pod that is
talking into, containerd and you're
716
00:36:05,218 --> 00:36:06,088
It's a second pod.
717
00:36:06,588 --> 00:36:07,038
Correct.
718
00:36:07,068 --> 00:36:08,268
It's adding that placeholder.
719
00:36:08,268 --> 00:36:11,358
And because the placeholder pod is
actually still in a paused state.
720
00:36:11,568 --> 00:36:13,188
It can have the same IP address.
721
00:36:13,238 --> 00:36:16,238
it's not actually routing traffic to
it, 'cause it's not an active pod yet.
722
00:36:16,608 --> 00:36:18,228
it'll be in a staged state.
723
00:36:18,678 --> 00:36:20,838
So you stage it up with all
the information, but it has
724
00:36:20,838 --> 00:36:22,278
to be named differently.
725
00:36:22,428 --> 00:36:25,068
And then when you do the cutover,
that's when you switch it from
726
00:36:25,068 --> 00:36:28,668
being inactive to active and
switch the old pod to be inactive.
727
00:36:29,008 --> 00:36:32,398
and that's the final stage when,
the clone pod becomes the primary.
728
00:36:32,848 --> 00:36:36,078
And because it's maintained the
exact, IP address within the
729
00:36:36,078 --> 00:36:38,848
system, it's not losing any traffic.
730
00:36:39,178 --> 00:36:42,538
So the networking system within
Kubernetes routes it to the new node
731
00:36:42,628 --> 00:36:45,698
and the routing tables are updated and
the pod goes to the new destination.
732
00:36:46,087 --> 00:36:50,227
Yeah, that does sound like the hard
part of like the old pod is shut
733
00:36:50,227 --> 00:36:52,417
down so the IP can be released.
734
00:36:52,417 --> 00:36:53,497
I assume the IP.
735
00:36:53,997 --> 00:36:57,057
Can't be taken out while
that old pod is still active.
736
00:36:57,067 --> 00:37:00,427
it's one of these things where it's like,
I understand it at the theory level, but
737
00:37:00,437 --> 00:37:05,717
I have no idea what containerd and coup
proxy and, all these different things that
738
00:37:05,717 --> 00:37:11,337
are binding to a virtual interface and the
order of the things that have to happen
739
00:37:11,337 --> 00:37:16,587
in the exact right sequence in order for
you to first assign that IP to the new
740
00:37:16,737 --> 00:37:19,407
node and then also replay all the packets.
741
00:37:19,457 --> 00:37:22,997
it does seem like a very discreet
order of things that have to happen.
742
00:37:22,997 --> 00:37:25,667
it has to go in a certain order or
they're just all fail it feels like.
743
00:37:25,993 --> 00:37:28,093
so that's the part that took
us about a year to figure out.
744
00:37:29,663 --> 00:37:32,483
there had been a lot of studies
and some research papers around,
745
00:37:32,533 --> 00:37:34,933
the moving of memory and the
snapshotting of different workloads.
746
00:37:34,933 --> 00:37:38,053
Like that part was a little bit
more straightforward because it
747
00:37:38,053 --> 00:37:41,083
was really out of the vMotion
playbook days, from early on.
748
00:37:41,423 --> 00:37:44,183
there were also some college
studies around using cryo to
749
00:37:44,303 --> 00:37:45,893
replicate and migrate containers.
750
00:37:46,393 --> 00:37:49,633
None of them had been able to
solve the IP side of things,
751
00:37:49,633 --> 00:37:50,983
the connectivity side of things.
752
00:37:51,303 --> 00:37:53,493
that's what Cast AI was able to solve for.
753
00:37:53,493 --> 00:37:56,083
and it took a lot of research,
took a lot of, in-depth work.
754
00:37:56,473 --> 00:38:02,143
We started on this, early 2024,
with a team of about five engineers,
755
00:38:02,643 --> 00:38:05,973
deep kernel level Linux engineers,
Kubernetes engineers, people
756
00:38:05,973 --> 00:38:07,603
very familiar with the, code.
757
00:38:07,883 --> 00:38:11,503
they contributed to the Kubernetes
open source project, it was.
758
00:38:12,003 --> 00:38:15,123
10 months before we had a demo
and that was using Calico.
759
00:38:15,153 --> 00:38:18,033
before we could demo, we had to
have a custom AMI at that point in
760
00:38:18,033 --> 00:38:22,363
time in AWS because everything was
kernel level at the a MI level.
761
00:38:22,763 --> 00:38:26,723
we knew that was not feasible going
forward to production, but that
762
00:38:26,723 --> 00:38:29,723
was the first demoable version Like
anything else, There's a lot of warts
763
00:38:29,723 --> 00:38:31,373
and vaporware in the first version.
764
00:38:31,823 --> 00:38:35,873
since then we were able to move
the logic up to a containerd plugin
765
00:38:35,873 --> 00:38:37,493
makes it a lot more portable.
766
00:38:37,493 --> 00:38:39,263
Now it can be applied to different clouds.
767
00:38:39,263 --> 00:38:40,463
It's much less invasive.
768
00:38:40,463 --> 00:38:41,963
You don't need a specific AMI.
769
00:38:42,343 --> 00:38:47,003
and under the hood anymore, we were able
to move it to the A-W-S-V-P-C-C-N-I.
770
00:38:47,033 --> 00:38:48,803
So you don't need the custom Calico CNI.
771
00:38:48,803 --> 00:38:53,073
all of those were iterative steps to
build this and make it more, production
772
00:38:53,073 --> 00:38:55,083
viable and adoptable by the industry.
773
00:38:55,583 --> 00:38:58,583
Now it's a matter of, we've
got kind of two forks going on.
774
00:38:58,703 --> 00:39:02,243
One is continuing to build out
additional platforms, figuring out
775
00:39:02,243 --> 00:39:04,253
cilium, figuring out Azure, CNI,
776
00:39:04,303 --> 00:39:09,153
The other is performance tuning the
existing migrations, reducing time to
777
00:39:09,153 --> 00:39:13,583
migrate, being able to reduce the size
of the deltas down further and further
778
00:39:13,583 --> 00:39:15,413
so we can migrate it faster and faster.
779
00:39:15,723 --> 00:39:17,733
so we've got kind of those
two tracks right now.
780
00:39:17,793 --> 00:39:19,863
the team's up to I think
10 or 12 engineers.
781
00:39:19,893 --> 00:39:23,763
kind of working on those two paths,
and this is probably one of our most
782
00:39:23,763 --> 00:39:27,393
heavily invested, areas in the company
is being able to further this technology.
783
00:39:27,423 --> 00:39:29,003
'cause we see how much value it brings.
784
00:39:29,387 --> 00:39:34,437
Yeah, I imagine it won't be very
long where, you know, this technology
785
00:39:34,497 --> 00:39:38,257
is pretty advanced, but like others
eventually will probably if it's truly
786
00:39:38,287 --> 00:39:39,637
the thing that we all are looking for.
787
00:39:39,637 --> 00:39:43,077
And it sounds like, it feels like it is,
it feels like the kind of, tooling where
788
00:39:43,577 --> 00:39:47,027
it's a hard problem to solve and we'll
maybe see other people attempt to do it.
789
00:39:47,057 --> 00:39:48,467
I mean, the.
790
00:39:48,967 --> 00:39:50,377
the research I had to do for the show.
791
00:39:50,377 --> 00:39:51,097
'cause I was very curious.
792
00:39:51,097 --> 00:39:52,387
I was like, what's the
history of all this?
793
00:39:52,387 --> 00:39:55,287
And we, someone mentioned Sierra
IU, which I believe you're
794
00:39:55,287 --> 00:39:56,367
using at least some of that.
795
00:39:56,707 --> 00:39:59,527
that's a project that's been around
for quite some time over a decade.
796
00:39:59,527 --> 00:40:04,387
And it's not a new idea, but like
a lot of these other technologies,
797
00:40:04,387 --> 00:40:05,707
the devil's in the details.
798
00:40:05,757 --> 00:40:10,167
we never really had an ability, to
capture and understand what a binary's
799
00:40:10,797 --> 00:40:14,647
true dependencies were, whether
it's disc or, networking things.
800
00:40:15,037 --> 00:40:18,567
until we had containers you mentioned
on here like it's in LXC, it's
801
00:40:18,567 --> 00:40:21,497
in Docker, it's in pod man, like
this actually tool is used widely.
802
00:40:21,547 --> 00:40:25,577
it's just maybe not, well known
to us end users because it's
803
00:40:25,577 --> 00:40:27,257
packaged as a part of other tooling.
804
00:40:27,677 --> 00:40:29,687
And I can sort of see a world where.
805
00:40:30,057 --> 00:40:34,017
if this becomes more widespread, you're
gonna end up with haves and have nots
806
00:40:34,017 --> 00:40:38,327
where my solution doesn't have live
migration or my solution does have live
807
00:40:38,327 --> 00:40:41,907
migration at some point, maybe it's
ubiquitous, you're building functionality
808
00:40:41,907 --> 00:40:45,787
on top of it, like your automation that
truly adds value around, spot instances
809
00:40:45,787 --> 00:40:49,807
where my company maybe has never done spot
instances because it was too risky for
810
00:40:49,807 --> 00:40:53,677
us and we didn't have the tooling to take
advantage of it without risking downtime.
811
00:40:53,727 --> 00:40:56,367
I definitely have a couple of clients
that I've worked with over the last
812
00:40:56,367 --> 00:40:59,377
couple of years that are like that,
where they're a hundred percent reserved
813
00:40:59,377 --> 00:41:03,387
instances because they want the cheapest,
but they also need to guarantee uptime.
814
00:41:03,387 --> 00:41:07,107
And they can't do that at a level
that live migrations would provide.
815
00:41:07,107 --> 00:41:11,477
So they have to pay that extra
surcharge, for avoiding, ephemeral
816
00:41:11,477 --> 00:41:12,737
instances and stuff like that.
817
00:41:12,787 --> 00:41:15,757
to me, it gives me comfort
that the
technology stack.
818
00:41:16,117 --> 00:41:18,937
is part open source,
part community driven.
819
00:41:18,937 --> 00:41:23,097
there's also the product, and private IP
side of this as well, but it's not like
820
00:41:23,097 --> 00:41:24,647
you're reinventing the Linux kernel.
821
00:41:24,647 --> 00:41:28,937
It wouldn't have been that long ago where
you had to actually throw in a Linux
822
00:41:28,937 --> 00:41:32,707
module or kernel module rather, that
would only work on certain operating
823
00:41:32,707 --> 00:41:36,037
system distributions of Linux and
that you would have to deploy a custom
824
00:41:36,367 --> 00:41:38,667
ISO that wasn't that far in the past.
825
00:41:38,667 --> 00:41:41,937
But now that we've got all these modern
things, I don't know if EBPF is involved
826
00:41:41,937 --> 00:41:45,792
in this at all, but we've got more modern
abstractions that it feels like you can
827
00:41:45,792 --> 00:41:48,387
just plug and play as long as you've
got the right networking components.
828
00:41:48,437 --> 00:41:51,677
From an engineering perspective,
pretty awesome because it allows you
829
00:41:51,677 --> 00:41:53,057
to build stuff on the stack like this.
830
00:41:53,107 --> 00:41:55,837
the team's not on the call, but the
team that's developing this, good job,
831
00:41:55,837 --> 00:41:57,767
Bravo, that's, some great engineering.
832
00:41:57,797 --> 00:42:02,087
obviously anytime something is a year
long effort to crack a nut like this, I
833
00:42:02,087 --> 00:42:05,497
feel sad for the people that had a six
month, like, no one's gonna see this
834
00:42:05,497 --> 00:42:08,587
feature for a year and I'm gonna work all
year on it and I hope someone likes it.
835
00:42:08,587 --> 00:42:12,237
So that's from a software development
perspective, that's the hard part.
836
00:42:12,237 --> 00:42:13,967
That's the true engineering.
837
00:42:14,517 --> 00:42:14,907
Um.
838
00:42:14,943 --> 00:42:15,783
Yeah, absolutely.
839
00:42:16,230 --> 00:42:19,590
we could talk about this forever,
but people have their jobs to do.
840
00:42:19,590 --> 00:42:19,800
okay.
841
00:42:19,800 --> 00:42:22,470
How do people get started do they just
go sign up for Cast AI and this is
842
00:42:22,470 --> 00:42:25,350
like a feature outta the box that they
can implement in their clusters or.
843
00:42:25,607 --> 00:42:26,447
Yep, absolutely.
844
00:42:26,497 --> 00:42:30,307
it's in the UI now, so if people
want to sign up and onboard, we do
845
00:42:30,307 --> 00:42:33,637
recommend having somebody on our sales
engineering team work with folks.
846
00:42:33,637 --> 00:42:36,747
So reach out to us, We'll also reach
out when people sign up as well.
847
00:42:37,067 --> 00:42:38,327
it's all straightforward.
848
00:42:38,327 --> 00:42:39,107
There's no caveats.
849
00:42:39,107 --> 00:42:42,617
It's helm charts to do the install
and then you set up the autoscaler.
850
00:42:42,987 --> 00:42:46,017
we will be adding support
for Karpenter today.
851
00:42:46,017 --> 00:42:48,817
It's using our autoscaler,
but we will support Karpenter.
852
00:42:48,867 --> 00:42:50,967
around end of Q4 or early Q1,
853
00:42:51,687 --> 00:42:52,317
Yeah, that's great.
854
00:42:52,387 --> 00:42:56,037
my usual co-host is with AWS, so
they would greatly appreciate that.
855
00:42:56,107 --> 00:42:58,167
I know that over the last
year, Karpenter's been
856
00:42:58,167 --> 00:42:58,987
out, a little over a year.
857
00:42:59,377 --> 00:43:02,807
We've had, a surprising number
of people on our Discord server.
858
00:43:02,957 --> 00:43:05,817
For those of you watching, there's
a Discord server you can join.
859
00:43:06,087 --> 00:43:08,367
There's a lot of people on Discord
talking about using Karpenter.
860
00:43:08,417 --> 00:43:10,747
I'm really impressed with
the uptake on that project.
861
00:43:10,747 --> 00:43:14,887
And in case you're wondering what
Karpenter is, it's with a K and it's
862
00:43:14,887 --> 00:43:18,387
for Kubernetes, you can look on this
YouTube channel later because we did a
863
00:43:18,392 --> 00:43:21,642
show on it and has had people talking
about it on the show and demoing it.
864
00:43:21,712 --> 00:43:22,762
whenever it was released.
865
00:43:22,762 --> 00:43:23,962
I think that was 2024.
866
00:43:23,962 --> 00:43:24,922
I can't remember exactly.
867
00:43:25,422 --> 00:43:27,642
alright, so everyone
knows how to get started.
868
00:43:27,642 --> 00:43:31,182
Everyone now knows that they wish they
had live migrations and they currently
869
00:43:31,182 --> 00:43:32,982
don't unless they're a Cast AI customer.
870
00:43:33,292 --> 00:43:34,432
Where can we find you on the internet?
871
00:43:34,432 --> 00:43:37,102
Where can people learn more
about what you're doing?
872
00:43:37,102 --> 00:43:38,632
Are you gonna be at conferences soon?
873
00:43:39,022 --> 00:43:41,512
I'm assuming, cast is probably
gonna have a booth at KubeCon again.
874
00:43:41,512 --> 00:43:42,627
They always seemed to have a booth there.
875
00:43:43,127 --> 00:43:44,517
we've got a big booth
at KubeCon this year.
876
00:43:44,537 --> 00:43:45,797
I think we've got a 20 by 20.
877
00:43:45,797 --> 00:43:48,137
We're gonna be doing demos and
presentations in the booth.
878
00:43:48,187 --> 00:43:49,507
this is gonna be a big part of that.
879
00:43:49,877 --> 00:43:51,647
we'll also be at reinvent in Vegas.
880
00:43:52,037 --> 00:43:54,917
In later, early December, I
guess is first week of December.
881
00:43:55,227 --> 00:43:56,727
so I'll be at both of those events.
882
00:43:56,727 --> 00:43:59,757
I'm also really active on LinkedIn, so
if anybody wants to reach out to me on
883
00:43:59,757 --> 00:44:03,207
LinkedIn, if you wanna set up a session
just to go into more detail, feel free
884
00:44:03,207 --> 00:44:06,527
to ping me I post a lot of Kubernetes
content in general, best practices,
885
00:44:06,737 --> 00:44:10,277
things that we see in the industry from
a Kubernetes evolution side of things,
886
00:44:10,277 --> 00:44:12,377
and also obviously a bunch of cast stuff.
887
00:44:12,717 --> 00:44:15,717
so, you know, feel free to follow or
connect Happy to share more information
888
00:44:15,717 --> 00:44:16,197
Awesome.
889
00:44:16,657 --> 00:44:20,527
well, I'm looking forward to hearing about
that continual proliferation of all things
890
00:44:20,527 --> 00:44:22,387
live migration on every possible setup.
891
00:44:22,727 --> 00:44:26,677
someday it'll be on FreeBSD with
some esoteric, Kubernetes variant.
892
00:44:26,987 --> 00:44:29,137
It's, pretty cool to see
the evolution of this.
893
00:44:29,557 --> 00:44:31,717
Well, thank you both for
being here, Philip and Dan.
894
00:44:31,777 --> 00:44:32,407
see you all later.
895
00:44:32,907 --> 00:44:33,267
Ciao.
896
00:44:33,767 --> 00:44:36,017
Thanks for watching, and I'll
see you in the next episode.