Is Docker Building the Best AI Stack?

1
00:00:04,333 --> 00:00:08,053
This is DevOps and Docker talk,
and I am your host, Bret Fisher.

2
00:00:08,413 --> 00:00:10,483
This was a fun episode this week.

3
00:00:10,483 --> 00:00:14,743
Nirmal Mehta is back from
AWS and our guest this week

4
00:00:14,773 --> 00:00:17,443
was Michael Irwin of Docker.

5
00:00:17,443 --> 00:00:22,183
He is a recurring guest if you've been
listening to this podcast for any length

6
00:00:22,183 --> 00:00:25,743
of time, I think we had him on earlier
this year, and he's been a friend for a

7
00:00:25,743 --> 00:00:30,693
decade, former Docker captain, now Docker
employee advocating for Docker everywhere.

8
00:00:30,693 --> 00:00:34,233
He's all over the place and, I
always have him on because he

9
00:00:34,233 --> 00:00:39,109
breaks stuff down to help us all
understand what's going on at Docker.

10
00:00:39,199 --> 00:00:45,109
And there is a giant list of things we had
to talk about this week, all AI related,

11
00:00:45,139 --> 00:00:49,789
all separate pieces of the puzzle that
can be used independently or together.

12
00:00:49,969 --> 00:00:54,259
So in this episode, we will be covering
the Docker Model Runner, which I've

13
00:00:54,259 --> 00:00:58,609
talked about a lot on this show over
the last four months, for running open

14
00:00:58,669 --> 00:01:03,109
or free models locally or remotely on
servers or on your machine, wherever.

15
00:01:03,229 --> 00:01:05,599
Basically you get to run your own
models and Docker will host them for

16
00:01:05,599 --> 00:01:10,289
you, Docker will encapsulate and allow
you to use the Docker CLI to manage it.

17
00:01:10,589 --> 00:01:15,539
Then we have the hub model catalogs,
so you can pick one of the dozens of

18
00:01:15,539 --> 00:01:19,649
models I guess, that are available in
Docker Hub and we talk about Gordon

19
00:01:19,649 --> 00:01:24,509
ai, which is their chat bot built into
Docker Desktop and the Docker CLI.

20
00:01:25,059 --> 00:01:31,509
We then get into MCP toolkit and the
hubs MCP catalog, and how to bring all

21
00:01:31,509 --> 00:01:37,504
our tools into our local ai or like,
using some other AI and you're just

22
00:01:37,504 --> 00:01:39,124
wanting to use your own MCP tools.

23
00:01:39,334 --> 00:01:43,204
We talk about how Docker manages
all that and it uses the MCP gateway

24
00:01:43,204 --> 00:01:47,764
that they recently open sourced
to front end all those tools and

25
00:01:47,764 --> 00:01:49,534
to help you manage your MCP tools.

26
00:01:50,074 --> 00:01:56,554
We also then get into compose and how
you add models and the MCP gateway plus

27
00:01:56,554 --> 00:02:00,049
MCP tools, all into your composed files.

28
00:02:00,319 --> 00:02:03,979
Then we talk about how to use that
for building agents a little bit

29
00:02:04,089 --> 00:02:10,209
And then Offload, which allows
you to build, run containers,

30
00:02:10,209 --> 00:02:15,999
run Docker models, all in Docker
Cloud, and they call it Offload

31
00:02:15,999 --> 00:02:18,264
It actually is a pretty unique name.

32
00:02:18,264 --> 00:02:21,564
Most other people just called it Docker
Cloud, but they call it Docker Offload.

33
00:02:21,834 --> 00:02:22,464
Great name.

34
00:02:22,464 --> 00:02:25,254
You just toggle inside your
Docker UI and you're good to go.

35
00:02:25,254 --> 00:02:26,064
Everything's remote.

36
00:02:26,154 --> 00:02:31,764
And then we at the very end talked
about Compose Now works at least

37
00:02:31,764 --> 00:02:35,364
the YAML files of Compose, now
work inside of Google Cloud Run.

38
00:02:35,704 --> 00:02:37,744
Probably coming to other places soon.

39
00:02:37,924 --> 00:02:43,504
So we go for almost 90 minutes on
this show because for good reason, we

40
00:02:43,534 --> 00:02:49,904
talk about the use cases of all these
different various AI parts of the puzzle.

41
00:02:50,264 --> 00:02:55,814
And I'm excited because it paints,
I hope for you a complete picture

42
00:02:56,114 --> 00:03:00,579
of what Docker has released, how it
all works together, when you would

43
00:03:00,579 --> 00:03:04,149
choose each part for problem solving,
different problems and all that.

44
00:03:04,359 --> 00:03:08,709
So please enjoy this episode with
Nirmal and Michael Irwin of Docker.

45
00:03:10,863 --> 00:03:11,533
Hi!

46
00:03:12,020 --> 00:03:13,071
Hey, Awesome.

47
00:03:13,444 --> 00:03:14,144
I'm doing all right.

48
00:03:14,174 --> 00:03:17,255
we have a special guest
Michael, another Virginian.

49
00:03:17,265 --> 00:03:17,955
Hello.

50
00:03:18,699 --> 00:03:19,179
Hello,

51
00:03:19,472 --> 00:03:21,222
we all have known each other a decade.

52
00:03:21,242 --> 00:03:22,592
So Michael, tell us who you are

53
00:03:23,052 --> 00:03:25,302
so I'm, Michael Irwin, I work at Docker.

54
00:03:25,302 --> 00:03:29,896
I'm in our developer success team,
and do a lot of teaching, training,

55
00:03:29,896 --> 00:03:34,966
education, breaking down the stuff,
trying to make it somewhat understandable.

56
00:03:35,056 --> 00:03:38,936
I've been with Docker for almost
three and a half years now, so it's

57
00:03:38,936 --> 00:03:41,726
gone pretty quickly, but, it's fun
to spend time with the community

58
00:03:41,776 --> 00:03:43,436
and just help developers out.

59
00:03:43,836 --> 00:03:47,006
Learn about this stuff, but also
how do you actually take advantage

60
00:03:47,006 --> 00:03:49,086
and do cool stuff with it?

61
00:03:49,385 --> 00:03:50,115
That's awesome.

62
00:03:50,145 --> 00:03:53,705
I mean, this is like a new wave
of, essentially development

63
00:03:53,715 --> 00:03:56,515
tooling and, it's pretty exciting.

64
00:03:56,915 --> 00:03:57,275
Yeah.

65
00:03:57,275 --> 00:03:58,265
we've got a list people.

66
00:03:58,365 --> 00:04:01,765
I only make lists on this
show a couple of times a year.

67
00:04:02,015 --> 00:04:05,495
Yeah, it does seem like every time I'm
on the show, there's a list involved.

68
00:04:06,065 --> 00:04:07,685
yeah, that's a pretty decent list.

69
00:04:07,965 --> 00:04:10,965
let's break it down because we got
a lot to get through and we realize

70
00:04:10,975 --> 00:04:13,895
that on Docker's channel and on
this channel, They've been talking

71
00:04:13,895 --> 00:04:15,175
about all these fun features.

72
00:04:15,205 --> 00:04:17,425
I've talked at length
about Docker Model Runner.

73
00:04:17,835 --> 00:04:23,155
Hub Catalog, I don't have a bunch of
videos on MCP or the MCP tools, but

74
00:04:23,725 --> 00:04:25,955
actually we've had streams on it before.

75
00:04:26,005 --> 00:04:30,605
But, I think the MCP toolkit and the
MCP gateway I'm most excited about that.

76
00:04:30,605 --> 00:04:31,935
So we're going to get
to that in a little bit.

77
00:04:32,329 --> 00:04:35,789
I think, technically, the first
thing out of the gate was Gordon AI.

78
00:04:36,794 --> 00:04:38,704
this to me is.

79
00:04:39,439 --> 00:04:44,509
From a user's perspective, it's a Docker
focused or maybe a developer focused

80
00:04:44,579 --> 00:04:52,219
AI chatbot, essentially similar to
ChatGPT, but it's in my Docker interface.

81
00:04:52,279 --> 00:04:56,329
if I'm staring at the Docker desktop
interface, there's an ask Gordon button.

82
00:04:56,929 --> 00:05:00,499
And if you've never clicked that,
or if you clicked it once a couple

83
00:05:00,499 --> 00:05:03,789
years ago, it has changed a lot.

84
00:05:03,849 --> 00:05:09,685
There have been enhancements so o now
we have memory and threads and it saves

85
00:05:09,685 --> 00:05:14,355
me from having to go to ChatGPT when
I'm working, specifically around Docker

86
00:05:14,355 --> 00:05:18,505
stuff, but anything it feels like I
can ask it developer related stuff

87
00:05:18,795 --> 00:05:21,275
Can you tell me now, is
this free for everyone?

88
00:05:21,275 --> 00:05:24,675
How does this play out in
terms of who can use this

89
00:05:24,675 --> 00:05:26,655
Yeah, so everybody can
access it right now.

90
00:05:26,655 --> 00:05:29,665
The only limitations may be
some of our business orgs.

91
00:05:29,705 --> 00:05:32,065
those orgs have to, enable it.

92
00:05:32,535 --> 00:05:35,825
We're just not going to roll out all
the AI stuff as most organizations

93
00:05:35,825 --> 00:05:37,155
are pretty cautious about that.

94
00:05:37,442 --> 00:05:39,522
but yeah, Gordon's available to everybody.

95
00:05:39,572 --> 00:05:43,562
it started off actually mostly as a
documentation help of just help me

96
00:05:43,572 --> 00:05:46,802
answer questions and keep up with
new features and that kind of stuff.

97
00:05:47,292 --> 00:05:51,472
But, as you've noted, we've added new
capabilities and new tools along the way.

98
00:05:51,937 --> 00:05:55,961
I was doing some experiments just
the other day of Hey Gordon, help

99
00:05:55,961 --> 00:06:00,601
me convert this Dockerfile to use
the new Docker hardened images.

100
00:06:00,611 --> 00:06:05,511
And it would do the conversion, find the
images that my organization has that match

101
00:06:05,511 --> 00:06:07,481
the one that I'm using this Dockerfile.

102
00:06:07,531 --> 00:06:10,981
you're starting to see more and more
capabilities built into it as well it's

103
00:06:10,981 --> 00:06:12,681
a pretty fun little assistant there.

104
00:06:13,057 --> 00:06:13,567
Yeah,

105
00:06:13,694 --> 00:06:18,714
highly recommend folks that are listening
that if you've not touched Docker and

106
00:06:18,714 --> 00:06:23,714
you just open it up, check out Gordon
and ask those questions that you probably

107
00:06:23,724 --> 00:06:27,104
have, compose all the things you could
probably put that in there and Gordon will

108
00:06:27,104 --> 00:06:28,804
probably try to compose all the things.

109
00:06:28,857 --> 00:06:32,887
you know what one of my common uses for
this is I need to translate a docker

110
00:06:32,887 --> 00:06:36,477
run command into a compose file or
back and forth like, please give me the

111
00:06:36,477 --> 00:06:40,807
docker run equivalent of this compose
file or please turn these docker run

112
00:06:40,807 --> 00:06:46,277
commands or this docker build command
into a compose file and it saves me.

113
00:06:46,677 --> 00:06:49,967
It's not, I mean, I've been doing this
stuff almost every day for a decade,

114
00:06:50,207 --> 00:06:55,697
so it's not like I needed to do it for
me, but it's still faster than a pro

115
00:06:55,747 --> 00:06:58,847
typing it in from memory, I'm at the
point now where I rarely need to refer

116
00:06:58,897 --> 00:07:03,067
to the docs, but it's still faster
than me at writing a Compose file

117
00:07:03,067 --> 00:07:04,467
out and saving it to the hard drive.

118
00:07:04,867 --> 00:07:08,757
You know, because sometimes we debate
around whether AI tools are useful for

119
00:07:08,767 --> 00:07:13,887
senior versus junior and blah, blah, blah,
and I don't know, as a senior, it saves

120
00:07:13,887 --> 00:07:17,727
me keystrokes, it saves me time, unlike
when it first came out a couple years

121
00:07:17,727 --> 00:07:23,562
ago, I have Tracked it, given feedback to
the team and it no longer makes Compose

122
00:07:23,562 --> 00:07:26,262
files with old Compose information.

123
00:07:26,262 --> 00:07:27,532
Like it, at least in the

124
00:07:27,721 --> 00:07:28,591
the version tag.

125
00:07:28,772 --> 00:07:29,192
Yeah.

126
00:07:29,192 --> 00:07:30,632
Cause it, the version tag.

127
00:07:30,642 --> 00:07:31,042
Yeah.

128
00:07:31,142 --> 00:07:33,772
I was a big proponent of Hey, this
hasn't been in there for five years.

129
00:07:33,772 --> 00:07:34,802
Why is it recommending it?

130
00:07:35,192 --> 00:07:36,122
and they fixed that.

131
00:07:36,132 --> 00:07:38,072
So, I'm very appreciative of it.

132
00:07:38,492 --> 00:07:41,042
We're going to save it, we're going
to talk about MCP tools, and then

133
00:07:41,042 --> 00:07:43,342
we're going to come back to this,
because it gets, it's gotten better,

134
00:07:43,592 --> 00:07:45,672
and one of the reasons it's gotten
better is it can talk to tools.

135
00:07:45,922 --> 00:07:49,382
So I don't want to really spoil that,
But yeah, I love the suggestions because

136
00:07:49,382 --> 00:07:53,625
sometimes when you're staring at a blank
chat box, it feels kind of like being a

137
00:07:53,625 --> 00:07:55,625
writer and staring at a blank document.

138
00:07:55,685 --> 00:07:57,845
It's like, I know this is
supposed to be a cool tool, but

139
00:07:57,845 --> 00:07:59,095
I don't know what to do with it.

140
00:07:59,125 --> 00:07:59,775
do I start?

141
00:07:59,885 --> 00:08:00,255
Yeah.

142
00:08:00,271 --> 00:08:00,951
analogy

143
00:08:01,375 --> 00:08:04,148
there's even some cool spots, like if
you go to the containers tab, You'll

144
00:08:04,148 --> 00:08:08,428
see on the right side a little like
magic icon there, and then you can

145
00:08:08,438 --> 00:08:10,338
ask questions about that container.

146
00:08:10,338 --> 00:08:13,348
So if you have a container that's
failing to start or whatever, you can

147
00:08:13,538 --> 00:08:17,668
basically start a thread around the
context of why is this thing not starting.

148
00:08:17,906 --> 00:08:18,386
Interesting.

149
00:08:18,386 --> 00:08:19,616
I have not tried that.

150
00:08:20,256 --> 00:08:23,506
it actually reminds me of the little
star we get for Gemini that's in every

151
00:08:23,506 --> 00:08:25,066
single Google app that I'm opening.

152
00:08:25,066 --> 00:08:29,896
I guess we're all, slowly converging
on the starlight AI icons now, I

153
00:08:29,896 --> 00:08:32,336
would have thought it would have
been a robot face, but we chose this

154
00:08:32,346 --> 00:08:34,666
vague collection of stars and pluses.

155
00:08:34,666 --> 00:08:35,646
I and that is pretty cool.

156
00:08:35,656 --> 00:08:38,336
I, so is there more, is
it like an images as well?

157
00:08:38,396 --> 00:08:40,746
So yeah, anywhere you see
that icon, you can start a

158
00:08:40,746 --> 00:08:42,636
conversation around that thing.

159
00:08:42,997 --> 00:08:45,137
I'm going to start clicking on that
more often to see what it tells me.

160
00:08:45,137 --> 00:08:47,307
Like, Hey, what's the, what data.

161
00:08:47,652 --> 00:08:49,402
Ooh, How do I back up this volume?

162
00:08:49,432 --> 00:08:50,332
That's, that is pretty cool.

163
00:08:50,332 --> 00:08:50,542
So they're

164
00:08:50,778 --> 00:08:52,568
that is the number one question

165
00:08:52,736 --> 00:08:54,386
I love that they're context aware.

166
00:08:54,386 --> 00:08:57,446
Like they know that this is a
volume, or at least the prompts.

167
00:08:57,956 --> 00:09:00,912
it's like prompt suggestions,
which This feels a little meta.

168
00:09:00,912 --> 00:09:03,162
Is the AI suggesting prompts for the ai?

169
00:09:03,242 --> 00:09:06,152
It's not in this case,
but it certainly could.

170
00:09:08,053 --> 00:09:08,433
All right.

171
00:09:08,973 --> 00:09:10,573
so that's ask Gordon,

172
00:09:10,673 --> 00:09:10,763
Woo!

173
00:09:11,283 --> 00:09:11,693
Alright.

174
00:09:11,976 --> 00:09:12,966
And that is what you said.

175
00:09:13,164 --> 00:09:15,614
that is available to everyone
running Docker Desktop.

176
00:09:15,804 --> 00:09:19,114
Is that, to be clear, that's
not available for people running

177
00:09:19,124 --> 00:09:21,404
Docker Engine on Linux, right?

178
00:09:21,424 --> 00:09:25,494
That is not There is a
CLI to it, isn't there?

179
00:09:25,594 --> 00:09:26,484
There's Docker AI.

180
00:09:26,705 --> 00:09:29,595
but that's part of the
Docker Desktop installation.

181
00:09:29,595 --> 00:09:32,435
that's a CLI plugin and it's going
to talk to the components that are

182
00:09:32,435 --> 00:09:34,115
bundled in Docker Desktop there.

183
00:09:34,165 --> 00:09:34,465
Yeah.

184
00:09:34,505 --> 00:09:37,895
so we have our, this is an AI
that's outsourced to Docker.

185
00:09:37,895 --> 00:09:38,795
It's not running locally.

186
00:09:38,825 --> 00:09:41,265
it's just calling APIs from
the Docker desktop interface.

187
00:09:41,315 --> 00:09:47,165
So then if I don't want to use Gordon
AI, but I want to run my own AI models

188
00:09:47,175 --> 00:09:52,295
locally, which I feel like this is
pretty niche because a lot of people I

189
00:09:52,295 --> 00:09:58,645
talk to, their GPU is underwhelming, and
they don't have an M4 Mac or a NVIDIA

190
00:09:58,655 --> 00:10:02,035
desktop tower with a giant GPU in it.

191
00:10:02,065 --> 00:10:05,185
granted, I guess there are
models that run on CPUs, but I

192
00:10:05,185 --> 00:10:07,195
am not the model expert here.

193
00:10:07,405 --> 00:10:11,075
I've been trying to catch up this year
because I actually do have a decent

194
00:10:11,075 --> 00:10:15,345
laptop now with an M4 in it so I can
run at least some of the Apple models.

195
00:10:15,675 --> 00:10:18,415
But this Docker model runner.

196
00:10:18,818 --> 00:10:22,318
it's a pretty cool feature, but
you can pick your models, so

197
00:10:22,318 --> 00:10:23,458
you can pick a really small one.

198
00:10:24,458 --> 00:10:28,178
On Windows, it has to be an
NVIDIA GPU on Windows to work.

199
00:10:28,188 --> 00:10:28,828
Is that right?

200
00:10:29,272 --> 00:10:32,542
So it supports both NVIDIA and
Adreno, so if you've got a

201
00:10:32,572 --> 00:10:35,298
Qualcomm chip, it'll work there.

202
00:10:35,668 --> 00:10:39,238
Okay, and on Mac, it just uses system
memory because that's how Macs work.

203
00:10:39,238 --> 00:10:41,768
They have the universal memory, right?

204
00:10:42,188 --> 00:10:45,588
And I mean, for me, it's been
great because I have a brand

205
00:10:45,588 --> 00:10:46,908
new machine with 48 gig of RAM.

206
00:10:46,948 --> 00:10:48,378
I know that is not normal.

207
00:10:48,808 --> 00:10:54,395
but these models, Mac, it can be a little
problematic because it's like, if I have

208
00:10:54,395 --> 00:10:57,475
a bunch of browser tabs open, I can't
run the full size model that I want to

209
00:10:57,475 --> 00:11:00,545
run, because that's one of the problems
is like, it's all using the same thing.

210
00:11:00,545 --> 00:11:03,635
So I have to do like we did back with
VMs, I have to shut all my apps down

211
00:11:03,975 --> 00:11:06,555
and then run, because I always want
to run the biggest model possible.

212
00:11:06,555 --> 00:11:09,390
So I have this like 40, 46 gig model.

213
00:11:09,390 --> 00:11:11,820
I think it's the Devstral new
model that's supposed to be

214
00:11:11,820 --> 00:11:13,110
great for local development.

215
00:11:13,570 --> 00:11:16,550
So for those of you listening, if you
want to go in depth, we're not going to

216
00:11:16,580 --> 00:11:20,440
have time for that, but it is, Docker
Model Runner lets you run models locally.

217
00:11:20,820 --> 00:11:23,820
you can technically run this now
on Docker infrastructure, right?

218
00:11:23,820 --> 00:11:25,070
Which we're going to
get to in a little bit.

219
00:11:25,759 --> 00:11:27,089
dun dun.

220
00:11:27,130 --> 00:11:27,450
Okay.

221
00:11:27,600 --> 00:11:29,290
yeah, there is this thing called offload.

222
00:11:29,380 --> 00:11:31,010
Michael's going to know
way more about that.

223
00:11:31,010 --> 00:11:33,850
Cause that's a really new feature,
but you don't have to actually

224
00:11:33,850 --> 00:11:34,970
run these models locally now.

225
00:11:34,970 --> 00:11:36,290
Like you could offload them.

226
00:11:36,290 --> 00:11:36,530
Right.

227
00:11:36,930 --> 00:11:37,240
Okay.

228
00:11:37,540 --> 00:11:38,530
What are you seeing?

229
00:11:38,530 --> 00:11:42,206
are people coming to Docker, like trying
to run the biggest models possible?

230
00:11:42,206 --> 00:11:45,446
They got like multiple GPUs or are you
just seeing people kind of tinkering out?

231
00:11:45,546 --> 00:11:46,556
What's some analogies there?

232
00:11:46,995 --> 00:11:48,655
there's a little bit of everything.

233
00:11:48,835 --> 00:11:52,785
I'd say a lot of folks are still exploring
the space and figure out what's the

234
00:11:52,785 --> 00:11:54,765
right ways of doing things as well too.

235
00:11:54,765 --> 00:11:58,205
So, one of the interesting things is,
and one of the things to keep in mind

236
00:11:58,775 --> 00:12:02,585
is that Let's break outta the AI space.

237
00:12:02,585 --> 00:12:04,595
when you say, for example, a database.

238
00:12:04,625 --> 00:12:04,805
Okay.

239
00:12:04,805 --> 00:12:05,945
I'm using Postgres database.

240
00:12:06,365 --> 00:12:10,585
If I'm using a Postgres database, a
managed offering out in the cloud.

241
00:12:10,645 --> 00:12:10,795
Okay.

242
00:12:10,795 --> 00:12:11,965
Let's just say RDS.

243
00:12:12,115 --> 00:12:12,355
Okay.

244
00:12:13,155 --> 00:12:17,085
during development I can run a Postgres
container and yeah, it's a smaller

245
00:12:17,085 --> 00:12:18,795
version and it's not a managed offering.

246
00:12:19,075 --> 00:12:21,595
and that works because all the
binary protocols are the same.

247
00:12:21,595 --> 00:12:22,915
I can basically just swap out.

248
00:12:23,120 --> 00:12:24,870
My connection URL, and it just works.

249
00:12:25,360 --> 00:12:26,690
But models are a little different.

250
00:12:27,130 --> 00:12:30,430
And so, I've seen some folks that
have been like, I'm going to use a

251
00:12:30,450 --> 00:12:34,810
smaller version of a model during local
development, but then when I deploy,

252
00:12:34,840 --> 00:12:39,637
I'm going to use the larger hosted
model that my cloud provider provides.

253
00:12:39,727 --> 00:12:43,937
Okay, so maybe it's a parameter
change, I'm using fewer parameters

254
00:12:43,937 --> 00:12:46,347
so it can actually run locally,
and then use the larger version.

255
00:12:47,977 --> 00:12:52,497
The analogy of, okay, me running a
local PostgreSQL container and me

256
00:12:52,497 --> 00:12:58,117
running a local model, if it's a
different model, it's a different model.

257
00:12:58,707 --> 00:13:02,347
And the results that you get from that
model are going to vary quite a bit.

258
00:13:02,627 --> 00:13:05,077
And so that's one of the things that
we have to kind of keep reminding

259
00:13:05,077 --> 00:13:08,562
people if you use for example a four
billion parameter, So, we're going to

260
00:13:08,562 --> 00:13:09,838
talk a little bit about how you can use
this model during local development.

261
00:13:09,838 --> 00:13:11,206
Yes, it can fit on your machine, but
then if you deploy and you're using

262
00:13:11,206 --> 00:13:16,047
the 32 billion parameter version in
production, those are very different

263
00:13:16,057 --> 00:13:18,627
models and you're going to get very
different interactions and different

264
00:13:18,637 --> 00:13:20,177
outputs from the models there.

265
00:13:20,177 --> 00:13:23,457
So, it's something to keep in
mind as folks are looking at

266
00:13:23,717 --> 00:13:24,707
building their applications.

267
00:13:26,157 --> 00:13:28,617
So, where are we seeing folks use this?

268
00:13:28,657 --> 00:13:33,132
Well Of course, if you can use
the same model across your entire

269
00:13:33,282 --> 00:13:35,822
software development life cycle,
then that works out pretty well.

270
00:13:36,322 --> 00:13:40,742
but we're also starting to see a little
bit of a rise of, using basically

271
00:13:40,752 --> 00:13:45,092
fine tuned use case specific models
or folks training their own models.

272
00:13:45,422 --> 00:13:48,442
and using those for the
specific application.

273
00:13:48,832 --> 00:13:53,422
and again, that's tends to be a little
bit smaller, more use case specific.

274
00:13:53,652 --> 00:13:55,892
And then, yes, that
makes sense to use there.

275
00:13:56,242 --> 00:13:58,312
okay, I need to be able
to run that on my own.

276
00:13:58,722 --> 00:13:59,392
et cetera.

277
00:13:59,442 --> 00:14:02,152
again, I think a lot of folks are
still feeling out the space and

278
00:14:02,152 --> 00:14:05,362
figuring out exactly how should I
think about how should I use this?

279
00:14:05,702 --> 00:14:08,532
and of course, the tooling has to
exist kind of before you can actually

280
00:14:08,532 --> 00:14:10,102
do a lot of those experiments.

281
00:14:10,102 --> 00:14:13,012
So, in many ways, we've been
building out that tooling to help

282
00:14:13,112 --> 00:14:14,582
support that experimentation.

283
00:14:15,052 --> 00:14:17,792
but I think in many ways, folks are still
figuring out exactly what this is going

284
00:14:17,792 --> 00:14:19,242
to look like for them going forward.

285
00:14:19,647 --> 00:14:20,047
Yeah.

286
00:14:20,361 --> 00:14:23,531
I'm trying to imagine what enterprises are
doing and building out and I'm imagining

287
00:14:23,531 --> 00:14:29,421
it's not like this, but one of my
imaginations is reminding me of 20 years

288
00:14:29,421 --> 00:14:33,581
ago, buying a Google box, which I don't
remember the name of, they called it, but

289
00:14:33,581 --> 00:14:37,331
it was this appliance you would put in
your data center that was, it was yellow.

290
00:14:37,381 --> 00:14:37,891
Racked

291
00:14:38,107 --> 00:14:39,067
search appliance.

292
00:14:39,491 --> 00:14:39,771
you go.

293
00:14:39,771 --> 00:14:40,071
Yeah.

294
00:14:40,351 --> 00:14:43,381
And I don't know if Nirmal, if you had
any customers back then with, if you

295
00:14:43,381 --> 00:14:44,811
were a consultant back then, but that,

296
00:14:45,047 --> 00:14:46,507
I can't name those customers.

297
00:14:46,831 --> 00:14:50,921
so I can, I was in, the city of
Virginia beach running the IT and

298
00:14:50,921 --> 00:14:53,131
the data center there, or at least
running the engineering groups.

299
00:14:53,461 --> 00:14:57,771
And those were back in the days
where we didn't want Google indexing

300
00:14:57,771 --> 00:14:59,481
our internal infrastructure.

301
00:14:59,631 --> 00:15:02,481
You don't want your internal data to be.

302
00:15:02,784 --> 00:15:07,154
accessed or used potentially
by this big IT conglomerate.

303
00:15:07,434 --> 00:15:07,674
Yeah.

304
00:15:07,674 --> 00:15:11,724
And so you bring, they, they sell you
a, on prem box and you put it in your

305
00:15:11,724 --> 00:15:14,894
data center and it would scan and have
all access to everything you could give

306
00:15:14,894 --> 00:15:18,384
it, back when apps didn't really have
their own search, Google was providing

307
00:15:18,384 --> 00:15:22,014
that for us, and they would index
our email and index our file servers

308
00:15:22,014 --> 00:15:24,724
and databases if we wanted to give
it access to them I'm not sure that's

309
00:15:24,734 --> 00:15:29,914
around anymore, but, at a time, Google
wasn't going to give away their software

310
00:15:29,914 --> 00:15:31,344
and we didn't all know how to run it.

311
00:15:31,694 --> 00:15:33,434
And, I'm sure it was running on Linux.

312
00:15:33,434 --> 00:15:37,864
And at the time in the mid 2000s,
we weren't yet Linux at the city.

313
00:15:38,624 --> 00:15:39,684
For different reasons.

314
00:15:39,684 --> 00:15:41,934
This kind of feels like the same
moment for enterprise where.

315
00:15:42,254 --> 00:15:44,814
They're going to have to buy GPUs
probably for the first time if they're

316
00:15:44,814 --> 00:15:47,314
going to run it on prem, they're
going to want to keep it separate.

317
00:15:47,654 --> 00:15:50,114
They're going to either buy something
for the first, probably GPUs if they're

318
00:15:50,114 --> 00:15:53,484
not training models, they just want to
run things internally to access their

319
00:15:53,484 --> 00:15:58,004
internal data, or maybe they're doing
it in the cloud and Nirmal's company,

320
00:15:58,034 --> 00:16:04,194
AWS, is providing them the GPUs and
then presumably because they're getting

321
00:16:04,194 --> 00:16:07,614
dedicated hardware, they won't have
to worry about OpenAI or Anthropic

322
00:16:07,624 --> 00:16:08,894
having access to all their stuff

323
00:16:08,914 --> 00:16:12,624
so with respect to ModelRunner and
again, just a reminder, these are

324
00:16:12,624 --> 00:16:16,074
my own opinions and not that of
my employer, Amazon Web Services.

325
00:16:16,375 --> 00:16:18,065
There's still a lot of use cases.

326
00:16:18,075 --> 00:16:21,055
what Michael was talking about with
respect to choosing lots of different

327
00:16:21,065 --> 00:16:26,625
models for different types of tasks, I
think there's probably a hybrid model

328
00:16:26,625 --> 00:16:32,150
at some point where folks are using
different fine tuned niche models

329
00:16:32,530 --> 00:16:37,430
for specific tasks that they're doing
locally, and then as hardware improves,

330
00:16:37,430 --> 00:16:43,260
and hopefully, you know, maybe your 3
year old developer laptop that you get

331
00:16:43,290 --> 00:16:47,980
at your corporation has enough, or the
models get, you know, Get, optimize it

332
00:16:47,980 --> 00:16:52,760
enough that they can run on CPU or the
GPUs that you have on a corporate laptop,

333
00:16:52,770 --> 00:16:57,230
but there'll be some tooling, probably
embedded into the development tooling

334
00:16:57,550 --> 00:16:59,690
and, or you can choose your own models.

335
00:17:00,200 --> 00:17:04,190
and then those, there'll be other models
where you need to reach out to the

336
00:17:04,240 --> 00:17:08,840
hyperscalers for those that just you're
not going to get the depth of reasoning

337
00:17:08,850 --> 00:17:13,340
you're not going to get the depth of
knowledge that you will from something

338
00:17:13,340 --> 00:17:18,640
like Claude in the cloud versus something
like deep seek running a quantized deep

339
00:17:18,640 --> 00:17:24,700
seek running on your mac but again this
is all changing very very fast and Right

340
00:17:24,700 --> 00:17:30,740
now, we're in a different state where
folks have to spend money to access

341
00:17:30,790 --> 00:17:35,620
those larger models, which hasn't been
the pattern with respect to software

342
00:17:35,620 --> 00:17:37,990
development in a long time, right?

343
00:17:38,210 --> 00:17:39,900
Not everyone has that advantage, right?

344
00:17:40,070 --> 00:17:45,280
I mean, if you want to use Claude Code
and, not hit any major limits, then

345
00:17:45,280 --> 00:17:50,480
you got to pay 200 a month, which, not
everyone's going to be able to afford 2,

346
00:17:50,480 --> 00:17:54,990
400 a year, to do software development,
there's also edge use cases, right?

347
00:17:54,990 --> 00:17:57,930
So IoT devices, just
trying to figure it out.

348
00:17:57,930 --> 00:17:59,020
Also just kicking the tires.

349
00:17:59,020 --> 00:18:01,880
I think that's probably the main,
like Michael said, I think that's

350
00:18:01,890 --> 00:18:04,950
just, everyone is just trying to kick
the tires as cheaply as possible.

351
00:18:05,156 --> 00:18:08,386
model runner feels like a gateway, like
it feels like a gateway drug to get

352
00:18:08,396 --> 00:18:13,616
me Hooked on like the idea, cause I, I
mean, I wasn't paying attention to, I

353
00:18:13,616 --> 00:18:18,236
wasn't a machine, I wasn't an ML person
or I wasn't building AI infrastructure,

354
00:18:18,256 --> 00:18:21,846
but Docker model runner and, well, and,
you know, to a lesser extent, Ollama,

355
00:18:22,166 --> 00:18:25,056
Ollama always felt like it was more
for those people that were doing that,

356
00:18:25,056 --> 00:18:28,916
but bringing this capability into a
tool that I'm already using actually

357
00:18:28,936 --> 00:18:34,136
felt like, Okay, this is meant for me
now, like this is, I don't have to be,

358
00:18:34,226 --> 00:18:39,736
I don't have to understand weights and
all the intricacies of how models are

359
00:18:39,746 --> 00:18:43,776
built and work and the different, I
don't have to, I don't have to understand

360
00:18:43,786 --> 00:18:46,976
the different file format and whether
that works on my particular thing.

361
00:18:47,026 --> 00:18:47,976
it just all kind of works.

362
00:18:47,976 --> 00:18:49,656
It's Docker easy at that point for me.

363
00:18:51,355 --> 00:18:55,695
think that's the key point here of
just, let's try to increase the access

364
00:18:55,695 --> 00:18:59,935
to these capabilities and let folks
start to experiment and I think Again,

365
00:18:59,935 --> 00:19:03,615
we, as an industry, are trying to still
figure out exactly how to use a lot

366
00:19:03,615 --> 00:19:08,545
of these tools and, okay, what is the
right size model for the job at hand?

367
00:19:08,965 --> 00:19:12,225
and so, again, in order to be able
to do that experimentation, you

368
00:19:12,775 --> 00:19:14,185
have to increase the access to it.

369
00:19:14,565 --> 00:19:17,305
and so, but that's still kind of,
you know, Where we are in many ways.

370
00:19:17,632 --> 00:19:19,622
So did we check off
another thing on the list?

371
00:19:20,381 --> 00:19:23,751
about to I'm gonna have to keep
saying this We're not a news podcast.

372
00:19:23,771 --> 00:19:28,591
We don't talk about the latest things
like in general in tech but you

373
00:19:28,591 --> 00:19:31,232
know, I got to give a shout out to
Fireship because I have Between that

374
00:19:31,232 --> 00:19:34,082
and a few other channels, I've really
learned a lot this year about models.

375
00:19:34,392 --> 00:19:38,152
And I was, there's this little,
diagram of sort of the state of a lot

376
00:19:38,152 --> 00:19:41,602
of the open, waiter open, or free,
I'm just going to say free models.

377
00:19:41,852 --> 00:19:44,562
They're free in some capacity,
that you can download.

378
00:19:44,592 --> 00:19:45,912
And, I was using Devstral.

379
00:19:45,932 --> 00:19:48,422
We talked about it a couple of
times on this show already, which

380
00:19:48,422 --> 00:19:49,802
is a model that came out in May.

381
00:19:49,842 --> 00:19:51,122
I actually had a newsletter on that.

382
00:19:51,122 --> 00:19:52,302
Shout out to, Bret.

383
00:19:52,302 --> 00:19:54,702
news, there's this guy, Bret,
he makes a newsletter, Bret.

384
00:19:54,852 --> 00:19:56,002
news, you can go check that out.

385
00:19:56,282 --> 00:20:01,372
and I talked about this, that maybe this
was the sweet spot because it was small

386
00:20:01,372 --> 00:20:07,212
enough, you could run it with a modern
GPU or modern Mac, and it wasn't the

387
00:20:07,212 --> 00:20:11,632
worst, nothing like the frontier models
that we get with OpenAI and Anthropic,

388
00:20:11,632 --> 00:20:13,682
but it was something that was better.

389
00:20:15,342 --> 00:20:16,792
And, that's called Devstral.

390
00:20:17,302 --> 00:20:19,962
And then we had Quinn three
just come out, I don't know, a

391
00:20:19,962 --> 00:20:22,872
week ago or something that is

392
00:20:23,198 --> 00:20:24,478
I think it was earlier this week.

393
00:20:24,892 --> 00:20:27,222
Oh, see, time warp of AI,

394
00:20:27,378 --> 00:20:27,778
I think it

395
00:20:27,922 --> 00:20:29,262
three days is like three weeks.

396
00:20:29,746 --> 00:20:30,926
It was two days ago.

397
00:20:31,198 --> 00:20:31,648
Yeah,

398
00:20:31,912 --> 00:20:32,752
Oh gosh.

399
00:20:33,242 --> 00:20:35,972
I always assume that I'm seeing a
Fireship videos late, but, yeah,

400
00:20:35,972 --> 00:20:37,432
one day ago, so yeah, that's true.

401
00:20:37,432 --> 00:20:41,387
One day ago, there's a newer model
coming out from Alibaba, right?

402
00:20:41,467 --> 00:20:45,687
And it is even better, although
it does take more GPU memory, I

403
00:20:45,918 --> 00:20:48,438
that's hard to run like on a laptop.

404
00:20:48,598 --> 00:20:54,901
I don't think you can, so that's a perfect
encapsulation of like why Docker model

405
00:20:54,901 --> 00:20:59,951
run is there, but also why it's going
to be a mix of models going forward.

406
00:21:00,421 --> 00:21:05,751
so we've, we have an AI that helps
you use Docker and get started.

407
00:21:05,871 --> 00:21:11,121
We have a tool that helps you run
models locally, and understand that

408
00:21:11,481 --> 00:21:13,611
what's the next step in this journey?

409
00:21:14,088 --> 00:21:17,378
by the way, you go to Docker Hub
to look for models, or you can

410
00:21:17,378 --> 00:21:20,388
do it in Docker Desktop, or you
can look at things in the CLI.

411
00:21:20,398 --> 00:21:22,148
there's many ways to see the models.

412
00:21:22,508 --> 00:21:24,128
you can pull things from HuggingFace now.

413
00:21:24,351 --> 00:21:24,981
That's pretty sweet.

414
00:21:25,090 --> 00:21:26,660
Can we build, can we make our own yet?

415
00:21:26,660 --> 00:21:27,500
Or do we have that?

416
00:21:28,480 --> 00:21:28,730
We can

417
00:21:28,874 --> 00:21:32,464
so, you can package, but you would
have to already have the gguff

418
00:21:32,484 --> 00:21:33,914
file and all that kind of stuff.

419
00:21:33,914 --> 00:21:36,174
So we don't have a lot of
the tooling there to help you

420
00:21:36,174 --> 00:21:37,814
actually create the model itself.

421
00:21:38,004 --> 00:21:41,554
Although a lot of folks will use
container based environments to do that.

422
00:21:41,914 --> 00:21:44,264
we don't have any specific
tooling around that ourselves.

423
00:21:44,275 --> 00:21:46,385
So is there, you're saying there's
a doc, oh, there, there is.

424
00:21:46,385 --> 00:21:47,375
Oh, I didn't realize.

425
00:21:47,415 --> 00:21:52,685
So there is now a Docker model
package CLI for creating, basically

426
00:21:52,685 --> 00:21:59,615
wrapping the GGUF or, other models
or whatever, into the Docker OCI

427
00:21:59,625 --> 00:22:04,895
standard format for shipping and
pulling and pushing essentially Docker.

428
00:22:05,505 --> 00:22:08,535
models that are in Docker hubs
that are in the Docker format.

429
00:22:09,034 --> 00:22:10,004
OCI format.

430
00:22:10,044 --> 00:22:10,434
Yep.

431
00:22:11,484 --> 00:22:15,324
So when we look at agentic applications,
step one is, you need models.

432
00:22:15,324 --> 00:22:17,174
It's kind of the brains of the operations.

433
00:22:17,469 --> 00:22:19,029
and then you need tools.

434
00:22:19,029 --> 00:22:21,439
And so you've got highlighted
here the MCP toolkit.

435
00:22:21,439 --> 00:22:25,299
That was kind of the first
adventure into the MCP space.

436
00:22:25,299 --> 00:22:29,419
And that one was focused a little bit
more on how do we provide tools to the

437
00:22:29,429 --> 00:22:32,539
other agentic applications that you
are already running on your machine.

438
00:22:32,539 --> 00:22:39,929
So Claude Desktop or using VS Code
Copilot on my machine or Cursor, etc.

439
00:22:40,579 --> 00:22:44,709
How do we provide those MCP servers?

440
00:22:45,599 --> 00:22:49,079
And containerized ways, secure
credential injection, etc.

441
00:22:49,259 --> 00:22:52,179
Basically manage the life
cycle of those MCP servers.

442
00:22:52,559 --> 00:22:57,809
Again, in the use case of connecting
them to your other agentic applications.

443
00:22:58,449 --> 00:23:00,700
And so, again, this is kind of
where we started our MCP journey.

444
00:23:01,080 --> 00:23:04,520
if you see flipping through a couple
of these, actually we just released a

445
00:23:04,520 --> 00:23:09,030
Docker Hub MCP server that allows you
to Search for images on hub or, you

446
00:23:09,030 --> 00:23:13,070
know, those within your organization,
which is super helpful for like maybe

447
00:23:13,460 --> 00:23:15,520
a write me a Dockerfile that does X.

448
00:23:15,550 --> 00:23:16,130
Well, cool.

449
00:23:16,130 --> 00:23:18,740
Let's go find that the right image
that should be used for that.

450
00:23:19,140 --> 00:23:22,060
so again, starts to open up some of
these, additional capabilities here.

451
00:23:22,826 --> 00:23:23,286
Yeah.

452
00:23:23,286 --> 00:23:28,956
And inside the Docker desktop UI,
there is now a beta tab, essentially,

453
00:23:29,346 --> 00:23:30,846
that's called MCP Toolkit.

454
00:23:31,346 --> 00:23:39,796
And it is a GUI that allows me to explore
and enable one of, 141 different tools

455
00:23:39,836 --> 00:23:41,636
and growing that Docker has added.

456
00:23:41,966 --> 00:23:47,676
So like a lot of the other places on the
internet that either they host models

457
00:23:47,676 --> 00:23:53,346
like Anthropic or OpenAI, or They're a
place where you can create AI applications

458
00:23:53,626 --> 00:23:58,036
and all those places have started to
create their own little portals for

459
00:23:58,036 --> 00:24:01,516
finding tools and they may or may not,
I mean, most of them all now settle on

460
00:24:01,516 --> 00:24:05,286
MCP, but before we had really MCP as
the protocol standard, they were already

461
00:24:05,286 --> 00:24:09,386
doing like OpenAI was doing this before,
but they were very, it was proprietary.

462
00:24:09,386 --> 00:24:14,116
You don't know how Evernote or
Notion got, showed up as a tool

463
00:24:14,126 --> 00:24:16,556
feature in ChatGPT, it did.

464
00:24:17,371 --> 00:24:21,021
But we just assumed that was their
custom integration and now we have this

465
00:24:21,021 --> 00:24:24,531
standard called MCP that everything
should interact with everything

466
00:24:24,551 --> 00:24:26,941
properly the way that they should.

467
00:24:27,631 --> 00:24:29,491
At least right now it's
the one that's winning.

468
00:24:29,491 --> 00:24:31,531
We don't know whether we'll
still be talking about MCP in

469
00:24:31,531 --> 00:24:33,321
five years, but it's here now.

470
00:24:33,631 --> 00:24:34,751
It's what we're talking about now.

471
00:24:35,091 --> 00:24:39,034
And this lights up a lot of capabilities.

472
00:24:39,544 --> 00:24:42,334
In other words, you turn on an MCP tool.

473
00:24:43,334 --> 00:24:46,184
And that sits behind
something called MCP Gateway.

474
00:24:46,434 --> 00:24:48,304
So tell me, what is MCP Gateway?

475
00:24:48,749 --> 00:24:49,219
Yeah.

476
00:24:49,219 --> 00:24:53,719
So at the end of the day, the toolkit is
a combination of several different things.

477
00:24:53,749 --> 00:24:57,199
the MCP gateway is actually
a component that we just open

478
00:24:57,199 --> 00:24:58,539
source at WeAreDeveloper.

479
00:24:58,539 --> 00:25:00,054
So you can actually run
this gateway directly.

480
00:25:00,344 --> 00:25:01,844
In a container, completely on its own.

481
00:25:02,314 --> 00:25:05,724
And the MCP gateway is what's
actually responsible for managing

482
00:25:05,734 --> 00:25:07,554
the lifecycle of these MCP servers.

483
00:25:07,711 --> 00:25:10,471
it itself is an MCP server.

484
00:25:10,511 --> 00:25:12,531
think of it more like an MCP proxy.

485
00:25:12,731 --> 00:25:17,351
it exposes itself as an MCP server that
then can connect to your applications.

486
00:25:17,661 --> 00:25:20,596
But when you ask that server,
Hey, what tools do you have?

487
00:25:21,056 --> 00:25:25,146
It's really delegating, or I mean,
it's using cache versions of, okay,

488
00:25:25,146 --> 00:25:27,546
what are the downstream MCP servers?

489
00:25:28,316 --> 00:25:31,336
And so it's acting as
basically a proxy here.

490
00:25:31,946 --> 00:25:37,141
so when requests come in and say, hey,
you know, from the agents at GAP, if

491
00:25:37,141 --> 00:25:42,233
I want to execute this tool, go do a
search on DuckDuckGo at that point, the

492
00:25:42,233 --> 00:25:47,353
MCP gateway will actually spin up the
container, that DuckDuckGo MCP server,

493
00:25:47,643 --> 00:25:52,013
and then delegate the request to that
container, which then does the search, and

494
00:25:52,013 --> 00:25:54,043
then the MCP gateway returns the results.

495
00:25:54,043 --> 00:25:58,023
So kind of think of it as a proxy
that's managing the lifecycle of all

496
00:25:58,023 --> 00:26:01,803
those containers, but also, you know,
injecting the credentials, configuration.

497
00:26:02,143 --> 00:26:05,203
it also does other things like actually
looking at what's going in and out of

498
00:26:05,203 --> 00:26:09,233
the prompts going through the proxy to
make sure, you know, secrets aren't being

499
00:26:09,233 --> 00:26:11,243
leaked or, that kind of stuff as well too.

500
00:26:11,293 --> 00:26:14,743
and we're even starting to do some
further explorations of what are other

501
00:26:14,743 --> 00:26:16,883
ways to kind of secure those MCP servers?

502
00:26:16,933 --> 00:26:20,283
You know, for example, a file system
one should never have network access.

503
00:26:20,483 --> 00:26:24,473
Cool, so let's, when that container
starts, you get no network access,

504
00:26:24,493 --> 00:26:27,923
or, the GitHub MCP server, you know,
it's talking to the GitHub APIs.

505
00:26:29,178 --> 00:26:32,898
Let's only authorize those host
names that it can communicate with.

506
00:26:32,908 --> 00:26:35,988
So, you know, start to do a little
bit more of a kind of permissioning

507
00:26:35,998 --> 00:26:40,238
model around these MCP servers, which
is where a lot of people are kind of

508
00:26:40,238 --> 00:26:44,038
most cautious and nervous about MCP
servers, because it's, they're completely

509
00:26:44,038 --> 00:26:48,458
autonomous for the most part, and you
have to trust what's going on there.

510
00:26:48,478 --> 00:26:52,338
this exact feature is both
necessary and also Solomon Hyke's

511
00:26:52,358 --> 00:26:54,128
prediction from a month ago.

512
00:26:54,128 --> 00:26:57,408
He was on this show and was saying
that we're going to see all these

513
00:26:57,408 --> 00:27:00,398
infrastructure companies and all
these tooling companies that are

514
00:27:00,398 --> 00:27:01,908
going to offer to lock this shit down.

515
00:27:01,908 --> 00:27:03,788
I think it's the quote
I have to get from him.

516
00:27:04,217 --> 00:27:04,777
that sounds right.

517
00:27:04,928 --> 00:27:08,578
he compared the origin of containers
and how it started with developers.

518
00:27:08,578 --> 00:27:12,268
And then eventually it took it and sort
of managed it in the infrastructure

519
00:27:12,268 --> 00:27:16,708
layer and provided all the restrictions
and security and limitations and

520
00:27:16,738 --> 00:27:17,858
configuration and all this stuff.

521
00:27:18,268 --> 00:27:20,618
And the same thing's happening to MCP.

522
00:27:20,828 --> 00:27:25,348
Where it started out as a developer tool
to empower developers to do all these

523
00:27:25,348 --> 00:27:29,328
cool things with AI that they couldn't do
and let the AI actually do stuff for us.

524
00:27:29,588 --> 00:27:34,038
And now very quickly in a matter of
months, IT is coming in and saying,

525
00:27:34,038 --> 00:27:35,218
okay, we're going to lock this down.

526
00:27:35,218 --> 00:27:35,898
It's crazy.

527
00:27:35,908 --> 00:27:39,088
You can, you know, your prompts can
delete your, drop your databases,

528
00:27:39,088 --> 00:27:42,748
your, as we just saw happen on
the internet recently this week.

529
00:27:43,038 --> 00:27:48,598
I want to, on this gateway topic
though, it can sound complicated.

530
00:27:49,013 --> 00:27:52,403
And maybe the internals are a
little, and there was obviously

531
00:27:52,403 --> 00:27:54,003
code built into this program.

532
00:27:54,223 --> 00:27:58,353
But for those of us that aren't
maybe building agents yet, or

533
00:27:58,353 --> 00:28:03,593
like really getting into building
apps that use AIs in the app, this

534
00:28:03,593 --> 00:28:05,733
just appears as kind of magic.

535
00:28:05,773 --> 00:28:11,263
Like it, you go into the Docker desktop
UI, I enable, I go through the toolkit,

536
00:28:11,273 --> 00:28:15,788
there's all these suggestions, everything
from, the MCP server for GitHub itself to

537
00:28:15,998 --> 00:28:20,418
an MCP server that could give me access
to Grafana data to accessing the Heroku

538
00:28:20,418 --> 00:28:24,248
API and you're looking at all these things
and you're just like, I'm enabling them.

539
00:28:24,258 --> 00:28:25,518
it's like a kid in a candy store.

540
00:28:25,518 --> 00:28:26,348
I'm just going check, check, check.

541
00:28:26,348 --> 00:28:27,778
Yeah, I want Notion.

542
00:28:27,778 --> 00:28:28,538
I want Stripe.

543
00:28:28,588 --> 00:28:31,518
I get them in a list, they're
enabled, which means they're

544
00:28:31,518 --> 00:28:32,968
not actually running, right?

545
00:28:32,968 --> 00:28:37,238
they're waiting to be called before
the gateway runs them in memory.

546
00:28:37,518 --> 00:28:40,448
That's all transparent to me, I
don't realize that's happening.

547
00:28:40,838 --> 00:28:47,108
And if I choose to use this toolkit
with Gordon, if I just go into Gordon,

548
00:28:47,158 --> 00:28:52,668
And in the Gordon AI, if I don't want
to run a local model myself, or I'm

549
00:28:52,848 --> 00:28:57,338
not using Claude Desktop or something
that gives me the ability to enable MCP

550
00:28:57,338 --> 00:29:02,318
tools, I can just go in here and say,
enable all my MCP tools, all 34 of them.

551
00:29:02,418 --> 00:29:08,188
I've got get ones and I've got, and
so now what that means is the Gordon

552
00:29:08,188 --> 00:29:13,618
AI can now use these tools, which
makes this free AI even smarter.

553
00:29:14,008 --> 00:29:20,592
And I can say, is there a
Docker hub image NGINX?

554
00:29:20,712 --> 00:29:21,812
I don't know if there is.

555
00:29:22,407 --> 00:29:22,787
Let's see.

556
00:29:22,817 --> 00:29:23,817
I've never even tested this.

557
00:29:23,817 --> 00:29:25,947
So it's kind of a, what could go wrong?

558
00:29:25,977 --> 00:29:28,137
let's use Google live on the
internet and see what happens.

559
00:29:28,137 --> 00:29:32,037
yeah, I was just saying, look at it,
going out and checking, using the

560
00:29:32,037 --> 00:29:37,297
Docker, the newly created Docker hub,
MCB, tool That you had, just, released.

561
00:29:38,107 --> 00:29:46,027
so is this going through a, MCP gateway
or is this not with the MCP gateway yet?

562
00:29:46,245 --> 00:29:53,165
when you and Gordon AI flip the switch to
say, yes, I want to use the MCP toolkit,

563
00:29:53,165 --> 00:29:59,319
basically what that's doing is, in the
Gordon AI application here, it's enrolling

564
00:29:59,529 --> 00:30:01,729
the MCP toolkit as an MCP server.

565
00:30:02,118 --> 00:30:05,796
And so then it's going to ask
The MCP toolkit, hey, what

566
00:30:05,796 --> 00:30:06,586
tools do you have available?

567
00:30:06,586 --> 00:30:10,086
And so when you saw that list of tools,
that's again coming from the gateway.

568
00:30:10,536 --> 00:30:15,961
Gordon is just simply treating the
MCP toolkit as An MCP server, which in

569
00:30:15,961 --> 00:30:18,141
itself is going to launch MCP server.

570
00:30:18,141 --> 00:30:21,301
So that's kind of why I mentioned, it's
kind of thinking of it like a proxy there.

571
00:30:21,327 --> 00:30:22,707
Yeah, it does feel like one.

572
00:30:22,707 --> 00:30:23,197
Yeah.

573
00:30:23,307 --> 00:30:27,387
And this, for those watching, like it
didn't actually work because I don't

574
00:30:27,397 --> 00:30:30,377
actually have access to hardened images,
but, I just wanted to see what it would,

575
00:30:30,377 --> 00:30:32,117
what it'd say, but it, the, the UI

576
00:30:32,318 --> 00:30:33,648
Which is the right answer?

577
00:30:33,708 --> 00:30:34,788
Which is the right answer for

578
00:30:35,027 --> 00:30:36,317
did the right thing.

579
00:30:36,367 --> 00:30:39,607
it didn't expose a
vulnerability in the MCP server.

580
00:30:39,657 --> 00:30:43,047
But yeah, so it basically,
I can give Gordon AI more.

581
00:30:43,467 --> 00:30:46,407
And I can do a lot more functionality,
more abilities to do things without

582
00:30:46,417 --> 00:30:49,497
having to run my own model, without
having to figure out Claude Desktop.

583
00:30:49,767 --> 00:30:54,927
But I will say, because I'm in love with
this toolkit so much, because I love

584
00:30:54,927 --> 00:31:00,237
this idea of one place for my MCP tools,
for me to enter in the API secrets so it

585
00:31:00,237 --> 00:31:03,987
can access my notion and my Gmail but I
don't want to have to do that in Claude

586
00:31:04,007 --> 00:31:08,057
Desktop and then in warp and then in
VS code and then in Docker and then in

587
00:31:08,057 --> 00:31:10,637
Ollama and like every place I might run.

588
00:31:11,367 --> 00:31:14,127
A tool that needs MCP
tools or access to an LLM.

589
00:31:14,417 --> 00:31:16,807
So I did it in Docker Desktop.

590
00:31:16,827 --> 00:31:18,907
I enabled the ones and set
them up the way I wanted.

591
00:31:19,347 --> 00:31:25,537
And then inside of my tools around
my computer that all support AI MCPs,

592
00:31:26,537 --> 00:31:31,787
they all have now added MCP client
functionality that lets me talk to

593
00:31:31,847 --> 00:31:36,364
another MCP, any MCP server that
speaks proper MCP through their API.

594
00:31:36,694 --> 00:31:40,454
And in this case, what I've done in the
warp terminal, because it does support

595
00:31:40,464 --> 00:31:46,494
MCP, is I just tell it, the command it's
going to run is docker mcp gateway run.

596
00:31:46,904 --> 00:31:50,124
And it uses the standard in and
standard out, which is one of

597
00:31:50,124 --> 00:31:51,714
the ways that you can use mcp.

598
00:31:52,124 --> 00:31:57,084
And then I suddenly have all 34 tools
that we enabled in my docker desktop.

599
00:31:57,514 --> 00:32:01,254
Available in Warp, just as long as Docker
Desktop's running, that's all I gotta do.

600
00:32:01,874 --> 00:32:06,864
And then, because Warp is using Claude
Sonnet 4, or whatever I told Warp to

601
00:32:06,864 --> 00:32:10,204
do, Docker doesn't care about that,
because I'm not asking it to use,

602
00:32:10,614 --> 00:32:12,704
think that, we talked about this it's
called bring your own key, I guess

603
00:32:12,704 --> 00:32:13,664
that's what everybody's talking about.

604
00:32:14,024 --> 00:32:19,334
Uh, bring your own key is when you want
to bring your own Model to whatever

605
00:32:19,384 --> 00:32:19,854
to access,

606
00:32:20,134 --> 00:32:21,464
you're a key to access the model.

607
00:32:21,464 --> 00:32:25,234
Yeah, but in warp in particular,
like this is nuanced, but in

608
00:32:25,234 --> 00:32:28,434
warp, you can't usually, you can't
yet access your own models yet.

609
00:32:28,444 --> 00:32:30,394
I think they're going to make that
an enterprise feature or something.

610
00:32:30,744 --> 00:32:32,754
But if I open up VS code,
I could do the same thing.

611
00:32:32,754 --> 00:32:36,304
If I opened up a ChatGPT desktop,
I could do the same thing.

612
00:32:36,664 --> 00:32:41,494
And, Aider or like any of the CLI tools,
although anything that accepts MCP

613
00:32:41,524 --> 00:32:44,314
so far, I've gotten to work this way.

614
00:32:44,444 --> 00:32:49,914
And it's been awesome because all these,
all these different IDEs and AI tools

615
00:32:50,154 --> 00:32:51,324
all set up a little bit different.

616
00:32:51,324 --> 00:32:54,374
Goose you set up differently
than Claude Desktop.

617
00:32:54,374 --> 00:32:54,854
So

618
00:32:54,865 --> 00:32:55,565
Client.

619
00:32:56,225 --> 00:33:00,936
they're all, and all they, all of
them have different knobs and, ways of

620
00:33:00,936 --> 00:33:05,976
controlling MCP servers and at varying
degrees of control and flexibility.

621
00:33:06,416 --> 00:33:09,926
So this is really nice because
then you can also just have

622
00:33:09,936 --> 00:33:11,706
all those tools running if you

623
00:33:11,815 --> 00:33:12,265
right.

624
00:33:12,335 --> 00:33:15,295
They look like one giant MCP server.

625
00:33:15,295 --> 00:33:19,875
Yeah, because normally I would have to
add each MCP tool as its own server,

626
00:33:20,021 --> 00:33:22,811
I think, it feels like some of the
tools are all standardizing on Claude

627
00:33:22,831 --> 00:33:25,451
Desktop as, that config file, which I

628
00:33:25,563 --> 00:33:25,713
the mcp.

629
00:33:25,713 --> 00:33:27,003
json,

630
00:33:27,191 --> 00:33:31,771
Yeah, it feels like that, like everyone's
settling on just using that one file.

631
00:33:32,131 --> 00:33:35,731
Which is, I guess it's kind of feels
kind of hacky, but I guess it's fine.

632
00:33:36,211 --> 00:33:38,731
it feels like every editor
using my VIM settings.

633
00:33:38,731 --> 00:33:39,351
It's like, no, no, no, no.

634
00:33:39,381 --> 00:33:39,851
Calm down.

635
00:33:39,851 --> 00:33:42,041
I don't necessarily want you
all to use the same file.

636
00:33:42,261 --> 00:33:45,031
I don't want you all overwriting each
other and changing the same file.

637
00:33:45,081 --> 00:33:49,901
so you're bringing up a really important
thing, which is, since we're new to this,

638
00:33:50,001 --> 00:33:55,451
the number of MCP servers is not too many
yet, even though it does feel like there's

639
00:33:55,452 --> 00:33:56,571
probably like a lot of MCP servers.

640
00:33:56,572 --> 00:33:59,541
I've been using 10 new MCP servers
since we've started this conversation.

641
00:34:00,201 --> 00:34:04,451
But it's still like a number of tools
that we can still rationalize about.

642
00:34:04,881 --> 00:34:10,319
But probably in another month or two at
this rate, there is a limit to how many

643
00:34:10,329 --> 00:34:17,434
MCP tools One single client, I guess
you could say, or one instantiation of

644
00:34:17,434 --> 00:34:23,824
a task that you're using like Claude
Code or Cline, it's context window can

645
00:34:23,844 --> 00:34:25,924
only use a certain amount of tools.

646
00:34:26,354 --> 00:34:33,424
And so, is there some ideas about breaking
up in the MCP gateway, like having maybe

647
00:34:33,424 --> 00:34:37,934
like sets of tools that have like specific
supersets or tools or something like that?

648
00:34:38,014 --> 00:34:38,284
Yeah.

649
00:34:38,284 --> 00:34:38,914
Good question.

650
00:34:38,934 --> 00:34:40,314
And so that's a good call out.

651
00:34:40,314 --> 00:34:43,774
And so I actually want to, zoom in on
that just a tiny bit there, because

652
00:34:43,844 --> 00:34:46,584
for folks that may be new to this,
they may not quite understand that

653
00:34:47,154 --> 00:34:51,584
the way that tools work is basically,
it's taking all the tool descriptions.

654
00:34:51,904 --> 00:34:53,034
Okay, here's a tool name.

655
00:34:53,034 --> 00:34:54,704
Here's when I'm going to use this tool.

656
00:34:54,704 --> 00:34:57,214
Here's the parameters that are
needed to invoke this tool.

657
00:34:57,594 --> 00:35:00,584
And it's sending that to
the model on every request.

658
00:35:01,754 --> 00:35:05,364
And so the model's having to read
all that and basically say, hey,

659
00:35:05,364 --> 00:35:07,764
based on this conversation, hey,
here's a toolbox of stuff that

660
00:35:07,764 --> 00:35:09,564
I may or may not be able to use.

661
00:35:10,234 --> 00:35:15,889
But as Nirmal just pointed out, like
that takes context window there and

662
00:35:16,619 --> 00:35:21,099
granted yes some of the newer models
have incredibly huge context windows but

663
00:35:21,099 --> 00:35:25,589
depending on the use case, it's going
to affect your speed, it's going to

664
00:35:25,589 --> 00:35:29,559
affect the quality, and so yeah, you do
want to be careful of like, okay, I'm

665
00:35:29,559 --> 00:35:32,619
not just going to go in there and just
flip the box on all the MCP servers.

666
00:35:32,629 --> 00:35:33,739
Now you have access to everything.

667
00:35:33,739 --> 00:35:36,619
Like, you do want to be a little
conscious of that as well.

668
00:35:36,929 --> 00:35:40,599
in fact, I found it funny in a, I
was playing in Cursor not long ago,

669
00:35:40,599 --> 00:35:44,119
and you know, they have even just
YOLO mode, just go crazy with it.

670
00:35:44,529 --> 00:35:48,339
But even they have A warning once
you, I think it's after you enable

671
00:35:48,339 --> 00:35:54,009
the 31st tool of like, hey, heads
up, you're getting a little crazy.

672
00:35:54,499 --> 00:35:58,599
So like, I'm like, for the one that
has YOLO mode to call me out for being

673
00:35:58,599 --> 00:36:02,569
crazy for too many tools, like it was,
it's again, kind of a reminder of just,

674
00:36:03,029 --> 00:36:05,899
okay, you do want to be conscious of
the number of tools that you're using.

675
00:36:06,369 --> 00:36:07,619
so to actually answer the question.

676
00:36:07,619 --> 00:36:07,739
Yeah.

677
00:36:07,739 --> 00:36:12,099
It's been something that we've been
exploring and kind of waiting to just see

678
00:36:12,099 --> 00:36:14,169
what the feedback is gonna be on that.

679
00:36:14,229 --> 00:36:17,349
Are there separate tool sets
that, clients can connect to.

680
00:36:18,139 --> 00:36:21,929
you know, that's certainly a possibility
as well, since this MCP gateway is an

681
00:36:21,929 --> 00:36:26,779
open source container, when you run
this for your application, not only can

682
00:36:26,779 --> 00:36:30,229
you say, these are the servers I want,
but then you can even further filter

683
00:36:30,239 --> 00:36:33,559
through, these are the tools from those
servers that I actually want to expose.

684
00:36:33,559 --> 00:36:36,069
So, for example, I think the
GitHub official one is up to

685
00:36:36,069 --> 00:36:38,109
72 tools now or something.

686
00:36:38,349 --> 00:36:39,829
It's a crazy number.

687
00:36:40,179 --> 00:36:41,949
but most of the time, I only
need maybe three or four of them.

688
00:36:42,269 --> 00:36:43,559
So, I want to filter that.

689
00:36:43,559 --> 00:36:46,529
And that's why you see, Cloud and
VS Code and many of these others.

690
00:36:46,539 --> 00:36:50,309
Even though you're pulling in
these MCP servers, many of those

691
00:36:50,329 --> 00:36:53,249
provide client side functionality
to kind of filter that list as well,

692
00:36:53,249 --> 00:36:53,629
Yeah.

693
00:36:54,290 --> 00:36:57,730
I wonder if we get to a state because
the MC, this is getting a little bit

694
00:36:57,730 --> 00:37:01,760
meta, but everything when you talk about
agentic AI gets meta really quickly.

695
00:37:02,450 --> 00:37:07,910
So, I wonder if the MCP gateway itself
is an MCP server, so it can rationalize

696
00:37:07,920 --> 00:37:12,700
about itself, I wonder if we get into
the pattern of, okay, there's this new

697
00:37:12,700 --> 00:37:17,850
task that I want this agent to do, and
the first thing, after it comes up with

698
00:37:17,850 --> 00:37:22,210
its task list, the steps it wants to
take, is go through that list and then,

699
00:37:22,690 --> 00:37:28,505
ask the MCP gateway to reconfigure
itself on each task and turn on Only

700
00:37:28,505 --> 00:37:32,505
the ones that it identified as likely
the ones that it needs for that task.

701
00:37:32,815 --> 00:37:36,265
And just dynamically, at any
given time, don't have anything

702
00:37:36,265 --> 00:37:37,485
more than five running.

703
00:37:37,865 --> 00:37:38,905
So figure it out.

704
00:37:39,005 --> 00:37:41,555
You can choose whatever five
you want, but only have five.

705
00:37:41,618 --> 00:37:45,548
We've done some experiments with that,
not quite to that full dynamicness,

706
00:37:45,578 --> 00:37:49,548
but I've even done some ones of a
Okay, here's a tool to enable other

707
00:37:49,548 --> 00:37:51,398
tools, is basically what it is.

708
00:37:51,848 --> 00:37:55,368
And, okay, and give me parameters
of, okay, do you need, GitHub?

709
00:37:55,378 --> 00:37:56,648
Do you need, Slack?

710
00:37:56,648 --> 00:37:59,768
You know, tell me what it is
that you need, and then I'll

711
00:37:59,768 --> 00:38:01,288
enable those specific things.

712
00:38:01,288 --> 00:38:05,378
And then what's cool then is,
as part of the MCP protocol,

713
00:38:05,378 --> 00:38:06,488
there's also notifications.

714
00:38:06,488 --> 00:38:10,393
So the MCP server can then notify,
The client says, hey, there's a new

715
00:38:10,393 --> 00:38:13,573
list of tools available, and then
the next API request to the model

716
00:38:13,863 --> 00:38:15,213
then has this new set of tools.

717
00:38:15,985 --> 00:38:16,055
I

718
00:38:16,055 --> 00:38:16,635
think we're almost

719
00:38:16,763 --> 00:38:18,353
capability is there, but,

720
00:38:19,055 --> 00:38:20,385
I think that's likely the next step.

721
00:38:21,423 --> 00:38:26,793
but it's also kind of like a,
yeah, how do you safeguard that?

722
00:38:26,853 --> 00:38:31,398
So it's, Yeah, it's an
interesting time period, for sure.

723
00:38:31,975 --> 00:38:35,445
we got an interesting question, is
MCP Gateways intent to replace an

724
00:38:35,465 --> 00:38:37,545
API Gateway or in parallel to it?

725
00:38:37,935 --> 00:38:38,645
Great question.

726
00:38:38,645 --> 00:38:39,715
Michael, you want to take that one?

727
00:38:39,858 --> 00:38:40,638
yeah, great question.

728
00:38:40,688 --> 00:38:44,672
I'd say in some ways that there's
similar functionality, but they

729
00:38:44,732 --> 00:38:45,932
serve very different purposes.

730
00:38:45,932 --> 00:38:49,822
So an API gateway, I'll just take
the most basic example, but I know

731
00:38:49,822 --> 00:38:53,932
there's lots of different ones, An API
gateway, single endpoint, and I may

732
00:38:53,932 --> 00:38:55,652
have lots of different microservices.

733
00:38:55,672 --> 00:38:57,092
Let's just pick a catalog.

734
00:38:57,132 --> 00:39:00,452
Okay, so for product related ones,
it's going to go to this microservice.

735
00:39:00,642 --> 00:39:02,182
Users, it's going to go to this other one.

736
00:39:02,182 --> 00:39:03,862
Cart, another service, whatever.

737
00:39:04,202 --> 00:39:08,112
And the API gateway is routing all
those different requests and rate

738
00:39:08,112 --> 00:39:14,751
limiting, etc. In many ways, like this
MCP gateway serves in a similar fashion

739
00:39:15,371 --> 00:39:18,921
in which it's going to be routing
to the right MCP server to actually

740
00:39:18,941 --> 00:39:20,521
handle the tool execution and whatnot.

741
00:39:20,931 --> 00:39:24,231
But again, it's only for the MCP protocol.

742
00:39:24,531 --> 00:39:27,811
So it's not going to be replacing an
API gateway because it's not doing

743
00:39:27,811 --> 00:39:33,451
normal API requests, etc. It's only
for MCP related workloads and requests.

744
00:39:34,381 --> 00:39:35,931
different protocols at play here.

745
00:39:36,737 --> 00:39:38,607
I think that's probably the
best way to describe it.

746
00:39:38,657 --> 00:39:44,357
otherwise, you could also say that
MCP and API Gateway are likely

747
00:39:44,357 --> 00:39:46,407
going to be running in parallel.

748
00:39:46,767 --> 00:39:50,757
and so probably what I would see would
be, I have an API gateway that routes

749
00:39:50,757 --> 00:39:55,807
a request to an endpoint, and then that
particular application, let's just say

750
00:39:55,807 --> 00:40:01,977
it's an agentic application, can then have
its own MCP gateway to satisfy whatever

751
00:40:01,977 --> 00:40:03,997
agentic flow it needs to use there.

752
00:40:03,997 --> 00:40:07,467
I wanted to, while you guys were having
an awesome conversation, I was trying

753
00:40:07,467 --> 00:40:13,947
to draw up, just a visualization to
try to represent, okay, so just so

754
00:40:13,947 --> 00:40:16,907
people understand, because this MCP,
we could make a whole show on MCP

755
00:40:16,907 --> 00:40:20,107
tools, honestly, from an infrastructure
perspective, how do these things talk?

756
00:40:20,117 --> 00:40:20,927
How do they integrate?

757
00:40:21,347 --> 00:40:23,367
The fact that you're talking about
that they're just really adding to

758
00:40:23,367 --> 00:40:26,898
the context window is a fantastic
fact that, A lot of people could go

759
00:40:27,088 --> 00:40:31,498
months or years using MCP tools day
to day and never know that, right?

760
00:40:31,558 --> 00:40:35,458
a normal non engineer could use
MCP tools, not understand how

761
00:40:35,458 --> 00:40:36,478
these things are all working.

762
00:40:36,738 --> 00:40:40,288
for those that are into this, are
playing around with MCP tools elsewhere

763
00:40:40,698 --> 00:40:44,538
and understanding a little bit of MCP
server functionality and client versus

764
00:40:44,538 --> 00:40:46,138
server versus host and all that stuff.

765
00:40:46,458 --> 00:40:51,908
Before Docker's gateway, the MCP gateway,
you would have like an MCP client

766
00:40:51,978 --> 00:40:55,748
That whether it's your IDE, your terminal,
or AI, chat, desktop, or whatever you've

767
00:40:55,748 --> 00:40:58,068
got, that is acting as an MCP client.

768
00:40:58,358 --> 00:41:01,518
Assuming it supports MCP servers,
you can add them one at a time.

769
00:41:01,748 --> 00:41:05,978
So I would add GitHub's MCP server, then
I would add DuckDuckGo's MCP server.

770
00:41:06,228 --> 00:41:09,058
I might add Notion's MCP server,
since I'm a big Notion fan.

771
00:41:09,338 --> 00:41:14,368
And each one of those servers
has One to infinity, tools, which

772
00:41:14,368 --> 00:41:16,538
are, I look at as like API routes.

773
00:41:16,918 --> 00:41:20,308
and each one has its
own very niche purpose.

774
00:41:20,774 --> 00:41:23,414
depending on the tool, and this is part
of the frustration with the ecosystem

775
00:41:23,414 --> 00:41:26,414
right now is we're only months into this,
but it's amazing that all these tools are

776
00:41:26,414 --> 00:41:30,024
all starting to support each other, tools
have different ways where you manage this.

777
00:41:30,034 --> 00:41:33,244
Some of them you can disable
and enable specific servers.

778
00:41:33,454 --> 00:41:36,914
Some, you can actually choose
the tools individually, which

779
00:41:36,914 --> 00:41:38,494
is like choosing API routes.

780
00:41:38,944 --> 00:41:41,994
And to me, it's you're always trying
to get down to the smallest amount of

781
00:41:41,994 --> 00:41:44,134
tools that you need to prevent confusion.

782
00:41:44,144 --> 00:41:47,854
Cause I'm, my biggest problem is
I enable all the tools because

783
00:41:47,854 --> 00:41:49,514
I get tired of managing them.

784
00:41:49,674 --> 00:41:50,398
I just want them to work.

785
00:41:50,759 --> 00:41:52,829
I just want them all to
work when they need to work.

786
00:41:53,189 --> 00:41:54,589
And then I, so I enable them all.

787
00:41:55,379 --> 00:41:56,939
I end up with 50 plus tools.

788
00:41:57,239 --> 00:42:01,419
And then when I'm asking AI to do things,
it chooses the wrong tool because I

789
00:42:01,419 --> 00:42:06,699
wasn't precise enough in my ask to
trigger the right words that are written

790
00:42:06,699 --> 00:42:09,419
in the system prompt of that MCP server.

791
00:42:09,469 --> 00:42:14,769
So actually, maybe an easier update might
be to put another layer on top of the

792
00:42:14,769 --> 00:42:18,029
MCP server and kind of an in between.

793
00:42:18,109 --> 00:42:23,929
so I'm connecting the MCP gateway
now to multiple other MCP servers.

794
00:42:24,189 --> 00:42:25,049
So I get, yeah, you're right.

795
00:42:25,049 --> 00:42:26,009
I need another layer here.

796
00:42:26,009 --> 00:42:27,379
That's actually MCP servers.

797
00:42:27,739 --> 00:42:33,829
so, there's now this gateway in the
middle, and it, the only negative of this

798
00:42:33,829 --> 00:42:39,119
approach is for right now, because we
don't have this futuristic utopia yet,

799
00:42:39,539 --> 00:42:47,579
is that to my terminal, or my IDE, it
all looks like one giant list of tools.

800
00:42:47,859 --> 00:42:51,589
And in one MCP server, which is
just the nature of proxy, right?

801
00:42:51,639 --> 00:42:54,379
But behind one IP address is a
whole bunch of websites, like

802
00:42:54,379 --> 00:42:55,349
you don't realize it, right?

803
00:42:55,719 --> 00:42:58,129
So it is, the analogy still
works, I believe, there.

804
00:42:58,349 --> 00:43:04,289
But in this case, because it's connecting
all of them together into one proxy,

805
00:43:04,309 --> 00:43:07,929
and the nice thing is, it's, I can see
in the memory usage and the containers.

806
00:43:07,929 --> 00:43:12,229
In fact, when Michael was on weeks
ago, We saw the MCP gateway spinning up

807
00:43:12,229 --> 00:43:15,619
servers dynamically and then shutting
them down and you could see the container

808
00:43:15,619 --> 00:43:19,279
launch run, you know, run the curl
command or whatever, and then close.

809
00:43:19,399 --> 00:43:22,919
And it was so quick, we couldn't,
capture and swat toggle windows

810
00:43:23,129 --> 00:43:24,269
to see the tools launching.

811
00:43:24,269 --> 00:43:25,379
And, I mean, it's beautiful.

812
00:43:25,379 --> 00:43:26,819
It's exactly what containers were for.

813
00:43:26,894 --> 00:43:28,654
It's, it's ephemeral, it's wonderful.

814
00:43:29,379 --> 00:43:34,639
But, if your IDE or if your chat desktop
or whatever is acting as your MCP client,

815
00:43:34,639 --> 00:43:39,729
the agent thing, if that doesn't let
you choose individual tools, then this

816
00:43:39,729 --> 00:43:44,989
approach is a little hard because the
only way from my IDE that I can, I have to

817
00:43:44,989 --> 00:43:47,169
turn off all of Docker or none of Docker.

818
00:43:47,169 --> 00:43:47,762
It I think.

819
00:43:48,312 --> 00:43:51,342
This gets us to a conversation
of eventually we will have this.

820
00:43:51,532 --> 00:43:55,162
I'm thinking of it as like the model,
the plan model before the model that will

821
00:43:55,162 --> 00:43:57,112
go, okay, you used all these keywords.

822
00:43:57,132 --> 00:43:59,512
I'm going to pick out the right
tools and I'm going to hand those

823
00:43:59,512 --> 00:44:01,782
off to the next model, which
is going to do the actual work.

824
00:44:02,352 --> 00:44:03,912
That's probably already here.

825
00:44:04,022 --> 00:44:05,632
Solomon predicted it a month ago.

826
00:44:05,822 --> 00:44:06,422
Yeah, I'm sorry.

827
00:44:06,422 --> 00:44:06,682
What?

828
00:44:07,023 --> 00:44:09,213
so that's what Michael and I,
while you were drawing this

829
00:44:09,402 --> 00:44:10,562
Oh, is that what you
were just talking about?

830
00:44:10,763 --> 00:44:12,533
that's what Michael and
I, we were talking about.

831
00:44:12,734 --> 00:44:18,600
So the gateway itself has its own
MCP server that controls itself.

832
00:44:19,180 --> 00:44:24,030
And so we're a few months away from
exactly what you were just talking about.

833
00:44:24,290 --> 00:44:28,200
Bret, because of context windows,
because there's too many tools, because

834
00:44:28,200 --> 00:44:31,690
of all the things that you did, all the
challenges you just mentioned, Bret.

835
00:44:32,120 --> 00:44:37,300
the first step might be the client
going to the MCP gateway, MCP server

836
00:44:37,300 --> 00:44:41,500
first and saying, hey, these are
the things I'm about to go do.

837
00:44:41,500 --> 00:44:46,560
Out of the list, check your MCP
gateway and tell me the list of, MCP

838
00:44:46,560 --> 00:44:49,010
tools that I actually need for that.

839
00:44:49,480 --> 00:44:52,690
And then only turn those
on for the next task.

840
00:44:53,228 --> 00:44:53,698
Yeah.

841
00:44:54,250 --> 00:44:57,130
and then it'll just
repeat that cycle again.

842
00:44:57,240 --> 00:45:03,640
and then winnow down that list of
MCP tools to the only things that

843
00:45:03,640 --> 00:45:05,290
are needed for that task at hand.

844
00:45:05,740 --> 00:45:09,220
So there's another layer here,
which we're, and Michael and I,

845
00:45:09,220 --> 00:45:11,520
we were discussing while you were
building that beautiful diagram.

846
00:45:12,110 --> 00:45:14,530
It's, people are experimenting with that.

847
00:45:14,580 --> 00:45:19,460
All the pieces are in place, but that this
pattern isn't quite there just yet, but

848
00:45:19,460 --> 00:45:23,200
it will likely be, I'm pretty sure this
is what we're going to be doing pretty

849
00:45:23,253 --> 00:45:26,823
Nobody wants to go manually choose
every MCP server that they're going

850
00:45:26,823 --> 00:45:28,403
to need before every AI request.

851
00:45:28,953 --> 00:45:31,673
almost feels like it takes away
the speed advantage of using the

852
00:45:31,673 --> 00:45:33,373
MCP tool to go get the data for me.

853
00:45:33,373 --> 00:45:38,388
if I have to do all this work in each
tool independently, Because I often

854
00:45:38,388 --> 00:45:43,648
will have an IDE accessing AI, acting
as the MCP client, and then I'll have

855
00:45:43,648 --> 00:45:45,318
a terminal acting as an MCP client.

856
00:45:45,548 --> 00:45:49,728
At the same time, I've got ChatGPT
desktop running over here, also

857
00:45:49,728 --> 00:45:52,928
while VS Code, I think a lot of us,
eventually evolve to the point where

858
00:45:52,988 --> 00:45:57,418
we've got two or three tools all at
the same time managing MCP tools.

859
00:45:57,448 --> 00:45:59,828
We've got, I guess we have
multiple IDEs, I should say.

860
00:46:00,238 --> 00:46:04,848
and trying to understand how all this
comes together is only interesting right

861
00:46:04,848 --> 00:46:08,538
now, but in six months, we're not going
to want to be messing with all this stuff.

862
00:46:08,538 --> 00:46:11,598
We're just going to want this part to
work so we can work on building agents,

863
00:46:11,648 --> 00:46:11,958
All right.

864
00:46:11,958 --> 00:46:16,348
So Compose, my favorite tool, a lot of
people's favorite Docker tool, other

865
00:46:16,348 --> 00:46:17,588
than the fact that Docker exists.

866
00:46:17,838 --> 00:46:21,828
you announced at, we are developers
that Compose is getting more.

867
00:46:22,138 --> 00:46:24,878
There's functionality in the YAML
specifically, where I guess we're talking

868
00:46:24,878 --> 00:46:30,248
about the YAML configuration that drives
the Compose command line, that in just

869
00:46:30,248 --> 00:46:34,458
three months ago, you were adding model
support, and that was like an early

870
00:46:34,458 --> 00:46:40,078
alpha idea of what if I could specify
the model I wanted Docker Model Runner

871
00:46:40,078 --> 00:46:47,588
to run when I launch my app that maybe
needs a model, a local model, and I

872
00:46:47,588 --> 00:46:51,893
use the example and I have a An actual
demo over on GIST, that people can

873
00:46:51,893 --> 00:46:57,693
pick up that you simply, you, you write
your Compose file, you use something

874
00:46:57,693 --> 00:47:00,548
called open web, open, what is it?

875
00:47:00,598 --> 00:47:03,398
Open web, web, open web UI, I think.

876
00:47:03,618 --> 00:47:04,908
Yeah, horrible name.

877
00:47:05,448 --> 00:47:10,308
Extremely generic name for what
is a ChatGPT clone, essentially.

878
00:47:10,368 --> 00:47:14,428
the open source variant, which can
use any models or more than one model.

879
00:47:14,438 --> 00:47:16,448
It actually lets you
choose it in the interface.

880
00:47:16,908 --> 00:47:21,608
And all you need is a
little bit of compose file.

881
00:47:23,528 --> 00:47:27,368
So, I created a 29 lines, and it
probably needs to be updated because

882
00:47:27,368 --> 00:47:34,978
it's probably outdated, but, 29 lines of
Compose that's half comments that allows

883
00:47:34,978 --> 00:47:40,958
me to spin up an open web UI container
while also spinning up the models or

884
00:47:40,968 --> 00:47:44,768
making sure, basically, that I have the
models locally that I need to run it.

885
00:47:44,998 --> 00:47:49,138
And this gives me a ChatGPT
experience without ChatGPT.

886
00:47:49,138 --> 00:47:50,238
Thank you.

887
00:47:50,318 --> 00:47:52,098
And you guys, you enable this.

888
00:47:52,098 --> 00:47:53,928
Now you're not creating the models.

889
00:47:54,088 --> 00:47:55,938
You're not creating the open web UI.

890
00:47:56,138 --> 00:48:00,358
you're simply providing the
glue for it to all come together

891
00:48:00,358 --> 00:48:02,208
in a very easy way locally.

892
00:48:02,548 --> 00:48:02,788
Yeah.

893
00:48:02,788 --> 00:48:06,258
as we agentic apps need three things.

894
00:48:06,258 --> 00:48:09,578
They need models, they need tools, and
then the code that glues it all together.

895
00:48:09,978 --> 00:48:13,398
What the Compose file lets us
do now is define all three of

896
00:48:13,398 --> 00:48:15,788
those in a single document.

897
00:48:15,978 --> 00:48:21,658
here's the models that my app is going to
need for MCP gateway that I'm just going

898
00:48:21,658 --> 00:48:23,378
to run as another containerized service.

899
00:48:23,708 --> 00:48:27,408
And then the code, the custom code,
can be really any agentic framework.

900
00:48:27,418 --> 00:48:32,308
this example is, Open web UI, but that
Compose snippet what we've done is

901
00:48:32,308 --> 00:48:37,728
we've evolved the specification now
models are a top level element in the

902
00:48:37,728 --> 00:48:39,153
Compose file, which is pretty cool.

903
00:48:39,693 --> 00:48:42,453
This just dropped in the last couple
of weeks, so this is brand new.

904
00:48:42,819 --> 00:48:43,749
Gotta update my gist.

905
00:48:44,213 --> 00:48:47,713
yep, and so where before, yeah,
you had to use this provider

906
00:48:47,713 --> 00:48:49,613
syntax, and that still works.

907
00:48:49,933 --> 00:48:52,123
now it's actually part
of the specification.

908
00:48:52,543 --> 00:48:55,203
defining a model, This is
going to pull from Docker Hub.

909
00:48:55,213 --> 00:48:58,318
again, You can have your own models
and your own container registry.

910
00:48:58,318 --> 00:48:59,608
It's just an OCI artifact.

911
00:48:59,608 --> 00:49:01,058
You can specify that anywhere.

912
00:49:01,698 --> 00:49:03,108
then we've got the services,

913
00:49:03,378 --> 00:49:04,908
And then the app itself.

914
00:49:05,228 --> 00:49:08,958
What's cool about the model now is
with the specification evolution,

915
00:49:09,278 --> 00:49:13,658
you can now specify, hey, this is the
environment variable I want you to

916
00:49:13,658 --> 00:49:15,658
basically inject into my container.

917
00:49:16,038 --> 00:49:19,868
to specify what's the endpoint,
where's the base URL that I

918
00:49:19,878 --> 00:49:22,108
should use to access this model.

919
00:49:22,488 --> 00:49:24,098
And then what's the model name as well.

920
00:49:24,398 --> 00:49:29,718
So the cool thing then is I can
go back up to the top level model

921
00:49:29,718 --> 00:49:33,908
specification, I can swap that out
and the environment variables will be

922
00:49:33,918 --> 00:49:37,858
automatically updated and as assuming
that my app is using those environment

923
00:49:37,858 --> 00:49:39,848
variables, Everything just works.

924
00:49:40,308 --> 00:49:44,188
So again, think of Compose as, it's
the glue that's making sure that

925
00:49:44,188 --> 00:49:48,218
everything is there for the application
to actually be able to leverage it.

926
00:49:49,029 --> 00:49:49,479
Yeah.

927
00:49:49,539 --> 00:49:51,729
the gateway part here
was pretty cool to me.

928
00:49:51,729 --> 00:49:58,429
That I can add in my tools, my
MCP tools inside of the YAML file.

929
00:49:58,449 --> 00:50:00,039
when I saw that part, I was like, yes.

930
00:50:00,089 --> 00:50:05,329
that is like my vision, my dream is
that I can pass a composed file to

931
00:50:05,329 --> 00:50:09,159
someone else and it'll use their keys.

932
00:50:09,689 --> 00:50:10,469
Presuming, my.

933
00:50:10,934 --> 00:50:15,354
Team is all using the same provider
that we would have the same.

934
00:50:15,364 --> 00:50:19,544
Because open, well, open AI,
base URL, open AI model, and

935
00:50:19,544 --> 00:50:21,704
then open AI API key or whatever.

936
00:50:21,734 --> 00:50:24,364
if you're going to use ones in the
SAS, like those are all pretty generic.

937
00:50:24,514 --> 00:50:28,154
Even if you're not using OpenAI, they're
all pretty generic, environment variables.

938
00:50:28,164 --> 00:50:30,794
So I guess this would work
across teams or across people

939
00:50:30,838 --> 00:50:32,388
well, and that's a good point to call it.

940
00:50:32,398 --> 00:50:36,528
One of the things that OpenAI did when
they released their APIs was basically,

941
00:50:36,528 --> 00:50:40,008
hey, here's a specification on how to
interact with models that pretty much,

942
00:50:40,058 --> 00:50:42,168
Everybody else has adopted and used.

943
00:50:42,548 --> 00:50:47,068
and so the Docker model runner
exposes an OpenAI compatible API.

944
00:50:47,068 --> 00:50:49,428
And so that's why you see these
environment variables kind

945
00:50:49,428 --> 00:50:52,288
of using the OpenAI prefix.

946
00:50:52,718 --> 00:50:56,168
Because again, I can use now any
agentic application that can talk to

947
00:50:56,168 --> 00:50:58,278
OpenAI or use the OpenAI libraries.

948
00:50:58,278 --> 00:51:00,328
And it's just a configuration
change at this point.

949
00:51:00,433 --> 00:51:00,813
All right.

950
00:51:00,833 --> 00:51:01,923
Now, the coup de gras.

951
00:51:02,363 --> 00:51:03,753
Piece la resistance.

952
00:51:04,463 --> 00:51:08,393
I can't even do my pretend French, All
this stuff has been running locally.

953
00:51:08,393 --> 00:51:10,893
Like when we think of Docker desktop,
we think of everything locally.

954
00:51:10,893 --> 00:51:17,153
And then a year or two ago, Docker
launched, Docker build cloud, which was

955
00:51:17,163 --> 00:51:19,583
like getting back to Docker's roots.

956
00:51:19,583 --> 00:51:23,903
I almost feel like of providing more
a SaaS service that essentially is.

957
00:51:24,678 --> 00:51:26,528
Doing something in a container for me.

958
00:51:26,528 --> 00:51:30,228
And in that case, it was just building
containers using an outsourced build kit.

959
00:51:30,238 --> 00:51:34,538
So it was better for parallelization
and multi architecture.

960
00:51:34,588 --> 00:51:35,168
it was sweet.

961
00:51:35,178 --> 00:51:39,878
And I love it for when I need to build
like enterprise tools or big business

962
00:51:39,968 --> 00:51:41,838
things that take 20 minutes to build.

963
00:51:41,838 --> 00:51:45,598
None of my sample little examples do
that, but anything in the real world

964
00:51:45,598 --> 00:51:48,718
takes that long and you need to build
multi architecture and generally it's

965
00:51:48,758 --> 00:51:50,368
going to be faster in a cloud environment.

966
00:51:50,368 --> 00:51:51,278
So you provided that.

967
00:51:51,658 --> 00:51:58,428
Now it feels like you've upgraded, like
it's beyond just building, and it does any

968
00:51:58,428 --> 00:52:04,088
image or any container I want to run, any
model I want to run, I guess, not maybe

969
00:52:04,098 --> 00:52:07,748
any, I don't know if there's a limitation
there, but, bigger models, then maybe I

970
00:52:07,748 --> 00:52:11,238
can run locally, and then, also builds.

971
00:52:11,653 --> 00:52:15,443
So it can do building the image,
hosting the container, running the

972
00:52:15,443 --> 00:52:17,903
model endpoint for a, QIN3 or whatever.

973
00:52:18,423 --> 00:52:20,953
I can now do all that in
something called Offload.

974
00:52:21,303 --> 00:52:22,243
So tell me about that.

975
00:52:22,548 --> 00:52:26,588
Docker Offload, the way I explain it to
people is, hey, you need more resources?

976
00:52:26,768 --> 00:52:27,808
Burst into the cloud.

977
00:52:28,628 --> 00:52:31,838
And so it's basically, I'm going
to offload this into the cloud,

978
00:52:32,178 --> 00:52:35,548
but yet it's still, everything
still works as if it were local.

979
00:52:35,548 --> 00:52:38,058
So if I've got bind mounts, okay,
great, we're going to automatically

980
00:52:38,058 --> 00:52:39,518
set up the synchronized file shares.

981
00:52:39,528 --> 00:52:43,448
And so all that's going to work the way
using mutagen and some of the other tools

982
00:52:43,448 --> 00:52:44,948
behind the scenes to make that work.

983
00:52:45,278 --> 00:52:48,008
Port publishing that still
works as you would expect it to.

984
00:52:48,008 --> 00:52:54,798
So again, it gives that local
experience, but using remote resources.

985
00:52:54,848 --> 00:52:57,758
I'm just offloading this to
the cloud, but yet it's still.

986
00:52:58,393 --> 00:52:59,153
My environment.

987
00:52:59,553 --> 00:53:02,553
and so, yeah, to make it clear, like,
this is not a production runtime

988
00:53:02,553 --> 00:53:05,863
environment, I can't share this
environment out, or, I can't, you

989
00:53:05,863 --> 00:53:08,573
know, create a URL and say, hey, check
this out, colleague, or whatever,

990
00:53:08,573 --> 00:53:10,253
it's still for your personal use.

991
00:53:10,423 --> 00:53:13,853
Now, of course, can you make a,
Cloudflare tunnels, and I'm going

992
00:53:13,853 --> 00:53:17,013
to make it production, sure, but I

993
00:53:17,053 --> 00:53:17,243
wouldn't

994
00:53:17,304 --> 00:53:17,614
could hack

995
00:53:17,973 --> 00:53:18,223
that.

996
00:53:18,444 --> 00:53:18,884
but yeah.

997
00:53:18,945 --> 00:53:19,555
Yeah.

998
00:53:19,795 --> 00:53:20,755
So what is the intent?

999
00:53:20,765 --> 00:53:23,095
so what is the use case?

1000
00:53:23,935 --> 00:53:24,205
the big

1001
00:53:24,267 --> 00:53:26,147
should I use Docker offload for first?

1002
00:53:26,450 --> 00:53:27,620
Yeah, so, okay, great.

1003
00:53:27,630 --> 00:53:31,160
You're wanting to play around these
agentic apps and, you know, we

1004
00:53:31,170 --> 00:53:35,450
were talking about not everybody
has access to high end GPUs or,

1005
00:53:35,450 --> 00:53:37,930
you know, M4 machines and whatnot.

1006
00:53:37,970 --> 00:53:41,800
great with the flip of a switch, and
you had it there in Docker Desktop,

1007
00:53:41,800 --> 00:53:45,440
but at the top you just flip a
switch, and now you're using offload.

1008
00:53:45,810 --> 00:53:51,990
and so now you've got access to a pretty
significant NVIDIA GPU, and additional

1009
00:53:51,990 --> 00:53:57,120
resources, and so yeah, as you're, We
see the use case, especially more for

1010
00:53:57,120 --> 00:54:00,640
the agent applications, because that's
where those resources are needed.

1011
00:54:01,580 --> 00:54:07,070
It does open up some interesting doors
for, maybe I'm just on a super lightweight

1012
00:54:07,100 --> 00:54:09,930
laptop that I'm using for school and
I don't have the ability to even run

1013
00:54:09,930 --> 00:54:11,640
a lot of my containerized workloads.

1014
00:54:12,220 --> 00:54:15,480
Great, I can use that for offload,
you know, offload that to the cloud.

1015
00:54:15,530 --> 00:54:19,460
it does open up some interesting
opportunities for, Use cases beyond

1016
00:54:19,460 --> 00:54:23,190
agentic apps, but that's kind of
where the big focus is right now.

1017
00:54:23,415 --> 00:54:26,765
So if you're like a Docker insider, or if
you're someone who's used Docker a while,

1018
00:54:27,420 --> 00:54:32,470
it's the Docker context command that we've
had forever, augmenting or changing the

1019
00:54:32,470 --> 00:54:36,100
environment variable docker underscore
host, which we've had since almost the

1020
00:54:36,100 --> 00:54:43,475
beginning, and it allows you from your
local Docker CLI, And even the GUI works

1021
00:54:43,475 --> 00:54:48,785
this way too, because I could always
set up a Docker remote engine and then

1022
00:54:49,075 --> 00:54:51,545
create a new context in the Docker CLI.

1023
00:54:52,025 --> 00:54:57,665
That would use SSH tunneling to go to that
server, and then I could run my Docker

1024
00:54:57,665 --> 00:55:02,015
CLI locally, my Compose CLI locally, and
it would technically be accessing and

1025
00:55:02,015 --> 00:55:06,615
running against, the remote host that I
had set up, but that was never really a

1026
00:55:06,615 --> 00:55:11,695
cloud service, like it was never, no one
provides Docker API access as a service

1027
00:55:11,695 --> 00:55:16,205
that I'm aware of, and, the context
command, while it's easy to use, and you

1028
00:55:16,205 --> 00:55:19,195
can actually use it on any command, you
can use docker run dash dash context,

1029
00:55:19,195 --> 00:55:22,815
I believe, or docker dash dash context
run, I can't remember the order, but,

1030
00:55:22,965 --> 00:55:24,125
you can change that on any command.

1031
00:55:24,275 --> 00:55:27,285
these are all things that existed,
but you made it, stupid easy.

1032
00:55:27,945 --> 00:55:30,445
It's just, like you said,
it's a toggle, it's so easy.

1033
00:55:30,885 --> 00:55:33,655
You just click that button
and then the UI changes, the

1034
00:55:34,015 --> 00:55:35,895
colors change, so now you know.

1035
00:55:36,365 --> 00:55:37,525
You're now remote.

1036
00:55:38,350 --> 00:55:41,830
Yeah, and so I'll go ahead and say too,
behind the scenes, it's using context.

1037
00:55:41,830 --> 00:55:43,250
It's using those exact things.

1038
00:55:43,610 --> 00:55:47,980
The tricky part is, because I've done
similar, development environments where,

1039
00:55:48,030 --> 00:55:52,280
I'm going to work against a Raspberry Pi
at home or, whatever else it might be.

1040
00:55:52,720 --> 00:55:56,050
the tricky part is when you want to
get into bind mounts, file sharing

1041
00:55:56,050 --> 00:55:58,620
kind of stuff, or port publishing,
and I want to be able to access

1042
00:55:58,620 --> 00:56:01,530
that port from my machine, like
automating all those different pieces.

1043
00:56:02,420 --> 00:56:03,730
That's not trivial.

1044
00:56:03,750 --> 00:56:04,900
I mean, it's possible.

1045
00:56:05,080 --> 00:56:07,810
a separate tool, yeah, you gotta
download ngrok or something,

1046
00:56:08,560 --> 00:56:12,310
And so this brings all that together
into a single offering here.

1047
00:56:12,902 --> 00:56:13,802
That's pretty amazing.

1048
00:56:13,812 --> 00:56:18,132
Like there's a lot going on underneath
the hood that, switch is hiding

1049
00:56:18,142 --> 00:56:19,922
a lot of different functionality.

1050
00:56:19,952 --> 00:56:20,152
Like

1051
00:56:20,152 --> 00:56:20,442
it's.

1052
00:56:20,997 --> 00:56:22,587
To make that very transparent

1053
00:56:23,250 --> 00:56:25,350
And this supports builds too, right?

1054
00:56:25,360 --> 00:56:25,890
So like.

1055
00:56:26,505 --> 00:56:30,475
When I toggle this in the
UI, or is there a CLI toggle?

1056
00:56:30,685 --> 00:56:31,305
Yeah, there is.

1057
00:56:31,591 --> 00:56:31,991
okay.

1058
00:56:32,251 --> 00:56:35,681
So if I toggle this, it's, yeah, you're
like, you're saying it's a context

1059
00:56:35,681 --> 00:56:41,161
change, but it's UI aware, and it takes
in all the other little things that we

1060
00:56:41,161 --> 00:56:42,761
don't think about until they don't work.

1061
00:56:42,761 --> 00:56:45,761
And then we're like, oh, yeah, it's
not really running locally anymore.

1062
00:56:45,761 --> 00:56:47,231
So now I can't use localhost colon.

1063
00:56:47,381 --> 00:56:49,631
Well, that all just, I'm going to show
you how this works and you don't even

1064
00:56:49,641 --> 00:56:52,721
have to know kind of like the rest of
the Dockery because you don't really

1065
00:56:52,721 --> 00:56:54,231
have to know how it works underneath.

1066
00:56:54,591 --> 00:56:57,531
but if you think it's too much magic,
I like to break it down and say,

1067
00:56:57,701 --> 00:56:59,241
it's just really the Docker context.

1068
00:56:59,271 --> 00:57:01,221
I don't actually look
anything at the code.

1069
00:57:01,241 --> 00:57:02,641
I don't know really how it's working.

1070
00:57:02,891 --> 00:57:06,391
But to me, when I went and checked,
it does change the context for me.

1071
00:57:06,391 --> 00:57:08,031
It actually injects it
and then removes it.

1072
00:57:08,311 --> 00:57:10,011
I did notice it from the CLI.

1073
00:57:10,211 --> 00:57:11,441
I could change context.

1074
00:57:11,867 --> 00:57:15,657
And it would retain the context, but
if I use the toggle button, it deletes

1075
00:57:15,657 --> 00:57:16,987
the context and then re adds it.

1076
00:57:17,107 --> 00:57:20,347
Regardless, it is in the
background, it's doing cool things.

1077
00:57:20,647 --> 00:57:23,817
I think the immediate request from
the captains was, can I do both?

1078
00:57:24,017 --> 00:57:30,157
Can I have, Per workload or per service
offload so that just my model's remote

1079
00:57:30,157 --> 00:57:34,087
and maybe that really big database
server and then all my apps are local.

1080
00:57:34,397 --> 00:57:38,497
I don't know why I would care, but like
it, that's something that people ask for.

1081
00:57:38,707 --> 00:57:40,927
I'm not sure that I
care that to that level.

1082
00:57:40,927 --> 00:57:44,857
I think I'm fine with either or, but I can
understand that if I'm running some things

1083
00:57:44,857 --> 00:57:49,167
locally already and I just want to add on
something in addition, it would be neat

1084
00:57:49,167 --> 00:57:51,477
if I could just choose for one service.

1085
00:57:52,462 --> 00:57:52,682
Yeah.

1086
00:57:52,682 --> 00:57:54,612
So as of right now, it
is an all or nothing.

1087
00:57:54,622 --> 00:57:55,742
you're doing everything local.

1088
00:57:55,742 --> 00:57:57,232
You're doing everything out in the cloud.

1089
00:57:57,232 --> 00:57:58,317
there's not a way to.

1090
00:57:58,787 --> 00:58:00,257
Split that up yet.

1091
00:58:00,307 --> 00:58:02,367
it's something that we've heard
from a couple of folks, but

1092
00:58:02,417 --> 00:58:05,237
again, it's that same thing of,
tell us more about the use cases.

1093
00:58:05,237 --> 00:58:08,447
So if that's a use case you have,
feel free to reach out to us and

1094
00:58:08,447 --> 00:58:13,907
help us better understand, why
you might want to split runtime.

1095
00:58:14,147 --> 00:58:16,977
hosting, split environment,
hybrid environment.

1096
00:58:17,237 --> 00:58:18,317
That's the correct term.

1097
00:58:18,476 --> 00:58:19,236
Why do you say it like that?

1098
00:58:19,436 --> 00:58:21,816
and just to be clear,
offload has its own cost.

1099
00:58:21,816 --> 00:58:24,416
like this isn't free forever for infinity.

1100
00:58:24,416 --> 00:58:26,246
You can't just take up a bunch of GPUs.

1101
00:58:26,506 --> 00:58:29,686
I was asking the team a little bit
and without getting too nerdy, it

1102
00:58:29,696 --> 00:58:33,826
sounds like it isolates, it spins up
a VM or there's maybe some hot VMs.

1103
00:58:33,826 --> 00:58:35,086
And I get a dedicated.

1104
00:58:36,026 --> 00:58:39,476
OS essentially it sounds like so
that I can get the GPU if I need.

1105
00:58:39,476 --> 00:58:43,206
And you kind of get an option of, do
I want GPU, servers with GPUs or not?

1106
00:58:43,216 --> 00:58:45,486
do I, am I going to run
GPU workloads or not?

1107
00:58:45,486 --> 00:58:48,476
And that affects pricing do we
get anything out of the box with

1108
00:58:48,476 --> 00:58:51,386
a Docker subscription or is it
just an, a completely separate

1109
00:58:51,726 --> 00:58:56,153
So actually it's a. Kind of private
beta, but yet people can sign up

1110
00:58:56,163 --> 00:58:57,353
for it and that kind of stuff.

1111
00:58:57,683 --> 00:59:01,633
folks will get the 300 GPU minutes,
which isn't a ton, but it's enough to

1112
00:59:01,653 --> 00:59:03,143
experiment and play around with it.

1113
00:59:03,443 --> 00:59:05,233
and then, start giving us feedback, etc.

1114
00:59:05,549 --> 00:59:08,169
Yeah, if you spin up the GPU
instance and then go to lunch, by

1115
00:59:08,169 --> 00:59:10,659
the time you get back, you'll have
probably used up your free minutes.

1116
00:59:10,738 --> 00:59:11,448
It's a long lunch,

1117
00:59:11,508 --> 00:59:12,668
hey, that's my kind of lunch.

1118
00:59:14,458 --> 00:59:17,998
but yeah, so we went an hour,
and we barely scratched the

1119
00:59:17,998 --> 00:59:19,228
surface, do we cover it all?

1120
00:59:19,298 --> 00:59:22,978
did we list at least all the
announcements of Major features and tools.

1121
00:59:22,978 --> 00:59:25,148
I don't even want to say we've covered
all the features because there's

1122
00:59:25,148 --> 00:59:26,798
probably some stuff with MCP we missed.

1123
00:59:27,188 --> 00:59:31,338
So you open source MCP gateway, but
we should point out you don't actually

1124
00:59:31,338 --> 00:59:36,488
have to know, like you can just use
Docker desktop and MCP tools locally.

1125
00:59:37,028 --> 00:59:40,618
But the reason you provide an
MCP gateway is open source is so

1126
00:59:40,618 --> 00:59:44,478
we could put it in the compose
file and then run it on servers.

1127
00:59:44,528 --> 00:59:45,498
think about it this way.

1128
00:59:45,518 --> 00:59:48,478
the MCP toolkit bundled with Docker
Desktop is going to be more for,

1129
00:59:48,728 --> 00:59:52,678
I'm consuming, I'm just wanting to
use MCP servers and connect them

1130
00:59:52,678 --> 00:59:54,558
to my other agentic applications.

1131
00:59:54,868 --> 00:59:57,888
And the MCP Gateway is going to
be more for, now I want to build

1132
00:59:57,888 --> 01:00:01,948
my own agentic applications and
connect those tools to, Those, those

1133
01:00:01,948 --> 01:00:03,508
applications that we're running there.

1134
01:00:03,949 --> 01:00:04,389
Yeah.

1135
01:00:04,729 --> 01:00:07,229
Do you see people using
MCP Gateway in production?

1136
01:00:07,249 --> 01:00:10,819
Do you see that as like a. Not that
you provide support or anything like

1137
01:00:10,819 --> 01:00:12,499
that, but is it designed so that

1138
01:00:13,499 --> 01:00:16,989
We've got a couple of folks that
are already starting to do so.

1139
01:00:17,059 --> 01:00:19,609
stay tuned for some use
case stories around that.

1140
01:00:19,659 --> 01:00:19,939
Yeah.

1141
01:00:20,569 --> 01:00:20,919
Awesome.

1142
01:00:21,419 --> 01:00:22,329
well, this is a lot.

1143
01:00:22,359 --> 01:00:28,139
I feel like I need to launch another
10 Docker YouTube uploads just

1144
01:00:28,149 --> 01:00:31,319
to cover each tool specifically,
each use case specifically.

1145
01:00:31,609 --> 01:00:35,314
there's a lot here, but
this is Amazing work.

1146
01:00:35,314 --> 01:00:39,884
I mean, I don't know if you have a fleet
of AI robots working for you yet, but

1147
01:00:40,118 --> 01:00:44,348
certainly feels like a lot of different
products that are all coming together very

1148
01:00:44,348 --> 01:00:48,678
quickly that are all somehow related to
each other, but also independently usable.

1149
01:00:49,143 --> 01:00:54,103
And, I'm having you on the show as
usual is a great way to break it down

1150
01:00:54,133 --> 01:00:58,233
into the real usable bits What do
we really care about without all the

1151
01:00:58,233 --> 01:01:03,053
marketing, hype of general AI hype, which
is always a problem on the internet.

1152
01:01:03,053 --> 01:01:05,173
But this feels like really useful stuff.

1153
01:01:05,203 --> 01:01:05,793
Um,

1154
01:01:08,058 --> 01:01:09,268
Eivor, another podcast.

1155
01:01:09,268 --> 01:01:10,738
I don't know, Eivor, what's up?

1156
01:01:10,748 --> 01:01:13,038
Are you requesting yet another podcast?

1157
01:01:13,398 --> 01:01:14,028
Um,

1158
01:01:14,432 --> 01:01:15,132
a whole new show

1159
01:01:15,178 --> 01:01:17,668
about Compose provider services?

1160
01:01:18,088 --> 01:01:18,888
Oh, yes.

1161
01:01:18,928 --> 01:01:24,448
Also, you can now run Compose directly
from, well, you can use Compose YAML

1162
01:01:24,468 --> 01:01:27,618
from directly inside of cloud tools.

1163
01:01:28,173 --> 01:01:29,923
The first one was Google Cloud Run.

1164
01:01:30,253 --> 01:01:33,373
So I could technically spin
up Google, which I love,

1165
01:01:33,383 --> 01:01:34,883
Google Cloud Run is fantastic.

1166
01:01:35,203 --> 01:01:38,403
Um, it would be my first choice
for running any containers in

1167
01:01:38,403 --> 01:01:39,673
Google if I was using Google.

1168
01:01:40,193 --> 01:01:45,902
Um, and so now they're, accepting
the Compose YAML, spec, essentially,

1169
01:01:46,422 --> 01:01:48,282
inside of their command line.

1170
01:01:48,559 --> 01:01:51,039
So this is like, this feels like the
opposite of what Docker used to do.

1171
01:01:51,039 --> 01:01:54,339
Docker used to build in cloud
functionality into the Docker tooling.

1172
01:01:54,339 --> 01:01:57,769
But now we're saying, Hey, let's partner
with those tools, those companies,

1173
01:01:57,989 --> 01:02:03,489
and let them build in cloud or
compose specification into their tool.

1174
01:02:03,849 --> 01:02:06,419
So we can have basically file reuse.

1175
01:02:06,479 --> 01:02:07,209
YAML reuse.

1176
01:02:07,209 --> 01:02:07,919
Is that right?

1177
01:02:08,549 --> 01:02:08,829
yeah.

1178
01:02:08,829 --> 01:02:13,249
So this is the first time exactly in which
it's not Docker tooling that's providing

1179
01:02:13,249 --> 01:02:15,959
the cloud support, but it's cloud native.

1180
01:02:16,216 --> 01:02:19,406
They're the ones building the tooling
and consuming the Compose file.

1181
01:02:19,590 --> 01:02:21,160
yeah, it's a big moment.

1182
01:02:21,170 --> 01:02:25,530
And as we work with Google Cloud
on this, yeah, you can deploy the

1183
01:02:25,530 --> 01:02:27,020
normal container workloads, etc.

1184
01:02:27,020 --> 01:02:30,890
But they already have support for Model
Runner to be able to run the models

1185
01:02:30,890 --> 01:02:34,460
there as well, it's pretty exciting
And I know, the provider services

1186
01:02:34,520 --> 01:02:37,410
this is how we started with models.

1187
01:02:37,815 --> 01:02:39,555
having support in Compose, where

1188
01:02:39,555 --> 01:02:44,964
That was another service
in which the service wasn't

1189
01:02:45,024 --> 01:02:46,704
backed by a normal container.

1190
01:02:47,174 --> 01:02:47,984
the old method.

1191
01:02:48,034 --> 01:02:52,894
yes, but what's cool about this is, so
first off, these hooks still are in place.

1192
01:02:53,309 --> 01:02:57,259
So that a Compose file can basically
delegate off to this additional

1193
01:02:57,279 --> 01:03:00,899
provider plugin to say, hey, this is
how you're going to spin up a model.

1194
01:03:01,159 --> 01:03:04,029
But it starts to open up a whole
ecosystem where anybody can make a

1195
01:03:04,029 --> 01:03:08,779
provider or, okay, hey, I've got this
cloud based database, just as an example.

1196
01:03:09,069 --> 01:03:12,429
And, okay, Now I can still use
Compose and it's going to spin up

1197
01:03:12,429 --> 01:03:17,439
my containers, but also create this
cloud based container and then inject

1198
01:03:17,629 --> 01:03:19,669
environment variables into my app.

1199
01:03:19,719 --> 01:03:22,039
again, it starts to open up
some pretty cool extensibility

1200
01:03:22,039 --> 01:03:23,589
capabilities of Compose as well.

1201
01:03:23,639 --> 01:03:28,279
I think we, yeah, we need to bring Michael
back just to dig into that because, it's

1202
01:03:28,279 --> 01:03:31,359
essentially like extensions or, Plugins

1203
01:03:31,807 --> 01:03:32,197
Yeah.

1204
01:03:32,328 --> 01:03:32,478
for

1205
01:03:32,507 --> 01:03:35,467
Yeah, so Compose is about to
get a whole lot more love.

1206
01:03:35,477 --> 01:03:38,777
It feels like it's already, I
mean, it's been years since we've

1207
01:03:38,777 --> 01:03:41,007
added a root extension or like a,

1208
01:03:41,158 --> 01:03:41,738
top level,

1209
01:03:41,947 --> 01:03:42,857
top level build.

1210
01:03:43,497 --> 01:03:46,247
it's not every day that Docker
decides there's a whole new

1211
01:03:46,807 --> 01:03:48,617
type of thing that we deploy.

1212
01:03:48,617 --> 01:03:52,127
Now we have models, we'll see if
providers someday become something.

1213
01:03:52,427 --> 01:03:52,777
that'll be cool.

1214
01:03:53,237 --> 01:03:56,687
and this is all due to the Compose
spec, which now allows other

1215
01:03:56,687 --> 01:03:59,877
tools to use the Compose standard.

1216
01:03:59,887 --> 01:04:02,157
And that's just great for everybody,
because everybody uses Compose.

1217
01:04:02,167 --> 01:04:05,117
it's like the most universal
YAML out there, in my opinion.

1218
01:04:05,527 --> 01:04:05,997
great.

1219
01:04:06,087 --> 01:04:07,677
Well, I think we've covered it all.

1220
01:04:07,917 --> 01:04:10,907
Nirmal and I need, another
month to digest all this, and

1221
01:04:10,907 --> 01:04:11,997
then we'll invite you back on.

1222
01:04:12,367 --> 01:04:12,627
do it.

1223
01:04:13,177 --> 01:04:16,487
but yeah, we've checked the
box of, Everything Docker first

1224
01:04:16,487 --> 01:04:18,917
half of the year, stay tuned
for the second half of the year.

1225
01:04:19,137 --> 01:04:22,357
I actually sincerely hope you don't
have as busy of a second half,

1226
01:04:22,387 --> 01:04:25,027
just because it's, these are a lot
of videos I got to make, you're

1227
01:04:25,027 --> 01:04:26,727
putting a lot of work into my, inbox,

1228
01:04:26,787 --> 01:04:28,607
We're helping you have content to create.

1229
01:04:28,903 --> 01:04:32,563
I know, yeah, there's no shortage of
content to create right now with Docker.

1230
01:04:32,903 --> 01:04:34,813
I am very excited to play
with all these things.

1231
01:04:34,813 --> 01:04:37,253
I sound excited because I am excited.

1232
01:04:37,253 --> 01:04:42,123
This is real stuff that I think is
beneficial and largely free, largely,

1233
01:04:42,438 --> 01:04:45,678
Like almost all of this stuff is
really just extra functionality that

1234
01:04:45,678 --> 01:04:49,218
already exists that in our tooling
without adding a whole bunch of SaaS

1235
01:04:49,218 --> 01:04:50,808
services we have to buy on top of it.

1236
01:04:51,168 --> 01:04:52,728
yeah, so congrats.

1237
01:04:53,728 --> 01:04:55,958
People can find out more docker.com.

1238
01:04:56,298 --> 01:05:00,368
docs.Docker.com, dockers got videos
on YouTube now they're putting up

1239
01:05:00,368 --> 01:05:02,618
YouTube videos, so check that out.

1240
01:05:02,768 --> 01:05:05,443
I saw Michael putting up some
videos recently on LinkedIn.

1241
01:05:06,443 --> 01:05:07,203
it's all over the place.

1242
01:05:07,203 --> 01:05:10,063
You can follow Michael Irwin on LinkedIn.

1243
01:05:10,113 --> 01:05:11,013
he's on BlueSky.

1244
01:05:11,013 --> 01:05:11,943
I think you're on BlueSky.

1245
01:05:12,516 --> 01:05:13,446
I think you're on BlueSky.

1246
01:05:14,256 --> 01:05:16,416
Um, or, or, where, yeah,

1247
01:05:16,441 --> 01:05:17,451
figured out where I'm hanging out.

1248
01:05:17,883 --> 01:05:18,753
Thanks so much for being here.

1249
01:05:18,934 --> 01:05:19,474
Thank you, Michael.

1250
01:05:19,474 --> 01:05:21,694
Thank you, Nirmal, for
joining and staying so long.

1251
01:05:22,087 --> 01:05:22,867
I'll see you in the next one.