TRANSKRIP 🔗
WEBVTT
NOTE
Transcription provided by Podhome.fm
Created: 07/03/2026 15:22:52
Duration: 2946.795
Channels: 1
1
00:00:04.000 --> 00:00:05.920
Oh, hello, Larry.
2
00:00:06.560 --> 00:00:07.280
Hello,
3
00:00:07.520 --> 00:00:08.400
Anton.
4
00:00:09.760 --> 00:00:13.280
Hey. Good to meet you. This is the first time we actually see each other.
5
00:00:13.884 --> 00:00:15.245
Yeah. Thanks
6
00:00:15.245 --> 00:00:16.445
for doing this, man.
7
00:00:17.404 --> 00:00:19.324
Oh, my pleasure. That was an
8
00:00:19.884 --> 00:00:22.364
interesting topic, so, you know,
9
00:00:23.005 --> 00:00:25.164
why not to why not to try it?
10
00:00:25.564 --> 00:00:26.125
Yeah.
11
00:00:26.525 --> 00:00:30.739
So maybe for the listeners, can you introduce yourself?
12
00:00:31.300 --> 00:00:40.019
Absolutely. What do you do? Who you are? Yeah. My name is Anton. I'm based in The Netherlands in a beautiful city that is called Harlem,
13
00:00:40.500 --> 00:00:41.699
right next to Amsterdam.
14
00:00:42.135 --> 00:00:45.495
I work for GitHub, has been four years already.
15
00:00:45.734 --> 00:00:48.214
I'm a solutions engineer. So I support
16
00:00:48.454 --> 00:00:50.135
large enterprise customers
17
00:00:50.614 --> 00:00:51.655
across, well,
18
00:00:52.375 --> 00:00:54.695
North Of EMEA, so that's been a
19
00:00:55.254 --> 00:00:59.570
plus all the Nordics, some UK in the past. Those are
20
00:00:59.890 --> 00:01:02.850
GitHub customers, Microsoft customers, and we,
21
00:01:03.090 --> 00:01:06.370
well, we work a lot with them, improving their
22
00:01:07.250 --> 00:01:14.575
SDLC, like the way they build software, the way they host code. And for the past few years, it was mostly,
23
00:01:15.455 --> 00:01:18.815
well, a lot at least about AI,
24
00:01:19.215 --> 00:01:22.095
gen AI solutions like GitHub Copilot that
25
00:01:22.815 --> 00:01:24.095
we have. So
26
00:01:24.910 --> 00:01:30.670
been an interesting journey since we started in the beginning. It was basically like a super simple auto complete,
27
00:01:30.910 --> 00:01:39.310
and we were really really busy trying to convince people that that thing actually works. Like, yeah, it's it looks weird, it's a surprise,
28
00:01:40.835 --> 00:01:43.314
but indeed computer can write code. It was
29
00:01:44.115 --> 00:01:48.835
if you look back right now, it was not the best experience ever because obviously
30
00:01:48.835 --> 00:01:50.594
models were very simple,
31
00:01:51.875 --> 00:01:57.280
the industry itself didn't really know how to tweak those models and how to move them and how to
32
00:01:58.479 --> 00:02:00.719
make sure the results are great.
33
00:02:01.280 --> 00:02:01.920
But we
34
00:02:02.240 --> 00:02:11.655
even then, we had a few large customers who agreed and trusted us and then joined the, you know, the joined us on this journey.
35
00:02:12.135 --> 00:02:19.175
And now, of course, Copilot is a massive beast. There is so much in there. There is a lot of surfaces where you can engage with it.
36
00:02:19.495 --> 00:02:24.775
But the question now is not about can I write code or not? The question is mainly about
37
00:02:25.040 --> 00:02:34.880
how can we do that at scale? How can we teach everyone? Are we doing it right? Are we doing it wrong? So all these questions, that's what I'm basically being
38
00:02:35.200 --> 00:02:42.195
busy with, helping helping everyone and also trying to learn from them as well because it's a new
39
00:02:42.915 --> 00:02:49.875
still for for many for many engineers, it's still somewhat new. So, yeah, it's a that's what I do with GitHub.
40
00:02:50.355 --> 00:02:52.675
That's fun. Yeah. That's lots of fun.
41
00:02:53.155 --> 00:02:54.275
Absolutely. Yes.
42
00:02:54.830 --> 00:02:57.710
Do you feel like a like a microsofter?
43
00:03:00.430 --> 00:03:01.230
Depends.
44
00:03:01.630 --> 00:03:03.950
Depends. We are still somewhat
45
00:03:04.750 --> 00:03:07.150
separate, so there is a lot
46
00:03:08.030 --> 00:03:10.190
the differences in the tooling that we use,
47
00:03:10.895 --> 00:03:13.215
the difference in the way we approach
48
00:03:13.614 --> 00:03:14.895
some conversations.
49
00:03:15.455 --> 00:03:19.375
We do have a lot of customers that are large Microsoft customers
50
00:03:19.375 --> 00:03:28.220
and GitHub customers at the same time, so for them, it doesn't matter. Right? So it's the same, well, what we call an account team. It's a mix of just
51
00:03:28.460 --> 00:03:29.260
GitHub
52
00:03:29.260 --> 00:03:40.220
and a lot of Microsoft employees there. But we have customers that don't really have a lot to do with Microsoft, so they they they mainly, like, could be AWS shop or GCP shop.
53
00:03:40.785 --> 00:03:41.425
And
54
00:03:41.665 --> 00:03:42.945
for them, we are
55
00:03:43.505 --> 00:03:44.465
they don't really
56
00:03:44.705 --> 00:03:46.625
think of us as Microsoft,
57
00:03:46.864 --> 00:03:48.625
but they think of us as GitHub
58
00:03:48.625 --> 00:03:50.465
as a separate unit.
59
00:03:50.705 --> 00:03:54.000
So, yeah, it depends who's asking. But of course,
60
00:03:54.560 --> 00:03:55.760
since the acquisition,
61
00:03:56.480 --> 00:04:00.960
there is some sort of transition happening. So we recently unified
62
00:04:01.040 --> 00:04:01.760
our
63
00:04:02.239 --> 00:04:05.920
office tools. So we've been a lot on the Google
64
00:04:06.480 --> 00:04:08.995
Workspace or Google Docs, Gmail,
65
00:04:09.795 --> 00:04:15.635
but now we are we've we've shifted over to Outlook, so that helps us better collaborate with
66
00:04:15.715 --> 00:04:17.315
our Microsoft colleagues
67
00:04:17.555 --> 00:04:23.475
because, well, at least we can share calendars now. That that makes much easier. So you feel more and more connected now. Yeah.
68
00:04:24.250 --> 00:04:27.930
Okay. Are you enjoying that Teams and stuff?
69
00:04:29.290 --> 00:04:31.850
Well, I've been it's been an experience.
70
00:04:32.010 --> 00:04:40.005
Let's put it this way. So you
Load full transcript
definitely change your habits. You have to, you know Yeah. UX is slightly different. Like, in the end of the day, just email.
71
00:04:40.245 --> 00:04:44.645
So that's kind of same thing. Calendars. Yeah. It's a calendar like,
72
00:04:44.965 --> 00:04:45.525
obviously,
73
00:04:46.645 --> 00:04:50.325
certain things are a little bit different, but we've got quite a good support
74
00:04:50.485 --> 00:04:54.325
for the transition. So we had a lot of engineers actively
75
00:04:54.420 --> 00:04:55.780
working with us,
76
00:04:56.180 --> 00:04:58.260
resolving, helping with the migration,
77
00:04:58.340 --> 00:04:58.820
and
78
00:05:00.180 --> 00:05:02.900
I don't know. At this point, like, it's been more like
79
00:05:03.300 --> 00:05:06.420
five months or so, maybe a little bit less than that.
80
00:05:06.900 --> 00:05:08.660
So, at this point, it's fine.
81
00:05:09.505 --> 00:05:12.705
Okay. Yeah. I was just just wondering what that's like.
82
00:05:13.825 --> 00:05:21.105
Okay. So you mentioned already GitHub Copilot in the very beginning. It's a very different thing than than it is now.
83
00:05:21.840 --> 00:05:26.320
I've been using it for years now, which makes me a veteran user.
84
00:05:26.719 --> 00:05:27.360
And
85
00:05:27.680 --> 00:05:45.605
back in the day we had, you know, the the value proposition of it was very different than now because it couldn't do the same things as now. So what do you think it was back then and how did you convince those customers to use it? Was it basically just a replacement for Stack Overflow or something else?
86
00:05:46.965 --> 00:05:48.085
Wow,
87
00:05:48.650 --> 00:05:49.530
That's
88
00:05:49.530 --> 00:05:51.370
that's a trip down the memory line.
89
00:05:52.170 --> 00:06:02.730
Yeah. You know, at least for me, because I do work with a lot with like regulated customers and FSIs and banks and, you know, manufacturing.
90
00:06:04.010 --> 00:06:06.425
For for them, it was more
91
00:06:06.985 --> 00:06:09.305
the biggest concern was privacy,
92
00:06:10.104 --> 00:06:11.785
what happens with data,
93
00:06:12.025 --> 00:06:13.465
other any legal
94
00:06:14.104 --> 00:06:15.465
implications that
95
00:06:16.585 --> 00:06:22.720
what it even means, like if you had something that generates your code, what does it mean in terms of the
96
00:06:23.199 --> 00:06:24.320
ownership,
97
00:06:25.120 --> 00:06:26.720
in terms of the licensing,
98
00:06:26.720 --> 00:06:28.320
and what's not. So
99
00:06:28.960 --> 00:06:29.680
mostly,
100
00:06:30.080 --> 00:06:35.200
I had to go and show that it works. So that was the first thing to do, of course.
101
00:06:35.919 --> 00:06:38.815
That it works not just on the examples.
102
00:06:38.895 --> 00:06:46.895
So, like, you you can literally open any code base and just start typing and you're getting those suggestions in. And they,
103
00:06:47.375 --> 00:06:47.775
to
104
00:06:48.460 --> 00:06:56.460
the most to like, lot of people were surprised to see that, yep, it does it. Of course, back then we had like what we used to call hallucinations.
105
00:06:56.460 --> 00:07:01.580
Right? So it's like, it would invent some APIs that don't exist. It would invent some
106
00:07:02.194 --> 00:07:02.995
packages
107
00:07:02.995 --> 00:07:05.235
that are outdated or
108
00:07:05.315 --> 00:07:10.195
like it wouldn't know about the internal packages that every customer or company has.
109
00:07:11.395 --> 00:07:16.035
But the second piece was almost like, are we even allowed to use it? Like, what happens to the data?
110
00:07:17.169 --> 00:07:24.850
So there was a lot of time where I had to spend understanding like all the details, like how the data flows, what do we do with it?
111
00:07:25.650 --> 00:07:31.705
I think the biggest question was, does it learn? Like, not really. Right? So those models are like, stat
112
00:07:32.185 --> 00:07:34.665
they they they have no memory. There is no
113
00:07:35.625 --> 00:07:36.504
yeah. It's
114
00:07:37.465 --> 00:07:41.305
there is nothing that you can can send to the model and then it's stuck there.
115
00:07:42.530 --> 00:08:05.945
So that was a was a bit of it. I wouldn't I wouldn't we never positioned it as a like a replacement for Stack Overflow. It's initially, it was just very, very smart autocomplete that can complete not just the next, you know, method name or function name, but it was actually building your lines of code, like, at the same time. And the way you steer it was very different
116
00:08:05.945 --> 00:08:12.920
to the way you steer it right now. So, like, if you speak about autocomplete, you don't do anything, you just type. Right? And you get like all these Yeah.
117
00:08:13.320 --> 00:08:15.080
Lines offered
118
00:08:15.080 --> 00:08:18.120
to you and what we call a next edit suggestion.
119
00:08:18.520 --> 00:08:19.080
So
120
00:08:19.480 --> 00:08:20.920
just picking
121
00:08:21.240 --> 00:08:26.360
the the different part of the file that you're editing and saying like, oh, you're introduced to
122
00:08:26.875 --> 00:08:32.955
a property. So probably you want to initially start the go to the constructor
123
00:08:32.955 --> 00:08:35.755
and put, you know, the past the value in.
124
00:08:36.155 --> 00:08:48.180
So these things improve a lot. But no, it wasn't really a replacement for Stack Overflow because you still had to explore and research, like, what what I'm planning to build. And I think, like, for me,
125
00:08:48.740 --> 00:08:50.100
the eye opening
126
00:08:50.180 --> 00:08:53.380
moment was, like, once we got the preview internally,
127
00:08:54.015 --> 00:08:58.655
I felt like, that's interesting. Like, let let me play with it. And I took a weekend.
128
00:08:59.295 --> 00:09:05.375
I had I have a few personal projects that I run for some home automation, for some, like,
129
00:09:06.095 --> 00:09:07.855
photo management tool I have.
130
00:09:08.519 --> 00:09:26.005
And I just had a few features in mind that I
never had time to to to go back to. And I just sat and been coding for the weekend, and I was like, wow, I'm so much more productive now. Like, I can do so much and it just, you know, after a while you're gonna get used to it, and it, like, it just does it for me.
131
00:09:26.485 --> 00:09:31.205
That was such a brilliant experience. It got me really pumped and motivated to go and show.
132
00:09:33.365 --> 00:09:37.525
And if I look back at this right now, I was like, oh my god. There was so many old.
133
00:09:38.565 --> 00:09:38.885
Yeah.
134
00:09:40.110 --> 00:10:13.480
Yeah. But still felt magical. Right? Just like Yeah. The first time you use a cloud service or something. Yeah. Yeah. At this point, the magic, I think it's gone. Like, now, you at least I understand it very well. Like, what exactly is happening there? What's what's what's in the on the back end plus. Being a GitHub employee, I well, we we promote inner sourcing a lot, so I can see the code base of those tools. If I need to understand something, can just open the repos, spin up Copilot, and ask it to find draw me some diagrams or even find particular pieces in the code that
135
00:10:13.959 --> 00:10:14.680
do Plus, something
136
00:10:15.880 --> 00:10:22.200
I think the second experience that helped me a lot was to build my own coding agent, like in the CLI coding agent.
137
00:10:22.695 --> 00:10:37.510
It wasn't very it's not like super advanced or anything, but it just helps you to see what's what's behind, and then it helps you to to drive it when you use any of them. Like, it's not just about Copilot. Like, they're all essentially the same insight, like, the same
138
00:10:38.710 --> 00:10:50.150
two nested loops, basically. So yeah. The those things help a lot. Like, the more you understand, the less magic you see there and the more, like, okay, it's a tool. It's a tool I can use. I can I can maybe trust it more?
139
00:10:50.895 --> 00:10:55.375
So, yeah, it was different. But back then, it was, yeah, true magic. It was like, wow.
140
00:10:55.935 --> 00:10:56.975
Yeah.
141
00:10:57.295 --> 00:10:58.015
Exactly.
142
00:10:58.575 --> 00:11:01.775
And now, it is a lot better and
143
00:11:01.775 --> 00:11:07.149
different as well. It can do a lot It is very diff. A more tools and buttons and bells and whistles.
144
00:11:07.470 --> 00:11:09.950
Mhmm. So how do you use it now?
145
00:11:11.709 --> 00:11:14.029
I went through, like, many stages,
146
00:11:14.269 --> 00:11:14.750
which
147
00:11:15.149 --> 00:11:16.110
to my
148
00:11:16.865 --> 00:11:23.505
what I call it a surprise, maybe not to my surprise, but I observed the same patterns when I speak to my customers,
149
00:11:23.505 --> 00:11:26.545
especially the long term that could've been with us for several years already.
150
00:11:27.425 --> 00:11:36.220
So you start with less trust and more control and you read what the output, then of course we got chat, which was also a bit
151
00:11:36.779 --> 00:11:37.339
yeah.
152
00:11:37.740 --> 00:11:42.394
Currently, if you look at it right now, it's like it wasn't a great experience. But
153
00:11:42.394 --> 00:11:49.834
like after, I think the first breakthrough was of course agentic capabilities. Like when we first implemented
154
00:11:49.915 --> 00:11:51.595
agentic capabilities,
155
00:11:51.915 --> 00:11:54.714
it was a tool we called GitHub
156
00:11:54.990 --> 00:11:56.510
Copilot Workspace.
157
00:11:57.310 --> 00:11:58.190
It's not
158
00:11:58.430 --> 00:12:00.910
non existent anymore. Was an experiment again.
159
00:12:01.390 --> 00:12:03.310
And it was a coding agent in the cloud.
160
00:12:03.950 --> 00:12:17.225
Models were like GPT three five or something goes like really weak model, so they didn't have a lot of context. So it was a a on cloud on the experience and it wasn't great, but it gave the idea that it's things possible.
161
00:12:17.704 --> 00:12:19.065
And then after,
162
00:12:19.945 --> 00:12:25.810
I think oh, I don't remember. I think it was Cursor who got the proper Cursor and Vinsor who got the proper
163
00:12:26.769 --> 00:12:35.009
agent flow because they got better models and they jumped on that ship. We were a little bit late to the game, had to fix a few things first,
164
00:12:36.205 --> 00:12:37.725
and then got Copilot
165
00:12:38.845 --> 00:12:39.965
agent, like,
166
00:12:40.285 --> 00:12:42.765
in the agent mode, we used to call it,
167
00:12:43.165 --> 00:12:45.725
in in the ID, in Versus Code, and other IDs.
168
00:12:45.885 --> 00:12:46.845
And that was the first,
169
00:12:47.820 --> 00:12:50.700
wasn't still great, like it does the job.
170
00:12:51.580 --> 00:12:53.660
But I think somewhere around
171
00:12:53.900 --> 00:12:59.340
end of last year or mid last year when the models like Sonnet three three seven, three five
172
00:13:00.515 --> 00:13:01.555
were released,
173
00:13:01.875 --> 00:13:03.315
then you see the power.
174
00:13:04.035 --> 00:13:06.115
And after that, like, I completely
175
00:13:06.355 --> 00:13:17.130
stopped coding, like writing code manually, that was on purpose, like, you would probably still want to let the agent finish and then go and, you know, do the manual, like, polishing.
176
00:13:17.450 --> 00:13:36.005
And that's what most of the engineers been doing back then. But my goal is to understand the art of possible, like how far I can push it. So I decided like, okay, I'm not writing any code manually anymore and just do with the agent, like learn how to bend it basically to to make sure it does what I want to do. And then
177
00:13:36.325 --> 00:13:41.125
I think like by the end of the year when like we we were we went into this
178
00:13:42.200 --> 00:13:56.680
really well, not we, we as an industry, not just GitHub, into this mode where the new frontier model is being released every what is it now? Like, every couple weeks or so. Right? Yeah. Yeah. So it's just insane how fast those things develop.
179
00:13:57.215 --> 00:13:57.775
And
180
00:13:58.415 --> 00:14:09.855
every single release was like felt like, wow, magic. Cool. It's so much better. I have I don't have to control it as much. It it knows a lot, and it gets better in using the tools and exploring the code base.
181
00:14:10.680 --> 00:14:18.920
But I already knew how to to push it because it was my personal goal. Not do not write anything manually, make sure the agent can do it for you.
182
00:14:19.480 --> 00:14:22.440
So I believe it was a very useful experience because
183
00:14:22.855 --> 00:14:32.455
all those engineers who's been using those tools for a long time, at this point, they're super good at it. Like, they they understand how to fix those little flows,
184
00:14:32.775 --> 00:14:34.855
how to provide tech,
185
00:14:35.740 --> 00:14:45.900
what we call right now, context engineering. Right? Like how to make sure that you give enough information that is good and useful for the model and as as as little as possible, so you maintain the context.
186
00:14:46.140 --> 00:14:52.475
Right? We know that, yeah, if it's too much in the context, it's not gonna be the great output if it's
187
00:14:52.715 --> 00:14:59.195
it's something distracting. So you have to and and and the context windows were very small back then. So you would have to, like, manually
188
00:14:59.675 --> 00:15:02.360
even, like, drag in files or something. And
189
00:15:03.480 --> 00:15:08.520
now it's not the case anymore. So I think somewhere around the
190
00:15:08.920 --> 00:15:10.200
the end of last
191
00:15:10.600 --> 00:15:11.800
year, I
192
00:15:12.200 --> 00:15:13.720
started moving
193
00:15:13.720 --> 00:15:14.600
towards
194
00:15:14.600 --> 00:15:15.959
the CLI experience.
195
00:15:16.645 --> 00:15:22.005
Still wouldn't say that it's my favorite. I don't yeah. I miss a lot in the CLI,
196
00:15:22.404 --> 00:15:27.365
but that was a moment where you realize that I have more trust
197
00:15:27.845 --> 00:15:31.605
to the like, I trust the tool now, so I can
198
00:15:32.010 --> 00:15:33.450
afford
199
00:15:33.450 --> 00:15:39.770
not to look at the every single piece of code that is that the tool generates for me, but rather,
200
00:15:40.170 --> 00:15:40.810
like,
201
00:15:41.290 --> 00:15:49.635
work in a more like outcome driven way. Like, I want that to happen in my code base, in my application, or even in experiment when which I built.
202
00:15:49.955 --> 00:15:50.435
And
203
00:15:51.235 --> 00:15:54.835
only then, like, run the code reviews, run the security
204
00:15:54.835 --> 00:15:59.154
tools to make sure that it's fine to to to merge in and and and deploy.
205
00:15:59.740 --> 00:16:03.980
And that that was like a a shift. Right? But now with
206
00:16:04.220 --> 00:16:07.340
more visual tools like we have in Versus Code, we have
207
00:16:08.780 --> 00:16:19.345
Versus Code sessions or agents. It was renamed to agents now, and the GitHub Copilot app. So Copilot app is probably the the one
I'm using 99%
208
00:16:19.345 --> 00:16:23.665
of the time Mhmm. At at the moment. I love the progress there. And
209
00:16:24.145 --> 00:16:31.720
that's that gives me like this combined experience of what I like about CLI, that it's a focus on the process and the outcome.
210
00:16:32.120 --> 00:16:34.520
And what I was missing from
211
00:16:35.160 --> 00:16:51.385
the ID perspective, like everything gets integrated, you can easily switch to see the codes, you can, like, there's a button, just open me open it in in Versus Code or another editor and just, you know, run the debugger and test it and, you know, do do like get more into the
212
00:16:51.705 --> 00:16:52.985
you know, get your hands dirty.
213
00:16:54.280 --> 00:16:58.520
So yeah. So there were like periods and transitions. And the same I see with
214
00:16:58.680 --> 00:17:03.000
with the customers, like when we deploy it at scale and then we help them monitor and understand,
215
00:17:03.160 --> 00:17:09.634
we see the same. The users that are less experienced, they tend to stick more to typing code,
216
00:17:10.755 --> 00:17:14.114
running agents and double checking and verifying and manually
217
00:17:14.434 --> 00:17:15.075
editing.
218
00:17:15.475 --> 00:17:22.274
Then there is like a second tier where I would say, or phase, let's be let's call it the phase, where engineers are mostly
219
00:17:22.355 --> 00:17:23.554
trying to
220
00:17:24.490 --> 00:17:26.570
rely more on on agents.
221
00:17:26.570 --> 00:17:31.129
And you can I mean, we don't know what exactly they do? Like, we don't collect
222
00:17:31.370 --> 00:17:33.289
prompt and suggestions, so we don't know.
223
00:17:33.610 --> 00:17:36.809
But when you look at the patterns, you almost like see
224
00:17:37.130 --> 00:17:43.114
from the telemetry that customer show me, oh, this is what we have from GitHub. Like, can you explain us? Can you explain us?
225
00:17:43.355 --> 00:17:49.914
And you're gonna start recognizing the patterns. You see that, look, that person is doing this more organic foreclose.
226
00:17:49.914 --> 00:17:51.275
And you can see the outcome.
227
00:17:51.799 --> 00:17:57.400
Like if you cross check with other metrics, they produce more code, they produce more pull requests, they produce more releases,
228
00:17:57.400 --> 00:17:58.440
they close more
229
00:17:59.400 --> 00:18:02.119
tasks. They actually have more time to
230
00:18:02.440 --> 00:18:04.200
do fixing
231
00:18:04.919 --> 00:18:05.960
the tech debt.
232
00:18:07.065 --> 00:18:07.624
So
233
00:18:08.184 --> 00:18:13.705
and then the next stage is like running those in parallel. And this is where you need orchestrators,
234
00:18:13.705 --> 00:18:15.465
where you need cloud
235
00:18:15.625 --> 00:18:24.929
cloud agent when you more rely on the integrations between tools. So, like, the typical pipeline for me would be, I've got a plan, I've got an idea,
236
00:18:25.250 --> 00:18:29.409
spin up a few agents to run, make sure they cross check each other,
237
00:18:29.890 --> 00:18:31.890
get me a pull request, get
238
00:18:32.130 --> 00:18:34.930
the code review done, get the quality controls done.
239
00:18:35.644 --> 00:18:37.484
And on after that, I'll take a look
240
00:18:38.125 --> 00:18:40.524
on the code to, like, at the code. Locally,
241
00:18:40.524 --> 00:18:49.404
like, I will test that it works. It does the job for me, and all my tests are passing and green, but then I will just scan the code and see
242
00:18:48.770 --> 00:18:59.809
what's happening. So there's also like the the the journey is how how big the feature should be. I don't like big features to be implemented at once. I still want to know, you know, what what's happening with my code.
243
00:19:01.169 --> 00:19:03.090
So yeah. It's
244
00:19:02.684 --> 00:19:06.124
it's shifting. I don't know, like, six months from now, it would
245
00:19:06.365 --> 00:19:08.124
could be completely different.
246
00:19:09.005 --> 00:19:12.604
Still not not used to I don't like this idea
247
00:19:12.924 --> 00:19:19.139
of, you know, like remote sessions where you check on your phone the the status and your code there. I think I think we
248
00:19:19.300 --> 00:19:22.179
need a little bit of time for ourselves away from computers.
249
00:19:23.220 --> 00:19:29.379
But I do have agents running in the background on my repos and doing something, and then I come back and check on the results.
250
00:19:29.700 --> 00:19:30.020
So
251
00:19:30.605 --> 00:19:37.325
I'm kinda partially there, but not yet, and maybe I don't want yet completely to be there. So so
252
00:19:37.885 --> 00:19:44.605
you can see it's it's definitely a journey. Yeah. It's it's fun. It's so interesting to observe, like, being in the first, you know, row
253
00:19:45.480 --> 00:19:47.159
front seat, see
254
00:19:47.400 --> 00:19:51.640
where we go, like, the internal experiments that we run. Some of them are
255
00:19:52.200 --> 00:20:02.615
they're not going anywhere, like, we don't we realize that we don't wanna it's not it's not a product. And sometimes it's like, yes, yes, yes, yes. Ship it, please. And we have to wait. It's a bit annoying.
256
00:20:04.055 --> 00:20:07.414
Yeah. And and that's good. I see that you do
257
00:20:07.655 --> 00:20:15.260
sometimes deprecate things, which is good. I saw, for instance, GitHub models, which was a preview feature in GitHub. That's now gone
258
00:20:15.260 --> 00:20:19.900
away or going away. And that's good. You need to say goodbye to things that aren't working.
259
00:20:20.460 --> 00:20:22.700
And to just try things out, see if they are working.
260
00:20:24.345 --> 00:20:27.144
So, could the next phase perhaps be
261
00:20:27.385 --> 00:20:30.745
human out of the loop? You just feed the thing requirements
262
00:20:30.745 --> 00:20:31.945
and you test it?
263
00:20:33.544 --> 00:20:36.025
Hope. We'll see. At
264
00:20:36.025 --> 00:20:41.990
this point, I gave up on predicting future. So
many, everything is moving fast. So
265
00:20:42.390 --> 00:20:44.390
just just about the deprecations,
266
00:20:44.470 --> 00:20:49.909
I would say, yeah, we we try to deprecate things that are not really
267
00:20:50.230 --> 00:20:58.765
going anywhere or doesn't look like a product to us. Models was an interesting beast. I actually like the product, but it was mainly
268
00:20:59.485 --> 00:21:01.645
like a gateway to to show
269
00:21:02.205 --> 00:21:06.605
the world what what those models are capable of, how to test, how to eval them,
270
00:21:07.510 --> 00:21:09.750
how to interact, how to compare.
271
00:21:10.150 --> 00:21:10.710
But
272
00:21:12.070 --> 00:21:16.789
we're we have it was a front, like a proxy to
273
00:21:17.190 --> 00:21:26.544
Asia services anyway. So like if you want to run it in production, yeah, then you need to switch to something like Foundry and then get your models deployed there. And there is a lot of tooling.
274
00:21:26.705 --> 00:21:33.104
So that's also about like, should we build the same tools that exist in another product that all is owned by the same company?
275
00:21:33.424 --> 00:21:34.304
But
276
00:21:34.304 --> 00:21:36.679
I think it was a good gateway
277
00:21:36.840 --> 00:21:38.280
for many
278
00:21:38.840 --> 00:21:41.080
well, I don't know numbers unfortunately,
279
00:21:41.080 --> 00:21:43.240
but some some customers,
280
00:21:43.560 --> 00:21:47.480
some engineers I talked to, yeah, they used it. I did. All my AI
281
00:21:47.480 --> 00:21:48.440
experiments,
282
00:21:48.440 --> 00:21:53.755
I tend to spin up something on GitHub models and just test it there and then play with them.
283
00:21:54.555 --> 00:22:04.315
Actually, thank you for reminding me. I need to update one service that still relies on on models. I I have it Okay. Like, local bot for my like
284
00:22:04.610 --> 00:22:09.330
personal automation that I have a Telegram bot that kinda helps me with certain things. And
285
00:22:09.810 --> 00:22:16.930
it's updated. Oh, yeah. Good. I'll I'll make a note. Yeah. And tell your agents. Yeah. Yeah. And about the future,
286
00:22:17.330 --> 00:22:17.970
I don't
287
00:22:18.295 --> 00:22:20.135
know what I see,
288
00:22:20.535 --> 00:22:42.890
especially like internally Microsoft and GitHub teams that are very much invested. So we have to talk food, We have to make sure that we we test it all and and see the limit. I think some people say the best perk that we have is the the animated tokens. Right? So we can just spend as much as you as you want and learn because we have to be confident like what we're talking about. We have to try.
289
00:22:44.305 --> 00:22:48.065
So we do have teams that are very much in like
290
00:22:48.225 --> 00:22:53.825
into it. So they build this almost what sometimes called like a yeah. Software factories or
291
00:22:54.305 --> 00:22:57.505
they they they trying to see that what can we offload.
292
00:22:58.310 --> 00:22:59.990
And I think the
biggest,
293
00:23:00.470 --> 00:23:01.269
at least
294
00:23:01.510 --> 00:23:05.590
as of today, the biggest understanding is that you
295
00:23:05.750 --> 00:23:08.070
technically can build almost everything.
296
00:23:08.550 --> 00:23:14.975
And the neck the challenge is what what exactly you're going to build. Because when you can do almost anything
297
00:23:15.295 --> 00:23:19.374
and with a very short time span, because the agent can run overnight
298
00:23:19.775 --> 00:23:24.015
and just get you something when you wake up, get it ready for you. So
299
00:23:25.190 --> 00:23:28.470
do you then then we always have to ask ourselves,
300
00:23:28.630 --> 00:23:35.749
you really need this feature? Do you is it something that you're going to keep in your app, in your in your in your
301
00:23:36.630 --> 00:23:40.309
your code base? Like, it's cheap now, so you can throw it away easily.
302
00:23:41.275 --> 00:23:45.115
But still, like, is it is it worth being implemented?
303
00:23:45.355 --> 00:23:45.995
And
304
00:23:46.475 --> 00:23:49.515
we see that some engineering teams, they became
305
00:23:49.835 --> 00:23:50.715
more like
306
00:23:51.195 --> 00:23:52.635
agent experienced
307
00:23:52.890 --> 00:23:58.409
engine platform engineers. So they build everything that the agent needs to be successful,
308
00:23:58.890 --> 00:24:00.650
which is quite fun and interesting.
309
00:24:01.050 --> 00:24:11.004
But it shifts, like, they they still ship a lot of features. Like, look at our release page. Look, change look. Like, what is like? Yeah. 10 releases per day.
310
00:24:11.485 --> 00:24:14.524
Look at the tooling, like CLI or
311
00:24:16.285 --> 00:24:21.645
GitHub Copilot app or Versus Code, they're releasing daily to two times a day, sometimes three times a day.
312
00:24:22.530 --> 00:24:34.210
And they it's not just one bug fix, it's you read the change log, it's like five, seven entries in there. Yeah. It's crazy. Yeah. Yeah. But the the only way they can do it at this space is
313
00:24:34.934 --> 00:24:37.495
by building a lot of infrastructure
314
00:24:37.495 --> 00:24:38.214
around
315
00:24:38.534 --> 00:24:48.454
around it. So that when they have an idea of the feature, they can almost delegate that idea to the agent to a certain degree. You need to plan, of course, you need to provide information,
316
00:24:48.455 --> 00:24:53.739
you need to provide maybe like a basic idea like what it should and where it needs to take,
317
00:24:55.179 --> 00:25:11.234
you know, like what kind of code it needs to modify, what data is going to be needed for that. Definitely, there is a lot of guardrails in there. But still, once you get that foundation, you can see that the speed increases too much. Sometimes too much. Yeah. So it increases, but sometimes it's just difficult
318
00:25:11.794 --> 00:25:16.140
to keep up. So I think the typical example is doc docs. Click.
319
00:25:16.300 --> 00:25:24.380
Yes. So we still need to release docs. We still need to keep them up to date. Partially, it's possible to automate, but nobody wants to read
320
00:25:24.780 --> 00:25:26.220
AI generated
321
00:25:26.220 --> 00:25:27.660
documentation pages.
322
00:25:27.740 --> 00:25:34.014
Yeah. You want to have something that is written by a human with a lot of experience in in in technical
323
00:25:34.015 --> 00:25:34.815
writing,
324
00:25:35.294 --> 00:25:41.855
and that's where, like, you put a lot of pressure on something that is happening after the development. So your infra,
325
00:25:41.855 --> 00:25:42.414
your
326
00:25:42.975 --> 00:25:50.370
users actually getting lost sometimes, so I talk to engineers and they say, like, yeah, it's too many features. I don't understand. Like, how do I keep up? It's hard.
327
00:25:50.770 --> 00:25:51.330
So
328
00:25:51.730 --> 00:25:56.529
yep. It's a it's human is not going to be taken away, I I I think.
329
00:25:56.770 --> 00:25:58.850
But humans are shifting
330
00:25:59.455 --> 00:26:09.215
from a bit of like a manual labor when you just sit and type a lot to more abstract or more high level and even like product mindset is needed.
331
00:26:09.535 --> 00:26:10.175
So
332
00:26:10.335 --> 00:26:19.450
it's it's it is going to be interesting. So for what it will look like in the future, I can't predict, but what I see, I like so far.
333
00:26:20.250 --> 00:26:22.889
Yeah. Yeah. It's a very interesting times.
334
00:26:23.690 --> 00:26:33.695
You touched on this a couple times already, which is the value we can get out of these AI tools. Like we're talking about personal productivity here. We get the copilot.
335
00:26:33.935 --> 00:26:34.655
Yeah. What
336
00:26:35.135 --> 00:26:36.174
I hear from
337
00:26:36.415 --> 00:26:40.495
companies and customers a lot, because this stuff is not cheap, it's not for free. Like,
338
00:26:41.460 --> 00:26:42.659
is this actually
339
00:26:44.419 --> 00:26:48.259
creating value for us and how do we measure that especially?
340
00:26:48.340 --> 00:26:58.544
Like I have a customer that's doing a POC with GitHub Copilot and then they literally have a guy do a task without GitHub Copilot, a development task, write down how long it takes,
341
00:26:58.945 --> 00:27:01.024
and then one with GitHub Copilot,
342
00:27:01.025 --> 00:27:11.620
write down how long it takes. And that's the metric that they use there. Is that the same person building it twice? Yeah. Yeah. That's not a fair experiment. Right? So it's not clean.
343
00:27:12.660 --> 00:27:13.299
Well,
344
00:27:13.940 --> 00:27:15.460
you know, if
345
00:27:17.140 --> 00:27:18.260
if anyone
346
00:27:18.980 --> 00:27:20.500
builds like a proper
347
00:27:21.345 --> 00:27:23.345
way to measure developer productivity
348
00:27:23.345 --> 00:27:24.225
that that
349
00:27:24.785 --> 00:27:26.465
it it it you
350
00:27:26.625 --> 00:27:28.065
know, it's a good business, definitely.
351
00:27:28.945 --> 00:27:34.864
And so far, I didn't see like great
examples. Like, yeah, there are plenty of like DoraMetrics,
352
00:27:35.665 --> 00:27:37.010
SAFE frameworks,
353
00:27:37.010 --> 00:27:42.769
like a lot. But still, there is something that, well, first of all, humans don't like to be measured.
354
00:27:43.169 --> 00:27:47.409
And if you start measuring them, they will play the game and will
355
00:27:47.730 --> 00:27:50.895
optimize for that metric. So if you ask
356
00:27:50.895 --> 00:27:58.494
them, you know, how many pull requests you create and and you start measuring and ranking them, oh, there will be a lot of pull requests, very small ones.
357
00:27:59.615 --> 00:28:07.109
Or, you know, so it's it it it's a difficult game. Yeah. Absolutely. In the past, like in the premium request model,
358
00:28:07.270 --> 00:28:11.669
AI was basically free. Like, it was super cheap. Yeah. Now
359
00:28:12.070 --> 00:28:14.870
but that was very good for
360
00:28:14.235 --> 00:28:19.835
for many engineers to actually learn and see the like how far you can go, what you can do.
361
00:28:20.955 --> 00:28:24.315
With us moving to usage based billing with
362
00:28:24.875 --> 00:28:29.434
more other vendors doing the same, you can see this is where the industry is going. So
363
00:28:30.030 --> 00:28:34.430
there there is time to actually be mindful about what you built.
364
00:28:34.750 --> 00:28:40.510
An example, like, yes, you can run AI agent to give you, I don't know,
365
00:28:40.910 --> 00:28:43.575
a summary of to summarize.
366
00:28:44.055 --> 00:28:48.855
I don't know. For example, like, in my job, I have to read a lot of change logs, like what are we shipping?
367
00:28:49.575 --> 00:28:53.495
You can run an agent and and scrape the file, the the change log
368
00:28:53.735 --> 00:28:59.010
website and get your summaries, but you can write a script that would just give you like, you know,
369
00:29:00.130 --> 00:29:06.130
a text to like the the header and maybe like a few lines, it's gonna be the same. Like, is it interesting or not? Do I need to read it?
370
00:29:06.770 --> 00:29:07.330
So
371
00:29:07.810 --> 00:29:11.924
for sure, we don't have to replace every thing that we do
372
00:29:12.245 --> 00:29:12.884
with
373
00:29:13.045 --> 00:29:16.004
with AI tools, even though it's possible and fun.
374
00:29:16.325 --> 00:29:22.165
We don't have to forget, you know, we still have computers that are very fast and efficient and extra cheap.
375
00:29:22.660 --> 00:29:24.659
You can write an algorithm sometimes.
376
00:29:25.300 --> 00:29:29.220
But AI can make decisions, can drive more complex flows.
377
00:29:29.300 --> 00:29:35.220
Maybe something that you have inside your code base and you do certain things, maybe write a script and
378
00:29:35.655 --> 00:29:42.535
point your agent to the script and just do it. It's faster and more reproducible and reliable. So for sure,
379
00:29:44.055 --> 00:29:45.815
now when you have to pay
380
00:29:46.455 --> 00:29:52.330
for for for the actual consumption, You need to be like, we we have that. We we have conversations
381
00:29:52.330 --> 00:29:55.209
coming back again, like how much exactly are we saving?
382
00:29:55.529 --> 00:29:56.089
And
383
00:29:56.409 --> 00:30:02.730
it's always difficult because you cannot say that with this tool, you are going to save 25%.
384
00:30:03.705 --> 00:30:08.264
Because 25% of what? What are you measuring? What's important to you? For
385
00:30:09.785 --> 00:30:11.384
my big,
386
00:30:12.345 --> 00:30:13.865
very large manufacturing
387
00:30:13.865 --> 00:30:14.585
customer,
388
00:30:15.304 --> 00:30:16.744
for them, the
389
00:30:17.065 --> 00:30:22.500
frequency of releases is not something they're interested in because they need to build a machine,
390
00:30:22.900 --> 00:30:26.659
put the software on it, and ship it to the customer, to their customer.
391
00:30:26.900 --> 00:30:27.860
But
392
00:30:28.660 --> 00:30:34.475
you do that once a year, and then you need to make sure that that software works. So for them, it's a quality,
393
00:30:34.715 --> 00:30:39.514
of tests, amount of lab resource that they, you know, like like what they do.
394
00:30:40.395 --> 00:30:44.234
But for some sort of retail customer or
395
00:30:44.395 --> 00:30:45.434
that that build
396
00:30:45.889 --> 00:30:47.329
the ship software
397
00:30:47.330 --> 00:30:51.249
many times a day and updates and fixes for them, speed to market,
398
00:30:51.409 --> 00:30:53.009
time to market, the speed,
399
00:30:53.250 --> 00:30:56.769
the frequency of releases, the ability to react on
400
00:30:57.090 --> 00:30:58.289
the world that is changing,
401
00:30:59.575 --> 00:31:08.695
that is important. So even in in that case, like, you compare those two and for them, the the ROI is measured in different units.
402
00:31:09.815 --> 00:31:16.429
So it it is always a challenge. Like, I always start with what exactly do are you trying to improve in terms of
403
00:31:18.190 --> 00:31:20.509
yeah. You don't need more code. You need
404
00:31:20.830 --> 00:31:22.990
well, as a business, need to earn more money.
405
00:31:23.390 --> 00:31:31.295
Need to provide value to your customers. You need to do something. And what's what's in it for you? It's probably not the time the developer have saved.
406
00:31:31.455 --> 00:31:34.015
It's the value they bring back.
407
00:31:34.735 --> 00:31:38.975
Maybe they build more features. Maybe they actually your biggest problem is your
408
00:31:39.649 --> 00:31:41.570
stability or your
409
00:31:42.210 --> 00:31:44.609
older old bugs that never been addressed.
410
00:31:44.929 --> 00:31:50.690
Maybe you have, like some banks, have outdated software written in 2090.
411
00:31:52.450 --> 00:32:00.315
Yeah. You know, like in eighties or something, in seventies. So they need to update it because nobody understands it and you can just go into it
412
00:32:01.115 --> 00:32:07.034
and then make it yeah. It's not gonna bring you new features,
but it will fix your future,
413
00:32:07.755 --> 00:32:09.195
mitigate the risks that you have.
414
00:32:10.250 --> 00:32:11.530
So it's
415
00:32:11.690 --> 00:32:13.930
it's a challenge. It's
416
00:32:13.930 --> 00:32:18.570
a challenge how to measure. And mostly the challenge is not that we
417
00:32:19.290 --> 00:32:23.050
like the tool is not providing the value, it's about how you use it.
418
00:32:24.524 --> 00:32:27.484
Like I was talking to one engineer
419
00:32:27.565 --> 00:32:37.320
who's been using like AI productivity agent locally, so it wasn't a copilot to generate code, but it was to check their email, to give them, you know. And
420
00:32:37.480 --> 00:32:38.279
there was
421
00:32:38.600 --> 00:32:42.600
shocking number, like something like $1,800
422
00:32:42.679 --> 00:32:45.639
per week spent on tokens, which is insane.
423
00:32:46.360 --> 00:32:48.039
A bit of work optimization,
424
00:32:48.039 --> 00:32:50.760
it went down dramatically like a couple hundreds per month.
425
00:32:51.525 --> 00:32:53.125
And then same value.
426
00:32:54.165 --> 00:32:56.245
It's it's it's not
427
00:32:56.725 --> 00:33:00.005
the same tool, same value, different cost. Right? So then
428
00:33:00.405 --> 00:33:01.605
your productivity
429
00:33:02.005 --> 00:33:05.125
didn't change after the optimization, but your your your
430
00:33:05.590 --> 00:33:10.869
productivity ROI per per token spent, per dollar spent has increased a lot.
431
00:33:11.269 --> 00:33:13.109
So it's hard. Yeah.
432
00:33:13.350 --> 00:33:18.195
But it it it there are many dimensions here in how how you measure it. Obviously,
433
00:33:18.195 --> 00:33:21.874
it works. Like, there is no doubt. Like, everybody understand that you can do.
434
00:33:22.434 --> 00:33:26.835
But now, back to my previous observation is that you need to think about,
435
00:33:27.395 --> 00:33:30.434
do you need to do that? Does that bring you anywhere?
436
00:33:31.070 --> 00:33:35.149
And it's not about just features, it's also about the value of that. Is it
437
00:33:35.790 --> 00:33:37.309
yeah. You can build
438
00:33:37.710 --> 00:33:41.390
a dashboard for something. Do you need it? You can do,
439
00:33:43.705 --> 00:33:48.744
you know, a fun thing in your application, but does it bring you anything? You can
440
00:33:49.465 --> 00:33:53.224
you you need to be every engineer has to have this
441
00:33:53.625 --> 00:33:55.544
almost like product or entrepreneurial
442
00:33:56.690 --> 00:33:58.850
mindset. Like, what are we doing here?
443
00:33:59.330 --> 00:34:03.570
What what because you have you you can do a lot now. So that's that's that's a challenge.
444
00:34:04.050 --> 00:34:08.530
Yeah. It's a big shift when now you can create anything that you can actually think of
445
00:34:08.930 --> 00:34:12.775
right then and there. And so what? What are you going to do with that power
446
00:34:12.775 --> 00:34:13.575
basically?
447
00:34:13.975 --> 00:34:16.775
Yeah, yeah. So you just touched on
448
00:34:17.495 --> 00:34:18.295
ROI,
449
00:34:18.695 --> 00:34:19.735
so costs.
450
00:34:20.055 --> 00:34:28.780
We we know now GitHub Copilot also does uses based billing like basically all the other vendors as well, right? So we we charge per token here.
451
00:34:29.020 --> 00:34:31.500
So how can people optimize
452
00:34:31.500 --> 00:34:35.820
what comes out and how much they pay? Because there's lots of little tips and tricks.
453
00:34:36.220 --> 00:34:37.260
Oh, absolutely.
454
00:34:37.260 --> 00:34:37.500
Well,
455
00:34:38.515 --> 00:34:40.915
I actually have like even a full
456
00:34:40.995 --> 00:34:46.675
training on that that we give to to all the customers. It's public.
457
00:34:46.995 --> 00:34:48.275
But, obviously,
458
00:34:48.675 --> 00:34:52.275
I think the example is that productivity tool is the right example. So
459
00:34:53.090 --> 00:34:57.650
you can do the achieve the same results with with for a different cost.
460
00:34:57.970 --> 00:34:58.930
And that's where
461
00:34:59.730 --> 00:35:02.210
I think engineers who've been using tools
462
00:35:02.690 --> 00:35:08.425
for a long time, they are in advantage because they have that understanding on how to do
463
00:35:08.905 --> 00:35:10.185
more with
464
00:35:10.345 --> 00:35:11.865
less capable tools
465
00:35:11.945 --> 00:35:17.545
because they had those physical restrictions. Like, you can only put as much into your context window.
466
00:35:17.705 --> 00:35:18.825
You can only
467
00:35:19.065 --> 00:35:20.984
you need to be clear and
468
00:35:22.240 --> 00:35:24.400
in in in your prompts. You need to be
469
00:35:24.880 --> 00:35:28.960
so you need to have that two skill, like one skill is prompting.
470
00:35:29.040 --> 00:35:33.840
It's not as relevant anymore as it used to be, but still a good good skill to have.
471
00:35:34.320 --> 00:35:38.775
Context engineering is like, what are we going to put into the context? That is super
472
00:35:38.935 --> 00:35:39.655
critical
473
00:35:39.895 --> 00:35:44.455
because you can get your agent distracted, you can get your agent
474
00:35:45.494 --> 00:35:46.295
lost,
475
00:35:46.295 --> 00:35:49.015
and then it will recover. So now they will recover,
476
00:35:49.335 --> 00:35:51.975
but there will there would be cycles wasted.
477
00:35:52.375 --> 00:36:01.790
And I think the other part is make sure that you provide the good tools for your agents, like replace something with a script, so I'm gonna make an API call.
478
00:36:02.270 --> 00:36:04.430
Store data somewhere, like if it's
479
00:36:05.150 --> 00:36:08.430
if you're not talking about just coding, like, store it and and review
480
00:36:08.915 --> 00:36:12.355
everything that is happening. So I think that's also important.
481
00:36:12.675 --> 00:36:13.395
And then
482
00:36:14.835 --> 00:36:25.910
that's kind of like a mindset. So you need to understand how the tool works and what's behind it. It's it's a so it's super
complicated, but it's very interesting. What's what's behind? What what's under the hood?
483
00:36:26.310 --> 00:36:33.510
And after that, yeah, you have like simple solutions that what the model you're going to use. Do you really need
484
00:36:33.750 --> 00:36:36.150
that model that was released yesterday
485
00:36:36.755 --> 00:36:42.195
to get to provide you a git commit summary. Right? You know? Commit message.
486
00:36:42.595 --> 00:36:43.395
Maybe not.
487
00:36:44.275 --> 00:36:46.035
Do you really need to
488
00:36:47.235 --> 00:36:48.915
use the cheapest model
489
00:36:48.995 --> 00:36:49.715
ever
490
00:36:49.900 --> 00:36:53.180
to make something complex and and spend 25
491
00:36:53.180 --> 00:37:08.905
cycles on on on achieving the the result while you can use an expensive model and get it in one shot. Maybe like one, two, three cycles instead. So, you know, it's it's it's a balance, a fine balance, and it's experience a lot. It's very difficult to give recommendations because
492
00:37:09.305 --> 00:37:11.945
three, five weeks from now, we'll get new models again.
493
00:37:12.905 --> 00:37:18.585
But but still, it's it's important to to to know, to understand, and then to get this,
494
00:37:19.599 --> 00:37:20.240
like,
495
00:37:20.800 --> 00:37:23.200
what are you giving to the model?
496
00:37:23.920 --> 00:37:25.520
An amount, like, it's
497
00:37:25.840 --> 00:37:26.800
the amount of
498
00:37:27.200 --> 00:37:28.480
the data you sent
499
00:37:28.720 --> 00:37:31.760
and you don't really need a lot. Like my biggest,
500
00:37:32.825 --> 00:37:38.105
well, it's not the biggest, but a concern that I have with models that now take like up to 1,000,000
501
00:37:38.184 --> 00:37:39.785
tokens as a context,
502
00:37:40.105 --> 00:37:42.185
which is great. But at the same time,
503
00:37:42.825 --> 00:38:05.235
it gives you the opportunity to put so much information that that is not relevant for every task. So keeping that in mind is is important. And then there are many other techniques that are being done inside harnesses, so it's maybe because it's not something that engineers, like software engineers that build code with all those agents, they need they don't need to ideally, they don't need to think about all of those
504
00:38:06.275 --> 00:38:11.075
aspects. Instead, they need to focus on the software that they're building. And we're not there yet.
505
00:38:11.474 --> 00:38:14.595
We're still in in in in the middle. It's almost like
506
00:38:14.915 --> 00:38:16.195
you remember,
507
00:38:16.595 --> 00:38:19.795
what is it, like, fifteen ish years ago when Kubernetes
508
00:38:19.770 --> 00:38:20.730
started.
509
00:38:20.810 --> 00:38:24.010
Yeah. Everybody had to learn it, like all the KubeCTL
510
00:38:24.010 --> 00:38:26.170
commands and how you build and deploy.
511
00:38:26.810 --> 00:38:30.650
After a while, most of the companies they've been using
512
00:38:30.810 --> 00:38:32.250
managed Kubernetes
513
00:38:32.105 --> 00:38:39.945
with a layer on top of it, which was built by platform teams. And now, when you build software, just spin up and you repo from a template.
514
00:38:40.585 --> 00:38:46.904
You've got, you know, put some configuration in there, and then your buy build pipeline will prepare
515
00:38:46.580 --> 00:38:48.020
images, prepare
516
00:38:48.820 --> 00:39:00.420
Helm charts, will prepare everything, and it is just going to be shipped. You don't need to know as much of Kubernetes anymore as an engineer to use and deploy there. So I think it's almost the same we'll get eventually
517
00:39:00.645 --> 00:39:01.445
where
518
00:39:01.685 --> 00:39:08.965
most of those decisions would be taken away. But for now, you need to you need to know what what exactly are you is happening.
519
00:39:09.765 --> 00:39:16.485
So that is important. And then, unfortunately, we have a lot of on the market, those tools that
520
00:39:16.580 --> 00:39:17.700
almost like
521
00:39:17.940 --> 00:39:27.140
a silver bullet kind of tools. Like, you just put the skill in there and your token consumption is going to All this markdown stuff around the chip. Yeah. But unfortunately,
522
00:39:28.020 --> 00:39:38.315
everything that you know, if it's such a simple solution, and it's very competitive market, this market is insane in terms of the competition and everybody needs their share of users.
523
00:39:38.475 --> 00:39:39.035
So
524
00:39:40.235 --> 00:39:42.555
ask yourself why why wasn't it
525
00:39:43.515 --> 00:39:44.715
embedded like
526
00:39:46.410 --> 00:39:51.450
they they would have embedded it already into the harness if it was something that really
527
00:39:51.609 --> 00:39:54.570
makes the difference in a positive way. So
528
00:39:55.369 --> 00:39:56.730
but, yeah, everybody wants
529
00:39:58.010 --> 00:40:01.290
it's obvious that the desire here from the engineers is to
530
00:40:02.415 --> 00:40:12.735
get rid of that problem. Like, make it solved for me. I need to focus on something else. Still, there is a lot. There are a lot of engineers that want to be almost like
531
00:40:12.895 --> 00:40:16.095
knee deep into those details and understand everything.
532
00:40:16.619 --> 00:40:19.579
And they find a lot of joy like I do. But
533
00:40:20.940 --> 00:40:22.059
it's unfortunately,
534
00:40:22.059 --> 00:40:23.099
or fortunately,
535
00:40:23.099 --> 00:40:26.859
not for everybody. Like, somebody wants to build cool
536
00:40:27.099 --> 00:40:33.665
mobile apps and to make sure that they're beautiful and users are happy and they don't want to spend time setting up,
537
00:40:34.145 --> 00:40:42.865
you know Yeah. The tools. Yeah. The tools and then build extensions and and and then customize them. So definitely, we need to
538
00:40:43.425 --> 00:40:45.105
move into the area where
539
00:40:45.530 --> 00:40:49.770
it's a it's not a tool that you've spent a lot of time configuring,
540
00:40:49.770 --> 00:40:51.290
it's a tool
that just works,
541
00:40:52.490 --> 00:40:56.970
and then you provide just anything that is needed for to achieve your task.
542
00:40:58.194 --> 00:41:05.875
But for now, we need we we we we need to have that understanding, and we need to come up with best practices, and we need to train each other.
543
00:41:06.355 --> 00:41:06.674
So
544
00:41:07.714 --> 00:41:20.500
Yeah. But I I think we're already pretty far along as in Oh, absolutely. We do have it. Yeah. We have a text box. You just type something in there, and it does what you want. Sometimes it's, you know, maybe too expensive.
545
00:41:22.020 --> 00:41:26.099
But, yeah, it's also thinking yourself about, is this an automation?
546
00:41:26.495 --> 00:41:33.615
Or is this something the LLM should actually do? And is it worth it for the LLM to do this? So co co pilot in this case.
547
00:41:33.935 --> 00:41:37.455
That's, you know, we need to retrain ourselves with that as well.
548
00:41:37.935 --> 00:41:40.175
And we're all learning this, right, altogether,
549
00:41:40.175 --> 00:41:41.695
because it's still very, very new.
550
00:41:42.360 --> 00:41:43.080
Yep.
551
00:41:43.400 --> 00:41:50.120
That's my favorite question is, like, can you give us the best practices on how to implement it at scale? It's like, well,
552
00:41:51.000 --> 00:41:51.960
I have bad There news for
553
00:41:53.080 --> 00:41:59.105
is no such thing yet. We'll get there. Eventually, we'll learn what works, what doesn't.
554
00:41:59.345 --> 00:42:01.345
But embracing that
555
00:42:01.665 --> 00:42:06.305
means that you as a as an organization like engineering leader needs to
556
00:42:07.310 --> 00:42:10.350
be also creative and maybe even take a bit of a risk
557
00:42:10.590 --> 00:42:16.270
and be ready to back up. You know, step back and like, okay, okay, that doesn't work. We need to solve it in a different way.
558
00:42:16.670 --> 00:42:18.510
I Nobody is alone.
559
00:42:18.830 --> 00:42:21.435
We we all have a lot of brilliant
560
00:42:21.675 --> 00:42:22.395
minds
561
00:42:22.555 --> 00:42:25.115
trying to crack that nut, trying to understand
562
00:42:25.275 --> 00:42:26.075
how,
563
00:42:27.035 --> 00:42:31.195
what it should look like. Sometimes I said there are some visionaries
564
00:42:31.355 --> 00:42:38.890
that are too far ahead and their ideas are not taken yet. Then four or five months later, we think, oh, actually, was right.
565
00:42:39.290 --> 00:42:51.505
That's how we need to do it. So I don't know. We'll get there. We'll get there for sure. If there is a problem, we as a human kind, can if we put a lot of effort, enough effort, we solve it. And it is obviously
566
00:42:51.665 --> 00:42:55.985
something that we want to solve, so it will be solved. I have a lot of faith in it.
567
00:42:56.465 --> 00:42:57.185
Okay.
568
00:42:57.345 --> 00:42:57.905
Cool.
569
00:42:59.265 --> 00:43:07.300
So we've been talking for a while already. Maybe we can kinda end on this. So let's say,
if there are any engineers out there that
570
00:43:07.540 --> 00:43:13.140
are still on the fence with this stuff, maybe they think, you know what, this is not something that I need to use.
571
00:43:14.100 --> 00:43:16.740
What would you tell them? How could they get started?
572
00:43:17.765 --> 00:43:18.885
Well,
573
00:43:18.885 --> 00:43:19.845
first of all,
574
00:43:20.325 --> 00:43:25.685
it's a difficult task right now to find such an engineer who never ever ever used anything.
575
00:43:28.085 --> 00:43:29.365
I don't like
576
00:43:30.405 --> 00:43:32.085
putting that as a, you know, like,
577
00:43:32.829 --> 00:43:34.030
triggering
578
00:43:34.589 --> 00:43:35.710
FOMO, like
579
00:43:36.109 --> 00:43:36.910
Mhmm. Triggering
580
00:43:37.390 --> 00:43:39.710
any kind of anxiety about it.
581
00:43:40.030 --> 00:43:42.110
I do believe that
582
00:43:42.510 --> 00:43:43.950
it's inevitable.
583
00:43:44.109 --> 00:43:51.985
Like, you will be using those tools in one shape or another, maybe not something that is advanced and frontier because
584
00:43:52.625 --> 00:43:56.065
some compliance or specific type of software you're building,
585
00:43:57.800 --> 00:44:00.680
then you will have something less capable,
586
00:44:00.680 --> 00:44:05.720
but it's still going to be there, because we see it works, and it does a good job.
587
00:44:06.680 --> 00:44:07.400
So
588
00:44:08.040 --> 00:44:11.285
you would have to learn it at at one moment,
589
00:44:12.644 --> 00:44:14.325
you know, one way or another.
590
00:44:16.085 --> 00:44:17.205
Yeah. It's
591
00:44:17.444 --> 00:44:25.045
it's it's a difficult advice. It's like, how do you start? Just start using because so many surfaces where you can get access to it. I understand
592
00:44:25.230 --> 00:44:29.870
the the the biggest restriction is that some engineers, they have no access to those tools
593
00:44:31.070 --> 00:44:34.350
because of many reasons that, you know, I
594
00:44:35.470 --> 00:44:41.025
don't have a good advice for them, like, rather than maybe in your personal time,
595
00:44:41.425 --> 00:44:47.745
you know, get a ChatGPT subscription and just play with it. Just see what it can give you back.
596
00:44:48.385 --> 00:44:54.305
Because it once you get, like, a basic understanding, it doesn't matter what tool it is. You know the limits,
597
00:44:54.940 --> 00:44:55.740
you know
598
00:44:56.220 --> 00:45:01.020
how to drive it, you know when to trust and when not to trust and and verify.
599
00:45:01.340 --> 00:45:02.940
That's already a bigger
600
00:45:04.060 --> 00:45:05.980
big help. I hope education
601
00:45:06.060 --> 00:45:11.265
is catching up. I haven't been talking to like many universities in the the nearest
602
00:45:12.465 --> 00:45:24.760
like recently, but I hope that education is going to put a lot of effort into it and we, like schools and universities, are going to teach more of those skills. It's a bit of a challenge that
603
00:45:25.559 --> 00:45:27.880
a teacher needs to be good enough
604
00:45:28.599 --> 00:45:30.360
have a good enough understanding
605
00:45:32.280 --> 00:45:33.880
in order to teach.
606
00:45:34.440 --> 00:45:35.559
So it's
607
00:45:35.724 --> 00:45:38.685
yeah. But it's almost like, you know, using a computer.
608
00:45:39.005 --> 00:45:41.005
At some point, it was became
609
00:45:41.165 --> 00:45:53.580
a must think that you need to be able to know how to use it. Although a lot of people are using just their mobile phones right now and they're happy, but it's still a skill that you need to learn. So those things are going to be more
610
00:45:55.339 --> 00:45:59.340
or they're gonna be everywhere, and I I I believe they're going to be
611
00:45:59.900 --> 00:46:02.940
less demanding on the skill of the user,
612
00:46:03.260 --> 00:46:06.025
but that's a general, like, population.
613
00:46:06.345 --> 00:46:11.705
But for engineers, like software engineers, you need to know your tools, you need to know the technology,
614
00:46:11.705 --> 00:46:12.985
you need to understand
615
00:46:13.305 --> 00:46:16.025
the limits, you need to be able to
616
00:46:16.510 --> 00:46:20.030
drive it to to to squeeze the best outcome,
617
00:46:20.190 --> 00:46:26.110
results out of it. So, yeah, it's it's definitely something that we all need to to be learning.
618
00:46:28.475 --> 00:46:33.035
I don't know. I don't want to recommend people switching jobs just to get access to those But
619
00:46:33.835 --> 00:46:35.035
it's definitely
620
00:46:35.275 --> 00:46:39.435
something that we will probably see in the, you know, in the
621
00:46:39.755 --> 00:46:48.220
on the market where companies are going to advertise that you are going to get access to those tools. You're going to have a large budget on tokens. You're going to have
622
00:46:49.020 --> 00:46:49.820
tasks
623
00:46:50.619 --> 00:46:52.060
that are related to
624
00:46:52.540 --> 00:46:53.820
AI technologies.
625
00:46:54.220 --> 00:46:59.340
We'll probably see that already in there. And and it's been like that in the past, like if you have some advantage
626
00:46:59.340 --> 00:46:59.820
because
627
00:47:00.325 --> 00:47:02.325
engineers joining companies,
628
00:47:02.325 --> 00:47:04.085
they do that for two reasons.
629
00:47:04.165 --> 00:47:06.565
Yep. They want to do something, but they also
630
00:47:06.885 --> 00:47:13.685
have to think ahead like, what's next? Like, after that job, what am I going to be in terms of like my employability,
631
00:47:13.685 --> 00:47:19.980
my how attractive I am going I'm I'm going to be on the markets in three after three years at this company.
632
00:47:20.700 --> 00:47:23.740
And I think the direction is clear right now. So
633
00:47:23.980 --> 00:47:25.740
you need to choose wisely.
634
00:47:26.540 --> 00:47:27.900
That's good advice.
635
00:47:28.540 --> 00:47:29.579
Okay. Cool.
636
00:47:30.684 --> 00:47:35.725
Where can people find you? You have an awesome blog, for instance. Other places as well?
637
00:47:37.645 --> 00:47:41.565
At this point, yeah. My blog and my LinkedIn profile are the two things.
638
00:47:43.290 --> 00:47:47.930
I do have a X or Twitter account, but it's a bit abandoned now.
639
00:47:48.090 --> 00:47:49.930
Mhmm. That's that's another
640
00:47:50.010 --> 00:47:52.890
result of all these new technologies.
641
00:47:52.970 --> 00:47:59.365
A lot of stuff, you you do it for work. You need to learn a lot, and then it takes time. So,
642
00:47:59.765 --> 00:48:02.645
yeah, probably my blog is going to be the best
643
00:48:03.605 --> 00:48:05.925
way to to to stay up to date.
644
00:48:07.125 --> 00:48:12.040
I I don't post very often, but I when I do, I try to do some good stuff in there.
645
00:48:12.440 --> 00:48:14.040
So it's good to be blog
646
00:48:14.360 --> 00:48:16.920
at cloudeng.nl.
647
00:48:17.480 --> 00:48:21.880
It's yeah. At some point, like, I was very interested into all these cloud technologies.
648
00:48:22.285 --> 00:48:23.725
Yeah. Ops
649
00:48:23.725 --> 00:48:29.645
in there, and I wanted I got the domain name, I wanted to build some sort of consultancy business around that.
650
00:48:30.605 --> 00:48:32.765
Well, look at me, didn't happen.
651
00:48:34.650 --> 00:48:36.650
Yeah. But the domain name is with me. So
652
00:48:37.530 --> 00:48:38.970
Okay. Excellent.
653
00:48:39.210 --> 00:48:39.770
So
654
00:48:40.010 --> 00:48:50.010
this will be in the show notes for listeners. You can go to the blog. Check it out. There's some good articles on there already, and then you can subscribe there for new stuff when it comes out.
655
00:48:50.785 --> 00:48:57.025
Thank you very much, Anton, for hanging out with me. Thank you, Barry. Yeah. Thank you. It was a big fun. All the good questions.