AI Hype vs. Reality. Real Stats from Laura Tacho, CTO of DX

1
00:00:00,000 --> 00:00:02,340
Laura Tacho: Why is it that
a developer can only spend

2
00:00:02,340 --> 00:00:03,900
one hour a day writing code?

3
00:00:04,199 --> 00:00:08,159
It's that they are in meetings all
the time, having interruptions.

4
00:00:08,240 --> 00:00:12,850
they are waiting on CI, they're trying
to navigate through code bases We see

5
00:00:12,850 --> 00:00:16,990
this reflected in the data and when
we look at AI time savings right now,

6
00:00:16,990 --> 00:00:22,630
which is median, like three hours,
45 minutes per developer per week,

7
00:00:23,320 --> 00:00:26,530
and then we look at how much time
they could be saving by reducing dev

8
00:00:26,620 --> 00:00:30,580
environment toil, by having better
documentation, by reducing meetings.

9
00:00:31,335 --> 00:00:36,406
AI doesn't even cover a 10th of what they
could be saving if the other parts of the

10
00:00:36,406 --> 00:00:37,996
developer experience were also increased.

11
00:00:37,996 --> 00:00:39,495
So, speaking to your kind of.

12
00:00:40,315 --> 00:00:43,435
Deep seated fear, maybe about
like the over promise of AI.

13
00:00:43,435 --> 00:00:47,575
I think one of the ways that AI has been
overpromised is that it is a silver bullet

14
00:00:47,575 --> 00:00:52,846
that is gonna solve so many problems
when in fact developer experience is

15
00:00:52,846 --> 00:00:59,746
still a way bigger lever for almost every
company than AI assisted engineering

16
00:00:59,746 --> 00:01:01,276
at least at this point in time.

17
00:01:06,189 --> 00:01:09,999
Bret: This is one of my most important
and favorite episodes this year.

18
00:01:10,119 --> 00:01:12,309
I'm so excited to let you listen to this.

19
00:01:12,309 --> 00:01:16,209
If you are currently using AI
or even thinking of using AI in

20
00:01:16,209 --> 00:01:20,049
software engineering, you'll wanna
hear from our guest on this show.

21
00:01:20,109 --> 00:01:23,649
Laura Taco, who's a friend of
the show, been on many times.

22
00:01:23,937 --> 00:01:28,107
Laura is the CTO of DX, that's the
DevEx and dev productivity company

23
00:01:28,256 --> 00:01:31,646
where they help organizations
measure development team work.

24
00:01:31,961 --> 00:01:36,911
And in this episode, we're focusing on
her team's work on the gains and losses

25
00:01:36,911 --> 00:01:39,371
of using AI in your software lifecycle.

26
00:01:39,791 --> 00:01:43,451
My co-host, Nirmal is back adding his
experience from what he's seeing on

27
00:01:43,451 --> 00:01:45,816
the topic at AWS and their customers.

28
00:01:46,721 --> 00:01:50,501
And if you haven't heard Laura before,
she's been on this show many times

29
00:01:50,501 --> 00:01:55,211
talking about containers and DevOps,
but years back she started focusing on

30
00:01:55,211 --> 00:02:00,161
engineering management and improving
productivity and measuring outcomes,

31
00:02:00,611 --> 00:02:05,141
which happens to be very important when
you start to introducing a new workflow

32
00:02:05,141 --> 00:02:08,351
to everyone's job like AI is doing.

33
00:02:08,861 --> 00:02:14,051
Her pivotal work at DX has her focusing
exclusively on how professional software

34
00:02:14,051 --> 00:02:20,051
teams are actually adopting ai, where
the snags and rough spots are, and how

35
00:02:20,051 --> 00:02:25,091
your team can properly adopt AI today
for maximum productivity benefits.

36
00:02:25,451 --> 00:02:28,811
Also, a quick reminder to smash
that like button and review

37
00:02:28,811 --> 00:02:30,146
the podcast in your player.

38
00:02:30,776 --> 00:02:35,096
If that's your chosen way to listen or
watch, because that's the way to help this

39
00:02:35,096 --> 00:02:39,746
episode cut through the hype and noise out
there on the internet around AI adoption

40
00:02:39,926 --> 00:02:44,006
so that other engineers might find Laura's
breath of fresh air on the subject.

41
00:02:44,336 --> 00:02:48,926
It turns out it's all not doom
and gloom and we all are probably

42
00:02:48,926 --> 00:02:53,966
still gonna have jobs and AI is
not as good as they claim it is.

43
00:02:54,926 --> 00:02:58,286
That's a spoiler spoiler on there,
TLDR, but thanks so much for

44
00:02:58,286 --> 00:03:00,236
watching and let's get to the show.

45
00:03:05,973 --> 00:03:08,463
Hey, welcome to the
Agentic DevOps podcast.

46
00:03:08,703 --> 00:03:13,383
I am Bret, I am here with my cohost
today, Nirmal Mehta, welcome, Nirmal.

47
00:03:13,655 --> 00:03:15,025
Nirmal Mehta: Thanks for
having me back, Bret.

48
00:03:15,085 --> 00:03:15,665
I'm Nirmal Mehta.

49
00:03:15,865 --> 00:03:19,195
I'm a principal specialist, solution
architect and containers tech lead at

50
00:03:19,195 --> 00:03:24,865
AWS and These are my opinions and not
of my employer, Amazon Web Services.

51
00:03:24,967 --> 00:03:26,133
Bret: I'm glad to have you back.

52
00:03:26,133 --> 00:03:29,073
And we're, we're
completing the trio today.

53
00:03:29,073 --> 00:03:31,143
Laura Tacho is back with us.

54
00:03:31,143 --> 00:03:35,032
She's been on so many podcast
episodes I can't keep track anymore.

55
00:03:35,203 --> 00:03:36,011
welcome Laura.

56
00:03:36,041 --> 00:03:37,181
I'm glad to have you here.

57
00:03:37,453 --> 00:03:37,893
Laura Tacho: Hi, Bret.

58
00:03:37,913 --> 00:03:38,463
Hi, Nirmal.

59
00:03:38,603 --> 00:03:42,413
It's so nice to have
the trio back reunited

60
00:03:42,621 --> 00:03:46,491
let's hope the audience can brace
themselves for what's about to come.

61
00:03:46,822 --> 00:03:47,482
Nirmal Mehta: You're so kind.

62
00:03:48,487 --> 00:03:49,612
Can't wait to get into it.

63
00:03:49,612 --> 00:03:51,982
We have such interesting
topics to dig into.

64
00:03:52,269 --> 00:03:52,532
Bret: Laura.

65
00:03:52,957 --> 00:03:54,247
you work at DX.

66
00:03:54,247 --> 00:03:55,497
Tell us about DX and what you do there.

67
00:03:56,476 --> 00:04:00,436
Laura Tacho: So I am CTO at DX DXs,
a developer intelligence platform.

68
00:04:00,436 --> 00:04:05,776
That means we are getting data insights
qualitative self-reported from developers

69
00:04:05,776 --> 00:04:08,176
about developer experience about velocity.

70
00:04:08,551 --> 00:04:12,241
Productivity metrics and helping
organizations use that data in

71
00:04:12,241 --> 00:04:17,441
order to improve, in order to make
and validate their AI strategy, in

72
00:04:17,441 --> 00:04:21,431
order to make developer experience
better and help engineers get more

73
00:04:21,431 --> 00:04:24,611
work done with less friction and
less drag across their systems.

74
00:04:24,933 --> 00:04:26,158
a very exciting role.

75
00:04:26,158 --> 00:04:28,858
I get to look into so many
different organizations.

76
00:04:28,858 --> 00:04:32,641
Netflix, ADP, Vanguard,
Pfizer, you name it.

77
00:04:32,671 --> 00:04:36,451
I've had a chance to work with some
really exceptional leaders who believe

78
00:04:36,511 --> 00:04:41,221
that developer experience is an
incredibly important driver, and that

79
00:04:41,221 --> 00:04:43,351
is such a nice environment to work in.

80
00:04:43,896 --> 00:04:48,946
Bret: So Nirmal and I have got to watch
your rise of fame recently around AI

81
00:04:48,946 --> 00:04:55,986
because you have been speaking heavily
for months now, if not a year, around,

82
00:04:56,136 --> 00:04:58,416
productivity specifically related to AI.

83
00:04:58,806 --> 00:05:02,200
Can you tell me, you recently
launched something significant.

84
00:05:02,200 --> 00:05:02,790
What was that?

85
00:05:03,018 --> 00:05:05,133
Laura Tacho: The AI measurement framework.

86
00:05:05,224 --> 00:05:06,724
Bret: AI measurement framework.

87
00:05:06,905 --> 00:05:07,775
What does that mean,

88
00:05:08,103 --> 00:05:09,043
Laura Tacho: What does that mean, Bret?

89
00:05:09,167 --> 00:05:10,432
Bret: Does it mean the AI is good or bad?

90
00:05:10,542 --> 00:05:11,172
Laura Tacho: yeah.

91
00:05:11,442 --> 00:05:12,852
Well, we help you find out.

92
00:05:13,072 --> 00:05:17,482
one of the things that I enjoy about my
job at DX is that we are not affiliated

93
00:05:17,482 --> 00:05:22,282
with any AI vendor or any other kind
of vendor for a tool in the SDLC.

94
00:05:22,282 --> 00:05:25,792
What we try to do is get you the
data so that you can interpret

95
00:05:25,792 --> 00:05:28,732
it and make decisions on what you
should do for your developers.

96
00:05:29,212 --> 00:05:32,182
because of that, we're in a
really unique position to.

97
00:05:32,732 --> 00:05:37,442
guide the industry on measuring the impact
of AI because I have a front row seat to

98
00:05:37,502 --> 00:05:40,982
400 plus companies and how they're using
AI, what's worked, what hasn't worked.

99
00:05:40,982 --> 00:05:42,962
I have a bunch of longitudinal data.

100
00:05:43,292 --> 00:05:46,242
I've been working on developer
productivity metrics, as my sole

101
00:05:46,242 --> 00:05:48,372
focus for four or five years now.

102
00:05:48,372 --> 00:05:51,202
And then before that, I've been in
developer tooling for like, 15 years.

103
00:05:51,202 --> 00:05:51,952
before Docker?

104
00:05:51,952 --> 00:05:53,992
Before, yeah, before we knew each other.

105
00:05:54,442 --> 00:05:58,102
back in January of last year, we wanted
to answer this question of like, how

106
00:05:58,102 --> 00:05:59,760
should I measure developer productivity?

107
00:05:59,760 --> 00:06:02,370
And that's where the DX
Core 4 Framework came out.

108
00:06:02,370 --> 00:06:05,576
So I worked on that with Abi
Noda, who's the co-founder of, DX.

109
00:06:05,576 --> 00:06:08,366
We worked with Nicole Forsgren,
other researchers and put

110
00:06:08,366 --> 00:06:09,506
together that guidance.

111
00:06:09,536 --> 00:06:13,146
Well, it didn't take
long before AI really.

112
00:06:13,411 --> 00:06:16,771
kind of blew everything up
late, you know, late 2024.

113
00:06:16,771 --> 00:06:20,461
A lot of companies were adopting it
and they were looking and thinking,

114
00:06:20,731 --> 00:06:22,531
do we need totally new metrics?

115
00:06:22,561 --> 00:06:24,751
Like how do we measure the impact of AI?

116
00:06:24,751 --> 00:06:27,121
How do I know if I'm having
a return on investment?

117
00:06:27,121 --> 00:06:30,791
How does this all relate to things
like DORA Metrics, the DX Core 4?

118
00:06:31,091 --> 00:06:34,031
we came up with the AI measurement
framework, which is really

119
00:06:34,031 --> 00:06:37,271
targeted specifically to measure
how and where AI is being used.

120
00:06:37,271 --> 00:06:40,831
when you use that together with the DX
Core 4, you get a comprehensive picture

121
00:06:41,101 --> 00:06:45,911
into, speed, quality, maintainability,
developer experience, business

122
00:06:45,911 --> 00:06:50,231
impact, all of those factors that AI
is supposed to be helping us with.

123
00:06:50,281 --> 00:06:53,761
maybe you can find out if it is or if
it isn't, and decide what to do next.

124
00:06:53,937 --> 00:06:56,817
Bret: Nirmal and I started this
podcast at London, which we were

125
00:06:56,817 --> 00:07:00,417
all at London for KubeCon earlier
this year, I think in April.

126
00:07:00,417 --> 00:07:04,377
And that's when Nirmal and I recorded
the first episode of this new podcast.

127
00:07:04,674 --> 00:07:08,634
at the time, we were still trying to
figure out, I feel like so much has

128
00:07:08,634 --> 00:07:10,134
happened since in the last five months.

129
00:07:10,464 --> 00:07:13,534
At the time we were still trying
to read the tea leaves around how

130
00:07:13,534 --> 00:07:19,414
AI is useful or not in the software
lifecycle, and where is it most useful?

131
00:07:19,534 --> 00:07:21,214
Where should we putting
our energy and attention?

132
00:07:21,214 --> 00:07:23,744
And that was really one of the major
premises for even starting this

133
00:07:23,744 --> 00:07:30,152
thing was everyone's talking about,
frontier models and dev tooling that

134
00:07:30,152 --> 00:07:32,062
adopts AI and everything's shiny.

135
00:07:32,092 --> 00:07:35,942
Everything's flashy, and we
were really interested in, where

136
00:07:35,942 --> 00:07:37,592
is it actually useful today?

137
00:07:37,622 --> 00:07:40,772
Because you can easily just sit down
and be overwhelmed with everything.

138
00:07:40,952 --> 00:07:44,742
But since then you've come out with
this framework, you've been talking

139
00:07:45,092 --> 00:07:51,542
over the place, multiple conferences
around the truth, and we've started

140
00:07:51,542 --> 00:07:55,562
to get some sensational headlines that
you talked recently on the Pragmatic

141
00:07:55,802 --> 00:08:00,632
Engineer Podcast around everything
from 30% of Microsoft Code is AI to.

142
00:08:01,877 --> 00:08:05,117
now we see things of despair, of
there's actually negative productivity

143
00:08:05,117 --> 00:08:08,267
in certain teams, and it just
feels like it's all over the board.

144
00:08:08,724 --> 00:08:11,864
can you distill all that down to what
you're seeing and what's fact from fiction

145
00:08:12,161 --> 00:08:13,001
Laura Tacho: let's rewind.

146
00:08:13,031 --> 00:08:16,711
There's a couple things that you
said that maybe we can kind of paint

147
00:08:16,711 --> 00:08:18,151
the picture of what's reality here.

148
00:08:18,151 --> 00:08:19,841
you mentioned one thing I
wanna dig into a little bit.

149
00:08:19,841 --> 00:08:23,801
You're talking about frontier models and
doing everything on the bleeding edge?

150
00:08:23,981 --> 00:08:29,051
all of us and our collective gray hair,
we were around for containerization.

151
00:08:29,051 --> 00:08:32,171
before the show started, we were talking
about how some companies were okay

152
00:08:32,171 --> 00:08:33,911
with pressing snooze on Kubernetes.

153
00:08:33,911 --> 00:08:36,281
They were okay with pressing
snooze on containerization.

154
00:08:36,611 --> 00:08:39,281
Maybe they're even okay with
pressing snooze on the cloud.

155
00:08:39,281 --> 00:08:41,861
And that just doesn't seem
to be happening with AI.

156
00:08:41,861 --> 00:08:44,441
there's really like a lot of
hunger to like adopt it, adopt

157
00:08:44,441 --> 00:08:46,181
it now, get the maximum gain.

158
00:08:46,631 --> 00:08:49,301
and you know, thinking about that
and what you said about the frontier

159
00:08:49,301 --> 00:08:52,608
models, it very much felt like what
we were doing How long ago was that?

160
00:08:52,608 --> 00:08:53,688
12, 13 years ago.

161
00:08:53,688 --> 00:08:56,778
With Docker, that very much
was on, on the frontier.

162
00:08:56,778 --> 00:08:59,288
And what we found was actually
the hardest problems are kind

163
00:08:59,288 --> 00:09:00,878
of the most boring problems.

164
00:09:00,878 --> 00:09:02,108
It was a lot of lift and shift.

165
00:09:02,108 --> 00:09:03,368
It was a lot of replatforming.

166
00:09:03,368 --> 00:09:07,523
It was a lot of just
like enterprise legacy.

167
00:09:07,573 --> 00:09:11,713
that is the majority of what
everyday developers are doing.

168
00:09:11,713 --> 00:09:16,573
They're not doing frontier work with
models they are trying to navigate

169
00:09:16,573 --> 00:09:21,783
and Cut through the spaghetti code
in their 10, 20 plus year code base.

170
00:09:21,833 --> 00:09:25,213
just showing up to work every day,
doing the best they can and enjoying

171
00:09:25,213 --> 00:09:29,523
the craft and, trying to help their
company hit their business objectives.

172
00:09:29,793 --> 00:09:34,438
when we talk about a lot of this, like
what's in the news, there is a lot about

173
00:09:34,448 --> 00:09:36,038
the frontier and the bleeding edge.

174
00:09:36,638 --> 00:09:40,148
What's really tricky though is
there's a big gap between the frontier

175
00:09:40,148 --> 00:09:43,608
and the bleeding edge and like the
experience of an quote unquote everyday

176
00:09:43,608 --> 00:09:46,758
developer and what they do every
day in the same way that we saw that

177
00:09:46,918 --> 00:09:51,638
in Kubernetes Docker time, there's
quite a gap between those things.

178
00:09:52,328 --> 00:09:54,638
So that's one thing I didn't
totally answer your question.

179
00:09:54,738 --> 00:09:55,218
Nirmal Mehta: so,

180
00:09:55,258 --> 00:09:56,248
Laura Tacho: a good place to start though.

181
00:09:56,249 --> 00:09:56,369
Bret: it.

182
00:09:56,498 --> 00:09:58,598
Nirmal Mehta: so there's a
difference that you're seeing, right?

183
00:09:58,598 --> 00:10:02,478
we have that traditional gardener
hype cycle, Like early adopters.

184
00:10:02,888 --> 00:10:05,988
and then the late
majority and the laggards.

185
00:10:06,358 --> 00:10:10,628
I think you were mentioning that
you've not seen some, situation where.

186
00:10:11,423 --> 00:10:17,063
The laggards and the late majority
are actually pursuing this technology

187
00:10:17,933 --> 00:10:23,613
aggressively as if they're early adopters
without understanding deeply the value

188
00:10:23,613 --> 00:10:25,753
proposition why is that happening?

189
00:10:26,056 --> 00:10:28,586
Laura Tacho: I think it usually
is, it comes down to money.

190
00:10:29,006 --> 00:10:32,036
there is a lot of sensationalization
sensational headlines about.

191
00:10:32,996 --> 00:10:37,526
What AI can do and its
impact on the marketplace.

192
00:10:37,576 --> 00:10:43,066
and its impact on sort of like bottom
line P&L on the, the economic situation.

193
00:10:43,216 --> 00:10:46,396
And I think that's been a huge
driving force for a lot of these

194
00:10:46,396 --> 00:10:50,026
companies that previously for any
other technology would've been very

195
00:10:50,026 --> 00:10:54,666
fine to be in the late adopter, or
late majority laggard kind of place.

196
00:10:54,666 --> 00:10:59,586
But the promise, the financial
gain for AI is just so promising.

197
00:10:59,866 --> 00:11:05,286
because it not only helps them operate
in a leaner way, but when their

198
00:11:05,286 --> 00:11:10,326
competitors also have access to that
technology, they will outpace them.

199
00:11:10,786 --> 00:11:15,976
I think that's the real threat
when I talk to CTOs and senior.

200
00:11:16,356 --> 00:11:20,816
exec teams from companies that previously,
would've been in that late majority,

201
00:11:20,816 --> 00:11:25,016
is that it's, it's not really about,
it's not about cutting headcount.

202
00:11:25,016 --> 00:11:26,576
It's not about necessarily cutting costs.

203
00:11:26,576 --> 00:11:29,966
It's that their competitors have access
to this technology and they're so worried

204
00:11:29,966 --> 00:11:35,316
about being outpaced they want to catch
up as fast as they can to not lose

205
00:11:35,316 --> 00:11:39,306
footing in the marketplace and their
competitiveness, because that's the risk.

206
00:11:39,406 --> 00:11:41,926
and that is a huge disadvantage
if that would come true.

207
00:11:42,326 --> 00:11:45,936
Nirmal Mehta: So that implies
that the CTOs and the leaders that

208
00:11:45,936 --> 00:11:51,876
you're talking to in companies that
do face, real competition, right?

209
00:11:51,926 --> 00:11:55,616
if they're representing industries
where there's not necessarily that

210
00:11:55,616 --> 00:11:58,326
much competitive, pressure, then that,

211
00:11:58,951 --> 00:11:59,371
Laura Tacho: Mm-hmm.

212
00:11:59,706 --> 00:12:02,466
Nirmal Mehta: that might not be
necessarily true and they might be

213
00:12:02,466 --> 00:12:06,636
able to hold out a little bit longer
and truly see if there's an advantage.

214
00:12:07,162 --> 00:12:09,292
Laura Tacho: Yeah, I think I see
competitive pressure on two ways.

215
00:12:09,292 --> 00:12:10,882
One is like in the marketplace, right?

216
00:12:10,882 --> 00:12:14,762
Like capitalism, but the other
places in the marketplace for talent.

217
00:12:15,182 --> 00:12:18,902
And that is also a very big concern
for these companies that you know, they

218
00:12:18,902 --> 00:12:22,982
feel like they're behind on AI, because
they're worried that because they're

219
00:12:22,982 --> 00:12:26,072
not adopting these tools, they're no
longer an interesting place or rewarding

220
00:12:26,072 --> 00:12:27,512
place for their developers to work.

221
00:12:27,512 --> 00:12:31,392
And that they're gonna have a brain drain
people leaving their company and going

222
00:12:31,392 --> 00:12:35,052
to work for places that are adopting
these tools, embracing them more.

223
00:12:35,202 --> 00:12:38,622
I think the market as it is right
now isn't very conducive to that

224
00:12:38,622 --> 00:12:42,892
happening, but it's definitely something
running, in the background of a lot

225
00:12:42,892 --> 00:12:44,812
of executives minds about, making it.

226
00:12:45,737 --> 00:12:49,577
attractive for talent to stay
reducing attrition risk, that kind of

227
00:12:49,577 --> 00:12:52,727
competition in the talent marketplace
along with, competition, in the

228
00:12:52,727 --> 00:12:54,017
marketplace for their software.

229
00:12:54,017 --> 00:12:55,557
Whatever businesses, they're selling.

230
00:12:56,097 --> 00:12:59,367
Bret: one of the thoughts I've had
early that we've discussed earlier

231
00:12:59,367 --> 00:13:03,680
was if there's any marketing win for.

232
00:13:04,745 --> 00:13:08,315
Big tech, specifically the AI
companies, which is all of big tech now.

233
00:13:08,705 --> 00:13:17,527
it's, they convinced us all from
average Dev to CIO or CEO, that AI

234
00:13:17,527 --> 00:13:21,697
is going to speed up everything and
that they need to adopt it everywhere.

235
00:13:22,137 --> 00:13:23,937
we talked earlier where's the precedent?

236
00:13:23,937 --> 00:13:27,177
where's the previous precedent of
what we're dealing with right now?

237
00:13:27,267 --> 00:13:31,407
I struggled to come up with one because
when the cloud started, it was one company

238
00:13:31,457 --> 00:13:34,667
then Docker came out, it was one company
and there were other ones trying to,

239
00:13:34,767 --> 00:13:36,777
play around with containers and stuff.

240
00:13:36,777 --> 00:13:39,387
But there was this singular
focus on a single company.

241
00:13:39,567 --> 00:13:45,187
And now we have an entire industry
whether it's CIO Magazine or the

242
00:13:45,307 --> 00:13:49,657
Pragmatic Engineer, you know,
newsletter, like they're all telling

243
00:13:49,657 --> 00:13:53,917
us that these tools are being
adopted by everyone else, not us.

244
00:13:54,067 --> 00:13:57,267
So we have the feeling of being left
out, the feeling of being left behind,

245
00:13:57,807 --> 00:14:01,787
and the main reason that everyone
believes this is better is because

246
00:14:01,787 --> 00:14:03,647
it improves the speed of business.

247
00:14:04,167 --> 00:14:09,587
we've all been hypnotized by it and
obviously we have people that are and,

248
00:14:09,662 --> 00:14:16,292
waiting and all that stuff, but it feels
like they are so, a minority as opposed

249
00:14:16,292 --> 00:14:19,472
to other trends that it does feel like.

250
00:14:19,892 --> 00:14:27,482
It's this just tsunami of tooling,
of solutions, of promises that I just

251
00:14:27,482 --> 00:14:30,892
don't think we're really prepared to an
industry to like separate the truth from

252
00:14:30,892 --> 00:14:32,812
the, fiction and the hype from reality.

253
00:14:32,812 --> 00:14:35,054
And we're just not really
well equipped right now.

254
00:14:35,054 --> 00:14:35,534
I feel for that

255
00:14:35,750 --> 00:14:36,152
Laura Tacho: Mm-hmm.

256
00:14:36,192 --> 00:14:38,082
I think you're very
levelheaded in your approach.

257
00:14:38,082 --> 00:14:42,932
you ask a lot of good questions
about wait, but what about X, Y, Z?

258
00:14:42,962 --> 00:14:44,132
where do you put yourself on that?

259
00:14:44,132 --> 00:14:45,782
skeptic versus advocate?

260
00:14:46,022 --> 00:14:47,282
I mean, you can be both, right?

261
00:14:47,282 --> 00:14:48,482
This isn't mutually exclusive.

262
00:14:48,482 --> 00:14:50,762
Like I'm skeptical about a
lot of things 'cause I see.

263
00:14:51,097 --> 00:14:54,224
The real data, but also like you
two truths, like we can be hopeful

264
00:14:54,224 --> 00:14:56,974
about something that will be there
in the future, but also skeptical

265
00:14:56,974 --> 00:14:58,384
about the current capabilities.

266
00:14:58,384 --> 00:14:59,674
how do you reconcile that?

267
00:14:59,824 --> 00:15:02,194
Especially 'cause you're steeped
in this every single day.

268
00:15:03,184 --> 00:15:06,394
Nirmal Mehta: great question and
thanks for throwing it back at us.

269
00:15:06,474 --> 00:15:11,064
I think, one thing that we've learned,
all three of us, in our journey adopting

270
00:15:11,334 --> 00:15:15,884
like bleeding edge stuff is, new
technology paradigms is to be, to have

271
00:15:15,884 --> 00:15:20,024
a healthy dose of skepticism, especially
around value proposition around the

272
00:15:20,024 --> 00:15:21,944
capabilities of technology, right?

273
00:15:21,944 --> 00:15:27,914
Like early containerization full of sharp
edges that have, have now been dulled,

274
00:15:28,434 --> 00:15:28,854
Laura Tacho: Mm-hmm.

275
00:15:29,204 --> 00:15:30,284
Nirmal Mehta: over the last 15 years.

276
00:15:30,284 --> 00:15:35,679
But, spinning up a container was
full of, issues back a long time

277
00:15:35,904 --> 00:15:36,324
Laura Tacho: Mm-hmm.

278
00:15:36,364 --> 00:15:40,829
Nirmal Mehta: and, we all had a healthy
dose of skepticism, but we also saw a path

279
00:15:40,829 --> 00:15:46,379
to value for folks that were adopting that
technology and immediate use cases, right?

280
00:15:46,379 --> 00:15:51,704
Like, going from spinning up a VM
and waiting an hour, as your chef

281
00:15:51,884 --> 00:15:57,506
scripts go through and install apache
server, to in two seconds having 20

282
00:15:57,506 --> 00:15:59,396
containers with the same configuration.

283
00:15:59,756 --> 00:16:04,346
It's a very direct line from seeing the
technology and the value proposition in

284
00:16:04,346 --> 00:16:06,206
terms of cost savings or whatever, right?

285
00:16:06,206 --> 00:16:07,436
If you wanna follow the money.

286
00:16:08,163 --> 00:16:13,203
My healthy dose of skepticism with AI
right now is that it seems like we're

287
00:16:13,203 --> 00:16:20,013
in a perfect storm of other factors that
are going on, and is a lot of pressure

288
00:16:22,293 --> 00:16:23,403
there's so much.

289
00:16:23,778 --> 00:16:31,248
Money involved, and so much pressure
from all sides a tremendous amount

290
00:16:31,308 --> 00:16:33,578
of fog over this technology.

291
00:16:33,948 --> 00:16:37,628
on one side, having adopted these
tools, I'm privileged enough

292
00:16:37,628 --> 00:16:40,928
to have almost unlimited access
to a lot of these technologies.

293
00:16:41,288 --> 00:16:44,728
And enablement from some
of the world's best folks.

294
00:16:44,728 --> 00:16:45,388
my colleagues

295
00:16:45,438 --> 00:16:45,858
Laura Tacho: Mm-hmm.

296
00:16:45,868 --> 00:16:49,078
Nirmal Mehta: create this stuff every
day, that use this stuff every day.

297
00:16:49,468 --> 00:16:52,348
it's hard to push back
against that feeling of magic.

298
00:16:52,348 --> 00:16:57,698
When I first started doing some of
the code generation with, ChatGPT

299
00:16:57,718 --> 00:17:02,648
last year, and then moving over to,
Cline earlier this year and doing some

300
00:17:02,648 --> 00:17:07,578
agentic, work with Cline and setting
up multi-agent to do operational tasks,

301
00:17:07,578 --> 00:17:09,738
to do code generation and prototyping.

302
00:17:10,138 --> 00:17:16,268
it does provide value, Like even,
helping, come up with, questions

303
00:17:16,268 --> 00:17:18,608
to ask guests on a podcast.

304
00:17:18,798 --> 00:17:20,238
there's value there.

305
00:17:20,838 --> 00:17:25,568
At the same time, we have the
ending of zero interest, in

306
00:17:25,568 --> 00:17:26,768
the United States economy.

307
00:17:26,768 --> 00:17:29,828
We have global economic turmoil.

308
00:17:29,878 --> 00:17:36,259
The promise of these tools to,
improve productivity in not only

309
00:17:36,259 --> 00:17:39,729
developer, code generation, but
all kinds of different avenues.

310
00:17:40,129 --> 00:17:47,125
and a kind of, under, under the
breath, like, oh, that also means

311
00:17:47,125 --> 00:17:48,475
we don't need as many people.

312
00:17:49,001 --> 00:17:56,077
And it's, I'm, I am healthily skeptical
of what if the technology will actually

313
00:17:56,077 --> 00:18:02,227
deliver on like these hyped up, premises,
like 60% productivity gains, At the

314
00:18:02,227 --> 00:18:08,297
same time, I don't know if that matters,
I don't know if the leaders at these

315
00:18:08,297 --> 00:18:13,887
companies will actually care about
whether there's value there versus using

316
00:18:13,887 --> 00:18:21,087
it as an excuse to execute against other
defensive maneuvers to navigate all these

317
00:18:21,087 --> 00:18:23,097
other things happening in the world.

318
00:18:23,097 --> 00:18:27,127
I end up very confused and, I really
appreciate that in your latest

319
00:18:27,127 --> 00:18:31,367
talk at lead dev, you have this big
slide that said, data beats hype.

320
00:18:32,077 --> 00:18:35,977
I think it's not only that data beats
hype, which I really appreciate and

321
00:18:35,977 --> 00:18:41,347
I would love for you to get into, but
We have to present that data as fast

322
00:18:41,347 --> 00:18:48,217
as possible because the hype seems to
be accelerating decisions are gonna

323
00:18:48,217 --> 00:18:51,137
be made before the data is even there.

324
00:18:51,547 --> 00:18:56,327
And I'm really worried that damage is
gonna be done maybe a year from now

325
00:18:56,327 --> 00:19:00,977
we're like this fever dream of adopting
these AI tools didn't come to fruition.

326
00:19:00,977 --> 00:19:05,147
But you know, it doesn't really matter
because we're all not working anymore.

327
00:19:05,297 --> 00:19:05,537
Right?

328
00:19:06,002 --> 00:19:06,422
Laura Tacho: Mm-hmm.

329
00:19:06,797 --> 00:19:07,547
Nirmal Mehta: Like, does that even

330
00:19:07,852 --> 00:19:08,142
Laura Tacho: Yeah,

331
00:19:08,177 --> 00:19:11,237
Nirmal Mehta: great, we have data
that shows that this is not like, this

332
00:19:11,237 --> 00:19:14,207
isn't me, all the things that, all
the hype, but it's kind of too late.

333
00:19:14,497 --> 00:19:15,937
I feel like that's where I am.

334
00:19:15,967 --> 00:19:19,477
And if I feel that way, I feel like
our audience is probably feeling

335
00:19:19,477 --> 00:19:21,907
that same way or will be soon.

336
00:19:22,297 --> 00:19:24,397
Laura Tacho: I think there's
already decisions being made

337
00:19:24,397 --> 00:19:30,049
based on hype for example, knowing
how much code that's running in

338
00:19:30,049 --> 00:19:31,639
production was authored with AI.

339
00:19:32,884 --> 00:19:37,204
Is a very important thing to understand
when it comes to your security posture.

340
00:19:37,474 --> 00:19:42,094
Understanding like changes to SRE,
understanding how you might need to

341
00:19:42,094 --> 00:19:44,014
change or harden your build and test.

342
00:19:44,014 --> 00:19:46,804
it's an important thing for businesses
to know, like how much of their

343
00:19:46,804 --> 00:19:50,224
code is actually AI generated code
is actually making to production.

344
00:19:50,224 --> 00:19:54,674
Because there's different decisions
you'll make if it's 0% or if it's 90% Up

345
00:19:54,674 --> 00:20:02,449
until three weeks ago, there was no way
to know, there was no technical solution.

346
00:20:02,829 --> 00:20:06,039
you know, broadly speaking, there were
some pockets of technical solution

347
00:20:06,114 --> 00:20:08,979
of being able to track acceptance.

348
00:20:09,159 --> 00:20:14,499
So a suggestion, like a code suggestion
in an IDE through an accepted suggestion.

349
00:20:14,499 --> 00:20:19,809
So that's in your IDE through human
authored edits, committed open.

350
00:20:20,124 --> 00:20:22,224
PR merged PR in production.

351
00:20:22,254 --> 00:20:24,024
There's just no way to do that.

352
00:20:24,024 --> 00:20:25,674
Sort of like provenance for code.

353
00:20:26,154 --> 00:20:30,234
But still, we see headlines six
months ago saying, oh, 30% of our

354
00:20:30,234 --> 00:20:32,214
code is being written by software.

355
00:20:32,244 --> 00:20:35,764
from my position on the inside,
I sit in AI data all day long.

356
00:20:35,764 --> 00:20:37,564
This is literally, it's my, it's all I do.

357
00:20:37,894 --> 00:20:38,434
It's all I do.

358
00:20:38,434 --> 00:20:39,304
It's all I think about.

359
00:20:39,844 --> 00:20:42,664
There is no way, there is no
way to know that reliably.

360
00:20:42,664 --> 00:20:46,114
You can, of course you can do
really structured sampling.

361
00:20:46,144 --> 00:20:49,024
You can do self-reported data
and self-reported data does

362
00:20:49,024 --> 00:20:50,914
get us most of the way there.

363
00:20:51,074 --> 00:20:52,064
but that's not what this was.

364
00:20:52,274 --> 00:20:57,404
This is coming from, this is coming
from like, you know, snippets or an

365
00:20:57,404 --> 00:20:59,624
executive estimating in some cases.

366
00:20:59,624 --> 00:21:02,204
How much of the code from
their teams is being written?

367
00:21:02,354 --> 00:21:04,034
Just very sort of unreliable.

368
00:21:04,214 --> 00:21:05,144
very lossy.

369
00:21:05,144 --> 00:21:05,984
Very lossy.

370
00:21:05,984 --> 00:21:06,464
Yeah.

371
00:21:06,494 --> 00:21:08,354
Vibe, vibe, metrics.

372
00:21:08,804 --> 00:21:13,104
And so, I do really have a lot
of empathy for A CEO who does not

373
00:21:13,104 --> 00:21:17,334
have a background as a developer
and cannot distinguish claims.

374
00:21:17,584 --> 00:21:21,184
due to lack of subject matter expertise
about what is actually authentic

375
00:21:21,184 --> 00:21:25,114
and what, what is technology really
capable of versus what it's not.

376
00:21:25,424 --> 00:21:28,244
of course the CEO should be seeking
out subject matter expertise, and

377
00:21:28,244 --> 00:21:32,304
it is the role of the CTO or VPs,
of engineering data, to bring that

378
00:21:32,304 --> 00:21:33,774
knowledge into the organization.

379
00:21:33,774 --> 00:21:37,344
But I a hundred percent believe there
are decisions being made about AI

380
00:21:37,344 --> 00:21:40,974
strategy and investment on, like,
we can call them vapor metrics.

381
00:21:40,974 --> 00:21:43,024
They're just sort of
like, or vibe metrics.

382
00:21:43,024 --> 00:21:43,444
I like that.

383
00:21:43,444 --> 00:21:45,724
Nirmal, it's like vibe metrics.

384
00:21:45,724 --> 00:21:47,614
it's about gut feel and not about.

385
00:21:47,909 --> 00:21:51,079
measurement, not out of Ill
will or malice, but just out of

386
00:21:51,079 --> 00:21:56,169
lack of telemetry because this
has been accelerating so fast.

387
00:21:56,359 --> 00:22:03,319
can you imagine, having to decide on your
budget for Docker spend or cloud spend

388
00:22:03,349 --> 00:22:07,919
for the next fiscal year when Docker
couldn't even tell you like docker ps,

389
00:22:08,239 --> 00:22:11,209
um, how many, how many processes were
running or what the size of an image

390
00:22:11,209 --> 00:22:14,609
was and you were trying to forecast
your, image registry costs that's

391
00:22:14,609 --> 00:22:16,049
kind of what this would feel like.

392
00:22:16,169 --> 00:22:17,699
which doesn't make sense, right?

393
00:22:17,699 --> 00:22:18,749
It's very dangerous.

394
00:22:18,809 --> 00:22:22,139
but because of the hype,
it's where we find ourself.

395
00:22:22,239 --> 00:22:23,739
right now, at least
in, in a lot of pocket.

396
00:22:23,739 --> 00:22:24,459
It is getting better.

397
00:22:24,459 --> 00:22:27,099
I have to say that in the last couple
months has gotten a lot of better, but.

398
00:22:27,387 --> 00:22:29,307
Nirmal Mehta: Yeah, just
one last comment on that.

399
00:22:29,697 --> 00:22:31,917
At the end of the day,
data will win, right?

400
00:22:31,917 --> 00:22:35,617
data that backs up the
value proposition will win.

401
00:22:36,267 --> 00:22:41,197
there might be some pain as folks
adopt these technologies preemptively

402
00:22:41,427 --> 00:22:43,377
to address competitive pressure.

403
00:22:43,977 --> 00:22:48,347
your approach at DX and doing the hard
work of collecting data, you know,

404
00:22:48,347 --> 00:22:53,077
even if it's anecdotal and not the
best, quality data at, at this time,

405
00:22:53,317 --> 00:22:59,848
it's still the only way that, both
leadership and folks that are being

406
00:22:59,848 --> 00:23:04,228
asked to adopt these technologies and,
and find those productivity gains or

407
00:23:04,228 --> 00:23:09,778
else we'll be able to really navigate,
getting something out of these tools.

408
00:23:09,808 --> 00:23:11,608
'cause at the end of the day, I do.

409
00:23:12,033 --> 00:23:15,963
unlike some of these other technologies
that have come out in the last 10 years.

410
00:23:16,643 --> 00:23:20,553
I do see, you know, in, in my own
adoption, I do see a path to value.

411
00:23:20,593 --> 00:23:24,703
I have received gains
from using these tools.

412
00:23:24,853 --> 00:23:28,213
So there is something there
with respect to the percentage.

413
00:23:28,243 --> 00:23:31,953
I think your approach with the AI
measurement framework and collecting

414
00:23:31,953 --> 00:23:39,393
that data is gonna be key to figuring
out where the actual value is and also

415
00:23:39,753 --> 00:23:44,313
changing how, you know, I think there's
another, uh, another aspect to this,

416
00:23:44,363 --> 00:23:48,904
from a potential developer or platform
engineer perspective, I'm seeing a trend

417
00:23:48,904 --> 00:23:52,414
of, their leadership telling them, they

418
00:23:52,414 --> 00:23:52,834
Laura Tacho: Mm-hmm.

419
00:23:52,839 --> 00:23:53,829
Nirmal Mehta: use these tools.

420
00:23:54,459 --> 00:24:00,659
And a lot of folks who have tried these
tools like maybe once or twice at some

421
00:24:00,659 --> 00:24:06,429
point in the past year, didn't work for
them, maybe don't really know how to use

422
00:24:06,429 --> 00:24:11,699
it properly or at that time was not the
most effective way or effective model.

423
00:24:11,819 --> 00:24:13,859
And because things have changed so much.

424
00:24:14,184 --> 00:24:16,464
we've gotten so many more best practices.

425
00:24:16,884 --> 00:24:17,874
They're, skeptical.

426
00:24:17,874 --> 00:24:19,404
Like, why do I have to try using this?

427
00:24:19,404 --> 00:24:21,264
Like, I'm being forced to use a tool?

428
00:24:21,634 --> 00:24:25,734
and that's never a great position, but
data is the way for those folks to push

429
00:24:25,734 --> 00:24:30,444
back against leadership and potentially
adopt those tools in a way that will

430
00:24:30,444 --> 00:24:31,934
actually be productive for them.

431
00:24:33,244 --> 00:24:33,994
Laura Tacho: absolutely right.

432
00:24:33,994 --> 00:24:38,224
I think, I've seen lots of companies use
the measurements that come out of the AI

433
00:24:38,224 --> 00:24:42,874
measurement framework, not just to show
the ROI to their leadership, but also to

434
00:24:43,024 --> 00:24:48,154
bring that back to the developers to show,
oh, you know, you might have been like

435
00:24:48,154 --> 00:24:51,094
thinking, sensing, feeling, perceiving
this about your own productivity,

436
00:24:51,094 --> 00:24:52,564
but here's actually what's happened.

437
00:24:52,834 --> 00:24:56,434
There has been a 30% lift in
pull requests or merge requests

438
00:24:56,434 --> 00:24:58,474
for those using AI daily.

439
00:24:58,474 --> 00:25:01,954
you're also reporting that code
maintainability is not slipping and that

440
00:25:01,954 --> 00:25:07,814
you're still having the same amount of
effort to review in AI assisted diff

441
00:25:07,814 --> 00:25:09,524
versus one that wasn't AI assisted.

442
00:25:09,584 --> 00:25:14,474
And that can actually be really powerful
for companies to have a validation loop

443
00:25:14,954 --> 00:25:19,304
and get people interested in maybe giving
it another try because it's not hype.

444
00:25:19,694 --> 00:25:20,324
It's data.

445
00:25:20,354 --> 00:25:23,564
It's not someone saying, oh, this is
gonna give you a 10x productivity boost.

446
00:25:23,564 --> 00:25:26,354
It's like, well you probably are
gonna save a couple hours and

447
00:25:26,354 --> 00:25:28,814
it's gonna make it a lot more fun
and reduce your cognitive load.

448
00:25:28,814 --> 00:25:30,164
So why don't you give it a try?

449
00:25:30,374 --> 00:25:33,984
Or we have this new use case
that helps you, modernize

450
00:25:33,984 --> 00:25:37,504
this old bit to, the new bit.

451
00:25:37,584 --> 00:25:41,094
with 70% of the way there, you can
just do it with AI and then you only

452
00:25:41,094 --> 00:25:42,684
have to take care of that last 30%.

453
00:25:42,684 --> 00:25:44,994
doesn't that sound much better
than doing it all by hand?

454
00:25:45,354 --> 00:25:47,154
Yeah, maybe I'm gonna give it a try.

455
00:25:47,154 --> 00:25:49,724
data is important both for
leadership, but also it's very

456
00:25:49,724 --> 00:25:51,284
important for developers themselves.

457
00:25:51,528 --> 00:25:54,098
Bret: in order to get the advantages
you actually need to do more

458
00:25:54,098 --> 00:25:55,748
than just add AI to your IDE.

459
00:25:56,028 --> 00:26:01,368
it's almost like a spray and pray has
been the attitude of, the AI hype cycle

460
00:26:01,418 --> 00:26:05,318
as long as you've got AI code generation
happening, as long as you just throw AI

461
00:26:05,318 --> 00:26:09,518
into your CI, like just, it's somehow
going to magically make everything faster.

462
00:26:09,518 --> 00:26:12,568
we're now seeing evidence
that, it's more than that.

463
00:26:12,818 --> 00:26:14,228
which is true of anything in tech.

464
00:26:14,228 --> 00:26:17,558
I don't know why we stopped
believing that for a moment

465
00:26:17,588 --> 00:26:18,758
'cause it's always been that way.

466
00:26:19,138 --> 00:26:20,488
what is some of the stuff
that's happened lately?

467
00:26:20,538 --> 00:26:22,733
Laura Tacho: Yeah, let me show
you actually, I can just talk

468
00:26:22,733 --> 00:26:23,713
you through some of the metrics.

469
00:26:23,713 --> 00:26:25,783
We can look at some of the new
stories and then I'll show you

470
00:26:25,783 --> 00:26:29,533
what's really going on out there.

471
00:26:29,813 --> 00:26:30,263
this is a good.

472
00:26:31,028 --> 00:26:34,898
Juxtaposition of what we're seeing a
lot of open up LinkedIn, it'll take you

473
00:26:34,898 --> 00:26:36,278
30 seconds to find something like this.

474
00:26:36,278 --> 00:26:38,258
So we've got sensational headline.

475
00:26:38,258 --> 00:26:39,698
I mean, this one's not
even that sensational.

476
00:26:39,698 --> 00:26:40,538
It's like pretty grounded.

477
00:26:40,538 --> 00:26:42,818
It's only saying 10% more
productive at Google.

478
00:26:43,068 --> 00:26:45,828
and then talking through their
measurement methodology TLDR on that one.

479
00:26:45,828 --> 00:26:47,628
They're measuring time
saved per developer.

480
00:26:47,948 --> 00:26:50,528
10% more productive is on the low end.

481
00:26:50,528 --> 00:26:55,378
We're seeing other reports saying, it's
writing 30% of our code, or it's saving

482
00:26:55,408 --> 00:26:58,168
our developers 20% of time per week.

483
00:26:58,348 --> 00:27:01,588
There's so much variability here because
there's so much variability in what

484
00:27:01,588 --> 00:27:06,298
actually is being counted when we say
AI and developers in the same sentence.

485
00:27:06,868 --> 00:27:09,268
For some, we're looking at only code gen.

486
00:27:09,478 --> 00:27:13,288
For some we're looking at AI assisted
engineering, which can be anything from

487
00:27:13,768 --> 00:27:16,528
refactoring, debugging, production,
debugging, lots of other things.

488
00:27:16,868 --> 00:27:19,958
Atlassian, their state of, DevEx report,
which I'm actually gonna talk about.

489
00:27:19,958 --> 00:27:21,898
I think this episode will come out after.

490
00:27:21,968 --> 00:27:25,318
but I'm doing a live event to talk
through their state of DevEx report.

491
00:27:25,318 --> 00:27:31,258
Atlassian has put AI in Jira,
Confluence, all of their productivity

492
00:27:31,258 --> 00:27:34,488
suite of, project management
tools, task management tools.

493
00:27:34,858 --> 00:27:38,248
they see a lot of developers
saving a lot of time when you

494
00:27:38,248 --> 00:27:40,168
add AI to Jira, for example.

495
00:27:40,228 --> 00:27:43,408
But that's really different than
adding AI to an IDE, you know?

496
00:27:43,408 --> 00:27:47,098
And so we have to ask the question,
and those of you listening, when

497
00:27:47,098 --> 00:27:49,788
you look at a number, ask the
question, what's the boundary?

498
00:27:49,788 --> 00:27:52,188
What's the start and stop
condition for this metric?

499
00:27:52,218 --> 00:27:53,658
what are they actually measuring?

500
00:27:53,878 --> 00:27:54,708
who's being measured?

501
00:27:55,093 --> 00:27:56,793
Is it, a random sample?

502
00:27:56,793 --> 00:27:59,073
Is it customers of a particular company?

503
00:27:59,323 --> 00:28:01,873
ask yourself, does that
company sell tools with AI?

504
00:28:01,873 --> 00:28:02,563
Are they trying to.

505
00:28:03,133 --> 00:28:06,283
show me favorable results so that I
might be interested in buying their tool.

506
00:28:06,283 --> 00:28:09,973
Like there's, there's a lot of questions
to ask, like data literacy wise.

507
00:28:10,513 --> 00:28:13,753
So over here we've got these
headlines about, 10% more productive.

508
00:28:14,123 --> 00:28:18,803
this graph here that we see is a study
for MITRE, and what they did was take

509
00:28:18,983 --> 00:28:24,053
16 developers and they kind of like
followed them around for a period of time.

510
00:28:24,323 --> 00:28:26,843
They are working on open source
projects and they have a pretty,

511
00:28:26,843 --> 00:28:30,353
already pretty good, like moderate
to heavy familiarity with AI tools.

512
00:28:30,353 --> 00:28:33,203
They're all professional
programmers and they tagged some.

513
00:28:33,638 --> 00:28:37,148
Tasks as AI allowed and AI not allowed.

514
00:28:37,388 --> 00:28:39,998
And so the ones that were AI not
allowed, they definitely couldn't use

515
00:28:39,998 --> 00:28:41,768
AI on the ones that were AI allowed.

516
00:28:41,768 --> 00:28:43,298
They could use it however they wanted to.

517
00:28:43,808 --> 00:28:48,968
And they had them kind of do some
pre-estimation of how much time

518
00:28:48,968 --> 00:28:51,938
do you think this is gonna take
you to do this task if you use

519
00:28:51,938 --> 00:28:53,618
AI versus if you don't use AI?

520
00:28:53,618 --> 00:28:56,888
And they had them do some kind of
predictive, estimations then they

521
00:28:56,888 --> 00:28:59,418
observed them actually, screen recordings.

522
00:28:59,418 --> 00:29:02,598
They also did some exit
interview style kind of things.

523
00:29:02,868 --> 00:29:06,828
What they found was that the
tasks the developers used AI

524
00:29:06,858 --> 00:29:11,508
were actually 16% slower than
the ones they didn't use AI for.

525
00:29:11,778 --> 00:29:16,248
So even though their perception
of the change in time, if

526
00:29:16,248 --> 00:29:21,588
AI allowed was positive, the
actual result was 16% slower.

527
00:29:22,308 --> 00:29:25,788
And this is very valid data
gathered in a rigorous way.

528
00:29:25,788 --> 00:29:29,628
It's hard to find 16 developers and
follow them around for 240, tasks.

529
00:29:29,628 --> 00:29:30,918
this was a very big footprint.

530
00:29:31,129 --> 00:29:31,369
Bret: Yeah.

531
00:29:31,458 --> 00:29:35,568
Laura Tacho: The the thing though, this
is about open source developers, um, and

532
00:29:35,568 --> 00:29:40,308
specifically they're talking about open
source developers who are working on tasks

533
00:29:40,308 --> 00:29:43,878
that are a little bit more like on the
bleeding edge on the frontier, where code

534
00:29:43,878 --> 00:29:48,798
generation was a huge part of the workload
that they were trying to use AI for.

535
00:29:49,338 --> 00:29:55,728
That is not very similar to, again,
that every day engineer or working

536
00:29:55,728 --> 00:29:59,958
in the enterprise who is not doing
bleeding edge frontier kind of work.

537
00:30:00,298 --> 00:30:03,928
and so while the MITRE study is
very interesting and, and definitely

538
00:30:03,928 --> 00:30:05,518
points out, we need more research.

539
00:30:05,908 --> 00:30:10,858
When we try to generalize this and say,
well, using AI for AI assisted engineering

540
00:30:11,038 --> 00:30:15,568
is gonna give your developers a 16%
slowdown across the board, irrespective

541
00:30:15,598 --> 00:30:19,858
of what type of task, irrespective
of type of developer, tenure company.

542
00:30:20,098 --> 00:30:23,878
That also oversimplifies to the
point of being incorrect as well.

543
00:30:24,188 --> 00:30:26,918
and so we're definitely seeing
this kind of from both ends.

544
00:30:27,018 --> 00:30:31,218
the skeptics using the MITRE study to say,
AI is a bunch of BS and you shouldn't use

545
00:30:31,218 --> 00:30:34,968
it to the people on the other side saying,
but you know, Google's saving 10% of time.

546
00:30:35,088 --> 00:30:38,238
It's just really hard to know and
there's not a lot of literature

547
00:30:38,388 --> 00:30:41,418
on it because it's so new and we
don't have longitudinal data because

548
00:30:41,418 --> 00:30:42,738
it's not been around for very long.

549
00:30:42,738 --> 00:30:45,168
We've only been able to study
this for like two years.

550
00:30:45,228 --> 00:30:49,818
It's not like we can look back to the
1970s and look at patterns and see

551
00:30:49,818 --> 00:30:51,348
how things have changed over decades.

552
00:30:51,743 --> 00:30:54,523
Nirmal Mehta: So you just brought up
a really interesting thing, which is

553
00:30:54,523 --> 00:30:57,583
the digging into that MITRE study.

554
00:30:58,033 --> 00:31:01,193
It was around a specific
use case, code-gen.

555
00:31:01,637 --> 00:31:05,417
what in your, framework and when
you were talking to your customers

556
00:31:05,447 --> 00:31:07,177
or to your survey respondents,

557
00:31:07,177 --> 00:31:07,407
Laura Tacho: What

558
00:31:07,497 --> 00:31:07,777
Nirmal Mehta: the

559
00:31:07,847 --> 00:31:08,327
Laura Tacho: yeah.

560
00:31:09,112 --> 00:31:10,232
Nirmal Mehta: outside of code-gen?

561
00:31:11,072 --> 00:31:11,362
Laura Tacho: Yeah.

562
00:31:11,392 --> 00:31:15,442
Nirmal Mehta: is maybe not the majority
of what people are doing every day, right?

563
00:31:16,211 --> 00:31:16,376
Laura Tacho: Yeah.

564
00:31:16,436 --> 00:31:17,926
And I think that's a
really important thing.

565
00:31:17,976 --> 00:31:20,346
So I'll also be transparent
when we do research.

566
00:31:20,346 --> 00:31:22,776
A lot of it, like if you see
me post something on LinkedIn.

567
00:31:23,166 --> 00:31:26,586
That's most likely aggregated
customer data that I have access to.

568
00:31:26,736 --> 00:31:30,276
When we publish reports and guides like
the one I'm about to show you, we do seek

569
00:31:30,276 --> 00:31:35,566
out external, participants as well to not
only include people that are customers of

570
00:31:35,566 --> 00:31:40,026
DX because that's, very self-selecting, in
a certain profile of company that really

571
00:31:40,026 --> 00:31:42,126
cares about DX developer experience.

572
00:31:42,322 --> 00:31:43,522
Bret: they're already tip of the spear.

573
00:31:43,522 --> 00:31:45,652
I've never been lucky enough
to work at a company that had

574
00:31:45,652 --> 00:31:47,722
anyone focus exclusively on DX.

575
00:31:48,052 --> 00:31:48,442
Laura Tacho: Yeah.

576
00:31:49,103 --> 00:31:52,703
Bret: I've always been fascinated in
your work because it makes every job that

577
00:31:52,703 --> 00:31:57,323
I've had and every contracting client
I've had, feel like they're amateurs.

578
00:31:57,373 --> 00:32:00,583
when I read the reports and the
information that you all put out in

579
00:32:00,583 --> 00:32:05,483
Dora and other, sort of, tip of the
spear, what are the elites doing?

580
00:32:05,533 --> 00:32:08,323
it always gives me like,
what are we even doing?

581
00:32:08,323 --> 00:32:09,823
The rest of us are just winging it.

582
00:32:09,873 --> 00:32:11,253
we don't have these methodologies.

583
00:32:11,253 --> 00:32:15,463
We don't have these pipelines of
data that we're pulling out of

584
00:32:15,463 --> 00:32:19,593
all of our utilities in order to
gather stats to change behavior,

585
00:32:19,593 --> 00:32:20,733
which I feel like is a lot of this.

586
00:32:20,733 --> 00:32:23,733
It's like, if it's not changing
behavior, then is it worth anything?

587
00:32:24,073 --> 00:32:26,143
so it's exciting to see some
real numbers come out of this

588
00:32:26,193 --> 00:32:26,483
Laura Tacho: Yeah.

589
00:32:26,509 --> 00:32:29,549
Bret: showing us that, yeah,
this is like any new technology.

590
00:32:29,549 --> 00:32:31,859
We have to figure out where
it works and where it doesn't.

591
00:32:31,859 --> 00:32:33,919
It's not universal, it's not a panacea.

592
00:32:34,099 --> 00:32:35,149
so yeah, let's jump in.

593
00:32:35,373 --> 00:32:35,793
Laura Tacho: Yeah.

594
00:32:36,503 --> 00:32:39,203
you say that so eloquently,
Bret, like if it's not changing

595
00:32:39,203 --> 00:32:40,913
behavior, is it even working?

596
00:32:40,913 --> 00:32:41,663
what's the point?

597
00:32:41,693 --> 00:32:45,563
Absolutely, and that's why I get up
every day is to like the companies that

598
00:32:45,563 --> 00:32:49,493
you work with that don't have this,
you know, trying to get them, giving

599
00:32:49,493 --> 00:32:52,643
them tools and resources, enablement
so that they can have this because

600
00:32:52,853 --> 00:32:54,803
you don't need crazy ETL pipelines.

601
00:32:55,773 --> 00:32:59,943
to have information like this, you can
get so far with self-reported data from a

602
00:32:59,943 --> 00:33:04,153
survey that you run in Google forms, it's
remarkable how well those methods work,

603
00:33:04,183 --> 00:33:08,933
even at scale, the overhead does get,
cumbersome at some point, but you don't

604
00:33:08,933 --> 00:33:10,883
need to wait for perfect ETL pipelines.

605
00:33:11,213 --> 00:33:15,173
Anyway, we looked at 180 different
companies and talked to developers

606
00:33:15,173 --> 00:33:18,863
who were saving the most time with AI.

607
00:33:18,903 --> 00:33:21,123
I just asked them like,
what, what are you doing?

608
00:33:21,153 --> 00:33:26,223
stack trace analysis and refactoring
existing code were the top two

609
00:33:26,223 --> 00:33:28,983
use cases that we're saving the
most time mid loop generation.

610
00:33:28,983 --> 00:33:32,103
So like, code gen didn't
come in until number three.

611
00:33:32,103 --> 00:33:34,623
And then we have test case
generation that's also up there.

612
00:33:34,623 --> 00:33:37,233
Like those top four are kind
of the biggest time savers.

613
00:33:37,653 --> 00:33:39,253
Actually, I think this is from AWS Nirmal.

614
00:33:39,273 --> 00:33:41,463
We talked about this at
KubeCon when we were in London.

615
00:33:41,533 --> 00:33:46,153
The average developer at AWS only spends
how much time of their week coding?

616
00:33:46,983 --> 00:33:49,553
it's double digits but it's
in the low double digits.

617
00:33:49,588 --> 00:33:53,908
that's very consistent with what we
see across the industry that developers

618
00:33:53,908 --> 00:34:00,238
are spending about 25, 20 ish to
30% of their time actually coding.

619
00:34:00,548 --> 00:34:03,448
a lot of it is planning, doing
maintenance work, doing other stuff.

620
00:34:03,448 --> 00:34:07,328
So, applying AI to other tasks, yeah.

621
00:34:07,658 --> 00:34:08,958
Nirmal Mehta: yeah, I did find it.

622
00:34:09,038 --> 00:34:15,868
AWS published in December 3rd, 2024, a
post that said that Amazon developers,

623
00:34:16,184 --> 00:34:20,114
spend most of their time on tedious
undifferentiated tasks, such as learning

624
00:34:20,114 --> 00:34:23,714
code bases, writing and reviewing
documentation, testing, managing

625
00:34:23,714 --> 00:34:27,374
deployments, troubleshooting issues,
or finding and fixing vulnerabilities.

626
00:34:29,064 --> 00:34:32,874
the stat from the article is no
longer in that post, but it was,

627
00:34:32,924 --> 00:34:37,034
developers reporting, spending an
average of just one hour per day.

628
00:34:37,344 --> 00:34:39,984
So this was at reinvent 2024 in the

629
00:34:39,985 --> 00:34:40,275
Bret: Yeah.

630
00:34:40,464 --> 00:34:44,914
Nirmal Mehta: by Matt Garmin, where he
said that they were spending on average

631
00:34:44,914 --> 00:34:47,616
one hour a day on actual code, development

632
00:34:48,066 --> 00:34:48,726
Laura Tacho: absolutely.

633
00:34:48,805 --> 00:34:48,865
Yeah.

634
00:34:48,986 --> 00:34:49,556
Bret: lines.

635
00:34:49,656 --> 00:34:50,046
Nirmal Mehta: Yeah,

636
00:34:50,142 --> 00:34:51,882
Laura Tacho: that was talking
about what are some of the use

637
00:34:51,882 --> 00:34:53,822
cases that can bring the most gain?

638
00:34:54,092 --> 00:34:58,182
I wanna show you this other thing,
which is really, this is like

639
00:34:58,182 --> 00:35:00,562
aggregated, kind of dummy data.

640
00:35:00,562 --> 00:35:03,532
This isn't data from any customer,
so I would never do that.

641
00:35:03,832 --> 00:35:06,372
but the spirit of this is accurate.

642
00:35:06,422 --> 00:35:09,752
and I've actually seen way more
pronounced versions of this,

643
00:35:09,752 --> 00:35:10,832
but this is looking in DX.

644
00:35:10,832 --> 00:35:13,902
So, we do quite a lot of,
visualizations to help you

645
00:35:13,902 --> 00:35:15,912
analyze the data in easier ways.

646
00:35:15,912 --> 00:35:17,112
But when you look at.

647
00:35:17,652 --> 00:35:21,552
Why is it that a developer can only
spend one hour a day writing code?

648
00:35:21,852 --> 00:35:25,812
It's that they are in meetings all
the time, having interruptions.

649
00:35:25,892 --> 00:35:30,602
they are waiting on CI, they're trying to
navigate through code bases Nirmal, like

650
00:35:30,632 --> 00:35:32,462
literally exactly what you just said.

651
00:35:32,672 --> 00:35:36,002
We see this reflected in
the data and way down here.

652
00:35:36,282 --> 00:35:40,372
it's not always this low for every
company, but when we look at AI time

653
00:35:40,372 --> 00:35:46,822
savings right now, which is median, like
three hours, 45 minutes per developer

654
00:35:46,822 --> 00:35:50,962
per week, and then we look at how much
time they could be saving by reducing

655
00:35:50,962 --> 00:35:55,372
dev environment toil, by having better
documentation, by reducing meetings.

656
00:35:56,127 --> 00:36:01,172
AI doesn't even cover a 10th of what they
could be saving if the other parts of the

657
00:36:01,172 --> 00:36:02,762
developer experience were also increased.

658
00:36:02,802 --> 00:36:04,172
I think speaking to your kind of.

659
00:36:04,992 --> 00:36:08,112
Deep seated fear, maybe about
like the over promise of AI.

660
00:36:08,112 --> 00:36:11,952
I think one of the ways that AI has
been overpromised is that it is a silver

661
00:36:11,952 --> 00:36:17,062
bullet that is gonna solve many problems
when in fact developer experience is

662
00:36:17,062 --> 00:36:23,962
still a way bigger lever for almost every
company than AI assisted engineering

663
00:36:23,962 --> 00:36:25,492
at least at this point in time.

664
00:36:25,862 --> 00:36:27,027
Nirmal Mehta: That's super interesting.

665
00:36:27,027 --> 00:36:31,617
So another way to interpret that
data, from what I'm looking at is

666
00:36:31,617 --> 00:36:38,657
that, the traditional DevOps, like the
DevOps maturity of the organization,

667
00:36:38,777 --> 00:36:43,367
CI/CD automation becomes even more.

668
00:36:43,427 --> 00:36:44,597
It was already critical

669
00:36:44,997 --> 00:36:45,417
Laura Tacho: Mm-hmm.

670
00:36:45,677 --> 00:36:46,247
Nirmal Mehta: competitive

671
00:36:46,383 --> 00:36:46,803
Bret: Mm-hmm.

672
00:36:46,997 --> 00:36:54,557
Nirmal Mehta: and value this AI world
get the value out of these AI adoption.

673
00:36:54,962 --> 00:36:56,912
It becomes even more critical.

674
00:36:56,912 --> 00:37:00,122
you have to have that automation in place.

675
00:37:01,022 --> 00:37:02,572
and, that's what I'm seeing here, right?

676
00:37:02,602 --> 00:37:08,642
CI, wait time, deployment, lead time,
those are, that's just pure CI/CD, right?

677
00:37:08,692 --> 00:37:08,982
Laura Tacho: Yeah.

678
00:37:09,327 --> 00:37:09,957
Nirmal Mehta: automation.

679
00:37:09,957 --> 00:37:13,107
so that, that tooling
needs to be in place.

680
00:37:13,107 --> 00:37:18,997
So you think that, folks that are
working on are, are, are pushing for AI,

681
00:37:19,047 --> 00:37:21,597
tooling adoption with their developers?

682
00:37:23,037 --> 00:37:29,157
net benefit is that, they'll, they'll
drag themselves into modern DevOps

683
00:37:29,577 --> 00:37:34,477
because they need to do that first, to
create the foundation for getting the

684
00:37:34,477 --> 00:37:36,697
value out of adopting these AI tools.

685
00:37:37,117 --> 00:37:37,747
Is that a fair

686
00:37:37,802 --> 00:37:38,222
Laura Tacho: Yeah.

687
00:37:38,322 --> 00:37:38,772
Nirmal Mehta: See that

688
00:37:39,646 --> 00:37:41,011
Laura Tacho: that is a
really fair way to see that.

689
00:37:41,061 --> 00:37:42,411
I posted about this before.

690
00:37:42,411 --> 00:37:44,471
Here's, a post from Abi.

691
00:37:44,835 --> 00:37:49,945
I think what you're describing
Nirmal is that like DevEx is AgentEx.

692
00:37:49,965 --> 00:37:51,555
I think there's, you know,
there's some subtleties.

693
00:37:51,555 --> 00:37:54,685
I think what you're saying is, maybe
by using AI these companies can

694
00:37:54,685 --> 00:37:57,955
drag themselves into the future,
which is great because the reality

695
00:37:57,955 --> 00:38:01,255
is that a lot of what will benefit.

696
00:38:01,735 --> 00:38:05,415
Agentic models, agentic workflows,
good documentation, organized

697
00:38:05,415 --> 00:38:08,265
code, fast feedback loops,
clear project requirements.

698
00:38:08,535 --> 00:38:10,545
Those are things that also benefit people.

699
00:38:11,115 --> 00:38:16,455
And it's too bad that a lot of
companies are only willing to invest

700
00:38:16,455 --> 00:38:21,195
in those things for a robot and not
for the people that were there before.

701
00:38:21,435 --> 00:38:22,185
But

702
00:38:24,645 --> 00:38:30,795
our opportunity now is that wallets are
open and the end result is the same.

703
00:38:31,185 --> 00:38:36,345
And so if the robot and pleasing the
robot is what's gonna get companies to

704
00:38:36,345 --> 00:38:40,635
spend money on these things that are
existential and critical and important,

705
00:38:40,635 --> 00:38:43,785
then you know what, I'm fine with that.

706
00:38:43,965 --> 00:38:47,275
Let's keep the wallets open and
let's keep moving into the future.

707
00:38:47,545 --> 00:38:47,935
Bret: The reason

708
00:38:47,958 --> 00:38:49,368
is less important than the outcome.

709
00:38:49,418 --> 00:38:49,628
Laura Tacho: Yeah.

710
00:38:49,854 --> 00:38:55,224
Bret: uh, mean, to be fair, we weren't
all pitching better documentation

711
00:38:55,324 --> 00:38:57,544
as a business productivity boost.

712
00:38:57,544 --> 00:39:00,094
that's never how I positioned it or
any other developer I worked with.

713
00:39:00,144 --> 00:39:00,434
Laura Tacho: Yeah.

714
00:39:00,550 --> 00:39:03,520
Bret: again, like we're going back to
this, that the hype cycle is promising

715
00:39:03,520 --> 00:39:08,620
AI will make business faster, it makes
us write code faster, supposedly.

716
00:39:08,800 --> 00:39:13,360
Therefore, business goals
will be met faster.

717
00:39:13,360 --> 00:39:17,290
And I mean, this podcast, we, we
actually like to think of this

718
00:39:17,290 --> 00:39:23,740
podcast as focusing on AI after the
commit, where so many other podcasts

719
00:39:23,740 --> 00:39:24,940
are focused on before the commit.

720
00:39:24,940 --> 00:39:27,280
That's why we don't talk
about which model we're using.

721
00:39:27,280 --> 00:39:28,210
And if we do

722
00:39:28,329 --> 00:39:28,599
Laura Tacho: Yeah.

723
00:39:28,840 --> 00:39:30,790
Bret: dev tooling, we're
usually just talking about

724
00:39:30,790 --> 00:39:32,080
it because we're all into it.

725
00:39:32,080 --> 00:39:36,950
It's fun, not because it's the focus
of an episode, we've had, other guests

726
00:39:36,950 --> 00:39:40,400
on this show and our, and Nirmal and
i's other show, DevOps and Docker talk.

727
00:39:40,760 --> 00:39:44,590
We've had multiple shows now over the
last four, eight months talking about

728
00:39:44,590 --> 00:39:49,690
the idea that AI is going to improve.

729
00:39:50,270 --> 00:39:55,370
the standardization and expectations
of teams when it comes to the rigor

730
00:39:55,610 --> 00:40:00,080
of everything from documentation
to automation, to standardizing

731
00:40:00,080 --> 00:40:03,320
on infrastructure as code, like
all the things that we've all

732
00:40:03,320 --> 00:40:05,150
been focusing on for over a decade

733
00:40:05,160 --> 00:40:05,580
Laura Tacho: Mm-hmm.

734
00:40:05,691 --> 00:40:07,651
Bret: to others and share our opinions.

735
00:40:07,891 --> 00:40:11,821
But we didn't necessarily go at it with
everyone knowing that, this is gonna save

736
00:40:11,821 --> 00:40:13,501
us money, this is gonna save us time.

737
00:40:13,591 --> 00:40:17,751
And I feel like, you're right, that
AI, I guess the negative is it's

738
00:40:17,751 --> 00:40:20,721
hyped, but the positive is we might be
able to use that hype to actually get

739
00:40:20,721 --> 00:40:24,111
business to move on some of these things
that we've been trying for so long.

740
00:40:24,411 --> 00:40:29,171
Because if I go to the team and
say, Laura's new project shows, oh,

741
00:40:29,181 --> 00:40:30,381
look at this new talk for Laura.

742
00:40:30,591 --> 00:40:36,261
It shows the stats that say, if we have
better confluence, if we have better in

743
00:40:36,261 --> 00:40:40,201
repo documentation, if we have better
project planning that's linked into

744
00:40:40,201 --> 00:40:44,121
other things, our AIs will actually
perform better for us and save us money.

745
00:40:44,151 --> 00:40:46,221
Maybe that's what will
actually move the target.

746
00:40:46,221 --> 00:40:47,031
So you heard it

747
00:40:47,045 --> 00:40:47,335
Laura Tacho: Yeah.

748
00:40:48,021 --> 00:40:50,571
Bret: just go spin that
to your leadership.

749
00:40:50,661 --> 00:40:55,991
maybe you can get more, AI tooling
and more of the, stuff that

750
00:40:55,991 --> 00:40:57,041
you actually wanted to work on.

751
00:40:57,041 --> 00:40:58,871
we need more documentation,
but I don't know anybody

752
00:40:58,871 --> 00:41:00,191
that's clamoring to make more.

753
00:41:00,581 --> 00:41:04,721
So I'm excited that AI might
help me write it better and keep

754
00:41:04,735 --> 00:41:05,490
Laura Tacho: Yeah, yeah,

755
00:41:05,651 --> 00:41:07,331
Bret: find the outdated stuff better.

756
00:41:07,361 --> 00:41:09,031
there's many opportunities
for improving it.

757
00:41:09,920 --> 00:41:10,160
Laura Tacho: yeah.

758
00:41:10,160 --> 00:41:15,180
I think documentation is a really good
use case for AI in a way that saves.

759
00:41:15,760 --> 00:41:20,320
Time as a second order consequence,
like a second order indirect effect.

760
00:41:20,530 --> 00:41:24,830
It's harder to estimate because,
the reason that documentation

761
00:41:24,830 --> 00:41:27,170
has never been, it's always
been like a nice to have, right?

762
00:41:27,170 --> 00:41:28,430
It's been a vitamin.

763
00:41:28,680 --> 00:41:29,730
not a painkiller.

764
00:41:29,730 --> 00:41:33,340
And I think that's been the experience
in general of developer experience

765
00:41:33,390 --> 00:41:35,850
It's something you do to take care
of yourself because you're good.

766
00:41:35,850 --> 00:41:38,970
It's not like my arm is cut
off and need to bandage it.

767
00:41:39,170 --> 00:41:41,240
in reality that's what
developer experience is.

768
00:41:41,240 --> 00:41:46,530
It's that your arm is cut off because
on average, developers waste 20% of

769
00:41:46,645 --> 00:41:49,850
their time each week because they
can't find the right documentation.

770
00:41:49,850 --> 00:41:52,730
They're waiting for builds to finish,
they're waiting for all this stuff.

771
00:41:52,730 --> 00:41:58,010
20% of time, if you can imagine a roofer
putting shingles on your house, throwing

772
00:41:58,080 --> 00:42:00,120
one out of every five shingles down.

773
00:42:00,525 --> 00:42:03,345
to the ground or one out
of five nails did not work.

774
00:42:03,465 --> 00:42:07,425
That company would go out of business
and no business leader would tolerate it.

775
00:42:07,635 --> 00:42:11,985
But because it's hidden in knowledge work
and behind developers and we can kind

776
00:42:11,985 --> 00:42:15,925
of brute force our way through it, we
don't know the boundaries of the problem.

777
00:42:15,925 --> 00:42:17,035
And that makes it squishy.

778
00:42:17,035 --> 00:42:20,725
It makes it seem like a vitamin of oh,
give the developers nicer documentation.

779
00:42:20,725 --> 00:42:24,775
They'll like that when actually,
if you were to measure and, and use

780
00:42:24,775 --> 00:42:28,855
some, you know, measurements like we,
we do in DX and try to identify how

781
00:42:28,855 --> 00:42:31,735
many times per week does a developer
lose 30 minutes or more because

782
00:42:31,735 --> 00:42:35,335
they can't find information due to
outdated or non-existent documentation.

783
00:42:35,435 --> 00:42:36,545
look at that number now.

784
00:42:36,575 --> 00:42:39,065
That number is your arm cut off?

785
00:42:39,125 --> 00:42:39,335
Right?

786
00:42:39,335 --> 00:42:40,685
That's not a vitamin anymore.

787
00:42:40,925 --> 00:42:43,685
there is a bit of marketing around
developer experience and a lot

788
00:42:43,685 --> 00:42:46,175
of talking about what it is.

789
00:42:46,175 --> 00:42:47,585
'cause it's not ping pong and beer.

790
00:42:47,945 --> 00:42:49,805
This is like existential problem.

791
00:42:49,805 --> 00:42:54,845
And I know that we all know this and
that's why we have been working on

792
00:42:54,845 --> 00:42:58,355
DevOps and developer tooling because
we know the real business impact of

793
00:42:58,355 --> 00:43:04,105
it, but it has been historically really
hard to tie that to Dollars and cents,

794
00:43:04,175 --> 00:43:07,805
which is what the business needs in
order to keep their wallets open.

795
00:43:08,025 --> 00:43:11,865
I think AI, if anything, as you said,
it's overhyped or, hyped the right amount.

796
00:43:11,865 --> 00:43:15,035
I don't know how much hype is the right
amount of hype, but, people sure have

797
00:43:15,035 --> 00:43:17,965
their wallets open and maybe now's
the time to, take advantage of that.

798
00:43:18,433 --> 00:43:21,673
Bret: I was just gonna reiterate and
repeat a little bit back to Laura,

799
00:43:21,673 --> 00:43:25,863
that, in my entire career, the three
things on every project that get cut

800
00:43:25,863 --> 00:43:29,133
first, and I always like to say this in
project meetings when we're all about

801
00:43:29,133 --> 00:43:33,903
to cut something, it's documentation
monitoring and disaster recovery testing.

802
00:43:33,903 --> 00:43:35,553
Like those are the, or just testing.

803
00:43:35,973 --> 00:43:38,593
Those are the three things I see
on every project that left, on the

804
00:43:38,593 --> 00:43:42,313
cutting room floor when the budgets are
tight or the timeline didn't get met.

805
00:43:42,443 --> 00:43:44,813
if AI could improve,
just those three things,

806
00:43:44,813 --> 00:43:47,163
Documentation, monitoring and testing.

807
00:43:47,553 --> 00:43:51,543
If we could just improve those three
things, I feel like a bigger portion of

808
00:43:51,543 --> 00:43:54,693
that graph that you were showing, and
for those that weren't seeing it, it

809
00:43:54,693 --> 00:43:58,130
was a line graph of all the different
places that devs spend their time.

810
00:43:58,263 --> 00:43:59,243
And at the very bottom

811
00:43:59,332 --> 00:44:00,262
code generation.

812
00:44:00,292 --> 00:44:02,902
Like the very bottom of it was ba pretty

813
00:44:03,006 --> 00:44:03,366
Laura Tacho: Yeah.

814
00:44:03,602 --> 00:44:04,682
Nirmal Mehta: so interesting.

815
00:44:04,682 --> 00:44:04,802
It's

816
00:44:04,972 --> 00:44:05,092
Laura Tacho: I.

817
00:44:05,132 --> 00:44:09,795
Nirmal Mehta: like a, an outcome of this
data is that, if you're out there, if

818
00:44:09,795 --> 00:44:13,665
you're listening to the show and you're
a dev or a platform engineer, engineering

819
00:44:13,665 --> 00:44:18,792
team, and you're in your sprint, retro,
and you're looking at your backlog, if you

820
00:44:18,792 --> 00:44:22,632
scroll all the way down, there's probably
a line that says, improve documentation

821
00:44:22,962 --> 00:44:24,672
that has never made its way up.

822
00:44:24,672 --> 00:44:28,662
Or, you know, refactor this
critical little piece of code

823
00:44:28,712 --> 00:44:28,982
Bret: Yep.

824
00:44:29,181 --> 00:44:31,161
Nirmal Mehta: improves the speed of CI/CD

825
00:44:31,381 --> 00:44:32,521
Bret: Increased test coverage.

826
00:44:32,601 --> 00:44:35,301
Nirmal Mehta: into, or,
one of the monitoring ones.

827
00:44:35,661 --> 00:44:38,941
There's probably these, These
user stories that have been on

828
00:44:38,941 --> 00:44:41,581
the backlog at the bottom You just
gotta scroll all the way down.

829
00:44:41,993 --> 00:44:45,413
It seems like you, and, and if your
organization, if you're sitting there

830
00:44:45,413 --> 00:44:47,543
and you're sprint retro and you're
figuring out, you're going through

831
00:44:47,543 --> 00:44:52,343
your backlog and planning for your
next sprint, and your organization is

832
00:44:52,343 --> 00:44:58,903
asking you to adopt AI tools, to, show
an outcome of 30% plus productivity

833
00:44:58,903 --> 00:45:00,553
gains with using these tools, right?

834
00:45:00,553 --> 00:45:06,343
Regardless of how you're gonna use the
tool, you might take this opportunity

835
00:45:06,343 --> 00:45:10,993
to take a look at those items on
your backlog and push 'em up into the

836
00:45:10,993 --> 00:45:14,953
new sprint because you can now say
that those are tied with this data.

837
00:45:14,953 --> 00:45:19,873
You can show that those, backlog
items, those forever backlog items

838
00:45:20,233 --> 00:45:25,813
are valuable in actually seeing those
productivity gains from these AI tools.

839
00:45:25,813 --> 00:45:28,393
Is that the right interpretation?

840
00:45:28,583 --> 00:45:29,873
is that what you're saying, Laura?

841
00:45:29,993 --> 00:45:32,603
Laura Tacho: I think that is,
a really good interpretation.

842
00:45:32,603 --> 00:45:38,003
I think where things get tricky is
building the business case around them,

843
00:45:38,103 --> 00:45:38,393
Nirmal Mehta: Okay.

844
00:45:38,838 --> 00:45:41,988
Laura Tacho: it's, really been
difficult for developers to

845
00:45:41,988 --> 00:45:44,718
say, sorry, we're not gonna do.

846
00:45:45,803 --> 00:45:52,153
X so that we can work on our documentation
increase test coverage build out different

847
00:45:52,153 --> 00:45:58,243
test cases, work on our CI to reduce
flaky tests or make it run 15% faster.

848
00:45:58,573 --> 00:46:01,443
It's been really difficult 'cause
this shouldn't be extra work, right?

849
00:46:01,443 --> 00:46:03,663
we don't wanna overload
devs with extra work.

850
00:46:03,693 --> 00:46:05,043
We need to take something away.

851
00:46:05,093 --> 00:46:08,933
That's where things get really tricky
because when we have a feature that

852
00:46:08,933 --> 00:46:13,703
we know is gonna bring in, you know,
we have a, it's $500,000, let's just

853
00:46:13,703 --> 00:46:20,573
say that I can't hold up documentation
improvements or, or CI/CD improvements

854
00:46:20,603 --> 00:46:26,963
next to $500,000 and make it as close
to an apple's to apple's comparison.

855
00:46:27,593 --> 00:46:31,703
It's so difficult to do that
without data because we can't.

856
00:46:32,243 --> 00:46:36,653
conceptualize how much money
that is costing our company.

857
00:46:36,953 --> 00:46:42,203
And so often what we end up is like,
sure, that feature might bring $500,000

858
00:46:42,203 --> 00:46:46,313
of new revenue in the door, but this is
gonna allow you to accelerate delivery

859
00:46:46,313 --> 00:46:49,073
for everything else in the future by 20%.

860
00:46:49,073 --> 00:46:53,423
And how much is that worth in terms of
salary, cost avoidance, and time to market

861
00:46:53,633 --> 00:46:55,583
realization for your future investments.

862
00:46:55,583 --> 00:46:57,233
we can do the math and figure that out.

863
00:46:57,543 --> 00:47:01,983
but that's where platform engineering
teams, where software development

864
00:47:01,983 --> 00:47:05,553
teams, like app development teams
are having the hardest time right

865
00:47:05,553 --> 00:47:07,233
now is making those arguments.

866
00:47:07,233 --> 00:47:11,313
And that's where DX, I mean that's our
really, our backbone is like drawing

867
00:47:11,313 --> 00:47:16,629
the boundaries around how much it's
costing you to have these problems.

868
00:47:16,729 --> 00:47:19,159
Helping you figure out how to
prioritize them so that you can

869
00:47:19,159 --> 00:47:23,509
reduce the friction and stop
wasting 20% of your time every week.

870
00:47:23,929 --> 00:47:26,779
'cause that adds up really
fast and it's just not fun.

871
00:47:26,942 --> 00:47:30,188
So, why don't we, why don't
we talk more about AI stuff?

872
00:47:30,628 --> 00:47:33,598
Nirmal Mehta: Yes, assistive
versus agentic was another

873
00:47:33,598 --> 00:47:34,558
thing, but Bret, go ahead.

874
00:47:34,739 --> 00:47:35,452
Bret: Yep, So

875
00:47:35,766 --> 00:47:36,306
Laura Tacho: Um,

876
00:47:36,442 --> 00:47:36,982
Bret: agentic.

877
00:47:36,982 --> 00:47:37,912
What does that even mean?

878
00:47:38,364 --> 00:47:39,294
Laura Tacho: what does that even mean?

879
00:47:39,294 --> 00:47:41,244
So I think that.

880
00:47:42,474 --> 00:47:46,134
When we talk about AI or like when AI is
talked about in general, it's kind of like

881
00:47:46,134 --> 00:47:48,894
this big pool AI is such a broad term.

882
00:47:48,994 --> 00:47:52,834
even thinking about those use cases that
I talked about AI for code gen versus

883
00:47:52,834 --> 00:47:58,746
AI for refactoring code versus AI for
running after the commit pipelines,

884
00:47:59,016 --> 00:48:00,846
that's already a huge variation.

885
00:48:00,901 --> 00:48:04,296
And then when we talk about the
interaction modalities of AI, we've

886
00:48:04,296 --> 00:48:08,556
got, am I talking to ChatGPT in a
separate window asking it like, oh,

887
00:48:08,556 --> 00:48:11,406
how would you plan, this feature?

888
00:48:11,406 --> 00:48:14,496
Like what are the questions I'm missing
If I'm gonna do a pen test on this?

889
00:48:14,496 --> 00:48:17,456
Like what are some edge cases that
I might have missed versus code

890
00:48:17,456 --> 00:48:23,636
completion in the IDE versus agentic
workflows that need human approval

891
00:48:23,636 --> 00:48:26,636
versus ones that are running in swarm
mode with many agents doing work.

892
00:48:26,786 --> 00:48:28,226
AI is such a huge term.

893
00:48:28,436 --> 00:48:32,436
And so when we talk about The shift in
the spectrum from assisted to agentic.

894
00:48:32,436 --> 00:48:38,106
We're talking about human in the loop
ness, maybe I would call it like that.

895
00:48:38,106 --> 00:48:42,216
with assisted engineering,
we've got chat modality, we've

896
00:48:42,216 --> 00:48:43,656
got auto complete in the IDE.

897
00:48:44,656 --> 00:48:49,186
as we get closer to agentic, maybe
we're relying more on workflows that

898
00:48:49,186 --> 00:48:55,456
are defined, and running, you have an
agent doing each of the workflow for

899
00:48:55,456 --> 00:48:59,656
you, but you're still saying, yes,
do this and verifying and then saying

900
00:48:59,656 --> 00:49:03,196
proceed and verifying and proceeding
very much human in the loop versus

901
00:49:03,196 --> 00:49:08,776
like a totally agentic workflow, which
is like, I need an app that is going

902
00:49:08,776 --> 00:49:13,967
to analyze my Pokemon card collection
tell me which one has the highest market

903
00:49:13,967 --> 00:49:17,087
value and send me a push notification
every morning with the highest value

904
00:49:17,087 --> 00:49:20,477
card, build it and deploy it to AWS.

905
00:49:21,302 --> 00:49:23,402
And you don't look, you know,
you're not in the loop at all.

906
00:49:23,432 --> 00:49:27,632
that's a very big spectrum versus
like pair programming with ChatGPT to

907
00:49:27,642 --> 00:49:30,342
giving something production access.

908
00:49:30,462 --> 00:49:34,722
So as an industry, if I were to imagine
all the companies I'm working with and

909
00:49:34,722 --> 00:49:38,502
see what they're doing and put them
on a scatterplot, we're definitely

910
00:49:38,502 --> 00:49:44,052
more toward the assisted with a
little bit of dabbling in the agentic.

911
00:49:44,082 --> 00:49:48,122
But I would say more like workflow
is starting to be that early adopter,

912
00:49:48,122 --> 00:49:50,252
early majority kind of bleeding edge.

913
00:49:50,382 --> 00:49:54,482
I've seen some interesting examples
of agentic, but nowhere near industry

914
00:49:54,482 --> 00:49:59,845
wide adoption of ag agentic models,
or patterns of working right now

915
00:49:59,891 --> 00:50:00,101
Bret: That

916
00:50:00,145 --> 00:50:01,525
Laura Tacho: September 1st, 2025.

917
00:50:01,922 --> 00:50:02,762
Bret: yeah,

918
00:50:03,006 --> 00:50:07,356
Nirmal Mehta: because Bret constantly
is asking me, but Nirmal, our, our

919
00:50:07,356 --> 00:50:11,616
companies and customers actually
using these agents to do like SRE

920
00:50:11,616 --> 00:50:13,686
tasks or operations or DevOps.

921
00:50:13,686 --> 00:50:17,226
Like are they, where's
the actual use cases?

922
00:50:17,231 --> 00:50:18,306
And I,

923
00:50:18,422 --> 00:50:19,532
Bret: me the money is, yeah.

924
00:50:19,712 --> 00:50:20,042
Show me

925
00:50:20,086 --> 00:50:21,376
Laura Tacho: Show me the money.

926
00:50:21,526 --> 00:50:24,456
Nirmal Mehta: I'm gonna deflect Bret's
question that he's constantly asking

927
00:50:24,456 --> 00:50:28,310
me to you, Laura, I know you just
mentioned that, on that scatter plot.

928
00:50:28,575 --> 00:50:34,225
it's definitely not the majority, but are
you seeing some early signs of, agentic

929
00:50:34,225 --> 00:50:39,695
workflows, not only for business use
cases, but also for DevOps, tasks like

930
00:50:39,695 --> 00:50:43,941
operations, SRE, CI/CD, things like that

931
00:50:44,345 --> 00:50:44,825
Laura Tacho: yes.

932
00:50:44,825 --> 00:50:47,285
I would say as a short answer, yes.

933
00:50:47,585 --> 00:50:51,465
I think the complexity is the
variable influencing, how and to

934
00:50:51,465 --> 00:50:53,055
what extent that's being used.

935
00:50:53,105 --> 00:50:57,315
looking at SWE-bench, this hasn't changed
since I, looked at it recently, but we've

936
00:50:57,315 --> 00:51:03,510
got 67% of tasks resolved and a lot of
these models are hovering in like the 55%.

937
00:51:03,510 --> 00:51:07,065
So like the technology
is also just like not.

938
00:51:07,815 --> 00:51:08,445
There.

939
00:51:08,475 --> 00:51:11,525
So this is, for those of you that don't
know about this leaderboard, this is

940
00:51:11,525 --> 00:51:16,115
like testing the model's capability
to complete a task end to end and

941
00:51:16,115 --> 00:51:18,755
have it be correct a coding task.

942
00:51:18,801 --> 00:51:19,101
Bret: Yeah.

943
00:51:19,120 --> 00:51:19,360
Nirmal Mehta: one

944
00:51:19,595 --> 00:51:19,765
Laura Tacho: um,

945
00:51:19,815 --> 00:51:20,985
Nirmal Mehta: or is it through a workflow

946
00:51:21,387 --> 00:51:21,537
Bret: It

947
00:51:21,641 --> 00:51:22,486
Laura Tacho: um, Ooh.

948
00:51:22,497 --> 00:51:23,067
Bret: tool.

949
00:51:23,097 --> 00:51:25,312
if you look at the list,
each one might be like.

950
00:51:26,457 --> 00:51:29,877
Just this model or this model
plus some additional tooling

951
00:51:29,877 --> 00:51:31,587
that turns it, like warp

952
00:51:31,721 --> 00:51:32,081
Laura Tacho: Yeah.

953
00:51:33,027 --> 00:51:36,447
Bret: You know that their agent model
that's using a particular LLM like

954
00:51:36,447 --> 00:51:39,418
sonnet four gets this percentage.

955
00:51:39,418 --> 00:51:45,058
So just to clarify, when Laura says 70%,
which she's saying is that that particular

956
00:51:45,058 --> 00:51:48,628
setup, like that LLM plus that tool

957
00:51:49,117 --> 00:51:49,537
Laura Tacho: Mm-hmm.

958
00:51:50,128 --> 00:51:54,828
Bret: 70% success rate on hundreds,
potentially of GitHub issues

959
00:51:54,978 --> 00:51:57,648
that it's trying to solve for
a particular software project.

960
00:51:58,038 --> 00:52:00,678
And that's what we've talked about
before we've actually mentioned this

961
00:52:00,678 --> 00:52:01,668
is I'm so glad you brought it up.

962
00:52:01,668 --> 00:52:02,028
Because

963
00:52:02,327 --> 00:52:02,747
Laura Tacho: Mm-hmm.

964
00:52:02,838 --> 00:52:09,232
Bret: best models are only getting three
out of four or two out of three PRs

965
00:52:09,282 --> 00:52:09,792
Laura Tacho: yeah.

966
00:52:09,858 --> 00:52:10,488
Bret: Correct.

967
00:52:10,758 --> 00:52:11,048
Laura Tacho: Yeah.

968
00:52:11,229 --> 00:52:12,103
Bret: it's, it's not fixing.

969
00:52:12,163 --> 00:52:13,873
And so, and that, and a lot of us.

970
00:52:14,267 --> 00:52:15,407
we are all very lucky.

971
00:52:15,527 --> 00:52:20,147
We have access to the best
models, to the best tooling.

972
00:52:20,247 --> 00:52:23,067
we live in a western country where
we can afford all these things.

973
00:52:23,347 --> 00:52:26,897
I do hear on occasion feedback that
a lot of the world isn't here yet.

974
00:52:27,047 --> 00:52:30,067
A lot of the world has to use either
free models or local models, or

975
00:52:30,067 --> 00:52:33,667
free stuff like the free ChatGPT
equivalent or whatever tokens they

976
00:52:33,667 --> 00:52:35,827
get for free with GitHub Copilot.

977
00:52:35,857 --> 00:52:39,487
And they don't necessarily have the
best tools unless their company buys

978
00:52:39,487 --> 00:52:40,997
them because they're just so expensive.

979
00:52:40,997 --> 00:52:45,617
Unless you live in a, in a country that
is indexed to the US dollar, essentially

980
00:52:46,411 --> 00:52:49,721
Laura Tacho: So I think the
technical capability isn't really

981
00:52:49,721 --> 00:52:53,531
there for organizations to rely
on agentic workflows for things

982
00:52:53,531 --> 00:52:54,821
that are gonna touch production.

983
00:52:55,091 --> 00:52:58,961
Have I seen lots of experimentation
with agentic workflows for stuff used

984
00:52:58,961 --> 00:53:02,471
internally and doesn't necessarily
hit end users Absolutely, definitely.

985
00:53:02,781 --> 00:53:05,651
there's a lot of experimentation,
but when we talk about it widespread,

986
00:53:05,651 --> 00:53:10,149
especially in the enterprise, we're
still away ways away, from that being

987
00:53:10,199 --> 00:53:16,123
reality for, for a lot of folks, I
would say like one step back toward

988
00:53:16,123 --> 00:53:19,799
assist, you know, one step away from
Ag agentic, which is like workflows.

989
00:53:19,959 --> 00:53:24,939
I have seen many kind of enterprise,
fast scaling, high growth organizations

990
00:53:24,939 --> 00:53:30,319
have their platform team responsible
for creating workflows that, for

991
00:53:30,319 --> 00:53:34,969
example, do legacy modernization or
dependency updates and make those

992
00:53:34,969 --> 00:53:40,129
available to other development teams,
their customer developers internally

993
00:53:40,609 --> 00:53:42,979
as part of their platform offering.

994
00:53:43,009 --> 00:53:44,509
So this is not quite agentic.

995
00:53:44,509 --> 00:53:47,809
There's still more human in the loop,
there's more human verification.

996
00:53:47,939 --> 00:53:51,689
but companies that are really
investing in AI have done a lot

997
00:53:51,689 --> 00:53:53,099
of training and enablement to.

998
00:53:53,759 --> 00:53:57,779
Get all of their developers with not
just an understanding of like how

999
00:53:57,779 --> 00:54:00,659
the, the tools are used, but like
organizational understanding of how

1000
00:54:00,659 --> 00:54:02,489
to apply it to their business context.

1001
00:54:02,709 --> 00:54:07,929
they are experimenting with, you
know, agentic adjacent, we'll call

1002
00:54:07,929 --> 00:54:09,919
it that, agentic adjacent, workflows.

1003
00:54:09,919 --> 00:54:11,089
that's exciting and promising.

1004
00:54:11,089 --> 00:54:15,709
I think, by the end of the year,
maybe in Q1 of next year, my answer

1005
00:54:15,709 --> 00:54:16,969
will likely be very different.

1006
00:54:17,754 --> 00:54:18,044
Nirmal Mehta: Well,

1007
00:54:18,180 --> 00:54:18,470
Bret: Yeah.

1008
00:54:18,639 --> 00:54:21,449
Nirmal Mehta: have to bring you back to
give us an update on what you've seen.

1009
00:54:21,529 --> 00:54:21,949
so

1010
00:54:21,980 --> 00:54:22,585
Bret: a moving target

1011
00:54:22,635 --> 00:54:26,145
Laura Tacho: I wanna share just one,
you know, kind of thinking about how AI

1012
00:54:26,145 --> 00:54:28,065
can't make up for all of the devex gains.

1013
00:54:28,065 --> 00:54:32,175
I think agentic has a lot of promises,
but there's still so much work to

1014
00:54:32,175 --> 00:54:37,005
do when it comes to identifying
organizational use cases for AI

1015
00:54:37,065 --> 00:54:38,985
to help deliver software faster.

1016
00:54:39,225 --> 00:54:42,075
This is, um, I'm gonna publish
this I think this week actually.

1017
00:54:42,135 --> 00:54:49,211
Um, this is looking at data from over
a hundred companies ish that have

1018
00:54:49,331 --> 00:54:51,731
developer populations between 105 hundred.

1019
00:54:52,121 --> 00:54:54,311
And I'm looking at their
annualized cost savings.

1020
00:54:54,311 --> 00:54:57,161
these companies have said, this is
what a full-time employee costs.

1021
00:54:57,521 --> 00:55:01,751
This is how much our developers are
using and saving from AI each week.

1022
00:55:01,751 --> 00:55:06,581
And so this is the annualized cost
savings in US dollar from using AI.

1023
00:55:06,991 --> 00:55:08,731
obviously there's gonna be a trend line.

1024
00:55:08,731 --> 00:55:12,181
The more developers you have, the
more you can save because math.

1025
00:55:12,431 --> 00:55:14,441
but even if you look
at some of these like.

1026
00:55:15,491 --> 00:55:17,411
200 developer, 300 developer mark.

1027
00:55:17,411 --> 00:55:23,091
We've got companies saving over 6
million, in recovered salary costs.

1028
00:55:23,091 --> 00:55:27,081
And like, you know, this company
not that much of a different size.

1029
00:55:27,081 --> 00:55:28,911
It's like not even breaking a million.

1030
00:55:29,271 --> 00:55:31,821
It's not that these companies
up here are using Ag agentic

1031
00:55:31,821 --> 00:55:33,321
workflows to do a bunch of stuff.

1032
00:55:33,711 --> 00:55:37,161
It's that they have done really
good training and enablement.

1033
00:55:37,161 --> 00:55:39,681
They've found organizational use cases.

1034
00:55:39,731 --> 00:55:42,971
they've equipped their organizations
with AI instead of spraying and

1035
00:55:42,971 --> 00:55:45,941
praying to their individuals and
hoping that curiosity and grit

1036
00:55:45,941 --> 00:55:47,171
will take it the rest of the way.

1037
00:55:47,381 --> 00:55:50,441
there's so much room for
optimization all around.

1038
00:55:50,841 --> 00:55:55,276
Ag agentic workloads are incredibly
interesting and I'm very optimistic.

1039
00:55:55,376 --> 00:55:58,946
the reality is that I just don't see
them being widely adopted, especially

1040
00:55:59,096 --> 00:56:02,566
in customer facing environments
and in the enterprise just yet.

1041
00:56:02,596 --> 00:56:04,471
But again, my answer will probably change.

1042
00:56:05,126 --> 00:56:06,056
Nirmal Mehta: So this is interesting.

1043
00:56:06,056 --> 00:56:06,206
Is

1044
00:56:06,272 --> 00:56:06,432
Bret: Yeah.

1045
00:56:06,596 --> 00:56:09,463
Nirmal Mehta: normalized based
on, the developer's salaries?

1046
00:56:10,027 --> 00:56:12,427
Laura Tacho: the companies that
I've included in here have given us

1047
00:56:12,427 --> 00:56:17,647
information about how much a full-time
developer at their median salary point.

1048
00:56:17,777 --> 00:56:18,647
this is adjusted.

1049
00:56:18,647 --> 00:56:21,317
I have another version of this,
which I'll publish along with this,

1050
00:56:21,497 --> 00:56:23,957
where I pivot basically on region.

1051
00:56:24,367 --> 00:56:30,567
and you can see like the, the lower
bands there's a lot of Europe down here.

1052
00:56:30,597 --> 00:56:34,137
'cause the salaries in Europe
are just generally not as high.

1053
00:56:34,437 --> 00:56:38,067
these are US companies and then
like Asia companies are kind of

1054
00:56:38,877 --> 00:56:43,057
spattered, across, but that seems to
be like a pretty in aggregate, like a

1055
00:56:43,057 --> 00:56:48,247
representative trend, for what we're
seeing, also extending to bigger companies

1056
00:56:48,297 --> 00:56:49,822
Nirmal Mehta: I would
love to see that pivot.

1057
00:56:50,002 --> 00:56:52,882
on the left hand side,
the annual cost savings.

1058
00:56:53,192 --> 00:56:58,122
per, normalized, developer salary
or something like that, like a

1059
00:56:58,122 --> 00:57:00,967
little bit normal normalized, just
so that it's easier to compare.

1060
00:57:01,067 --> 00:57:01,787
this is fascinating.

1061
00:57:01,817 --> 00:57:02,057
Laura Tacho: yeah.

1062
00:57:02,442 --> 00:57:03,222
Nirmal Mehta: information.

1063
00:57:03,652 --> 00:57:06,712
so what are some of the, you
kind of mentioned, I mean these

1064
00:57:06,928 --> 00:57:07,148
Bret: It

1065
00:57:07,252 --> 00:57:09,592
Nirmal Mehta: are actual, this is like
real money that we're seeing, right?

1066
00:57:09,842 --> 00:57:10,172
Bret: Yeah.

1067
00:57:10,172 --> 00:57:14,981
And if I had to describe this to the audio
listeners, looking at a graph that is.

1068
00:57:16,331 --> 00:57:17,141
all over the map.

1069
00:57:17,251 --> 00:57:22,741
there is nothing here that is certain,
there is no clear location where like,

1070
00:57:22,741 --> 00:57:25,411
this is where everybody, like the majority
of people are seeing this level of

1071
00:57:25,545 --> 00:57:25,665
Laura Tacho: Yeah.

1072
00:57:26,161 --> 00:57:29,071
Bret: It's basically every company
of like this size you said, right?

1073
00:57:29,131 --> 00:57:30,781
100 to 500 engineers, I think is what you

1074
00:57:31,030 --> 00:57:31,450
Laura Tacho: Mm-hmm.

1075
00:57:31,765 --> 00:57:33,235
Bret: and they're all over the map.

1076
00:57:33,955 --> 00:57:36,865
I mean, as you can might have
guess the ones that are saving

1077
00:57:36,865 --> 00:57:38,545
the most money are rare.

1078
00:57:39,125 --> 00:57:42,935
there definitely seems to be more
concentration on the low end of benefit,

1079
00:57:43,685 --> 00:57:46,255
but there is no clear cluster here yet.

1080
00:57:46,255 --> 00:57:50,945
this doesn't look like a typical sort
of Gartner, approach where everyone's

1081
00:57:50,945 --> 00:57:55,175
huddled in one place, I think this is
just indicative of any new technology

1082
00:57:55,175 --> 00:57:56,615
that we don't really know how to use yet.

1083
00:57:56,615 --> 00:57:57,725
We don't really know the patterns of

1084
00:57:57,969 --> 00:57:58,389
Laura Tacho: Mm-hmm.

1085
00:57:58,505 --> 00:58:02,075
Bret: know a lot of people are trying,
and you might be wasting your time.

1086
00:58:02,075 --> 00:58:02,795
You might not.

1087
00:58:02,845 --> 00:58:03,475
Laura Tacho: Absolutely.

1088
00:58:03,605 --> 00:58:05,015
that's exactly how I interpret this.

1089
00:58:05,015 --> 00:58:09,220
And I use this to set up, like it's,
just look at the, for the audio

1090
00:58:09,220 --> 00:58:12,730
listeners, we've got on the X axis
engineering organization size and on

1091
00:58:12,730 --> 00:58:16,720
the Y axis annualized cost savings
in USD, but only for companies

1092
00:58:16,720 --> 00:58:18,220
from a hundred to 500 engineers.

1093
00:58:18,220 --> 00:58:19,780
So if we look at companies that have.

1094
00:58:20,150 --> 00:58:24,160
200 to 250 engineers, We've got
all the way from, five and a half

1095
00:58:24,160 --> 00:58:27,910
million down to 800,000 for the
same exact size of engineers.

1096
00:58:28,090 --> 00:58:31,530
And so what that tells us is I mean,
maturity, we can look at how long

1097
00:58:31,530 --> 00:58:34,890
has AI been introduced, but what it
comes down to in my experience, kind

1098
00:58:34,890 --> 00:58:38,610
of like anecdotally and I'll put
together the dashboards to prove this.

1099
00:58:38,610 --> 00:58:42,480
'cause I do have data to prove it as
well, but it is developer experience and

1100
00:58:42,480 --> 00:58:44,790
it is training and enablement technique.

1101
00:58:45,000 --> 00:58:49,800
if a company makes licenses
available to engineers and says,

1102
00:58:50,100 --> 00:58:52,350
okay, go for, it encourages people.

1103
00:58:52,350 --> 00:58:57,120
Even make a Slack channel, share your
good practices that will not bring as

1104
00:58:57,120 --> 00:59:01,840
much, effectiveness as actually Nirmal
borrowing, like your experience-based

1105
00:59:01,840 --> 00:59:06,640
accelerator, like Amazon's kind of
method of bring a real business problem

1106
00:59:06,670 --> 00:59:09,520
and then bring the technology together
and like, let's figure out how to solve

1107
00:59:09,520 --> 00:59:11,200
the real business problem as a team.

1108
00:59:12,100 --> 00:59:16,390
And then create a blueprint
for that and spread that out.

1109
00:59:16,630 --> 00:59:19,000
That has been incredibly useful.

1110
00:59:19,000 --> 00:59:22,720
booking.com, for example,
they have 3,500 engineers.

1111
00:59:22,970 --> 00:59:29,270
they have a 30% increase in merger
requests, throughput, but they were

1112
00:59:29,270 --> 00:59:32,900
able with that technique of the
experience-based accelerator doing

1113
00:59:32,930 --> 00:59:37,910
modernization projects and getting
70 to 80% of the way there with AI

1114
00:59:38,060 --> 00:59:39,740
using that technique from Amazon.

1115
00:59:40,100 --> 00:59:45,980
the median adoption for developers in
an organization is a little over 50%

1116
00:59:45,980 --> 00:59:51,170
of developers using AI on a daily or
weekly basis for AI assisted engineering.

1117
00:59:51,480 --> 00:59:56,490
booking has 60 plus percent using
it on daily and weekly, and of those

1118
00:59:56,490 --> 00:59:58,350
70% are using it on a daily basis.

1119
00:59:58,350 --> 01:00:00,270
So they've been extremely successful.

1120
01:00:00,360 --> 01:00:03,750
They have a huge ROI, it comes
from the training and enable.

1121
01:00:04,330 --> 01:00:08,650
and treating it as an organizational
tool and focusing on the workflows

1122
01:00:08,650 --> 01:00:13,900
that AI helps to enable and not just
an individual productivity tool where

1123
01:00:13,900 --> 01:00:15,910
Nirmal can type twice as fast Now.

1124
01:00:16,290 --> 01:00:16,860
Nirmal Mehta: Interesting.

1125
01:00:16,860 --> 01:00:22,328
So this mirrors the same kind of
conversations I was having with

1126
01:00:22,328 --> 01:00:27,698
organizations two decades ago, or
a little under two decades ago,

1127
01:00:27,909 --> 01:00:27,989
Bret: Hmm.

1128
01:00:27,998 --> 01:00:31,208
Nirmal Mehta: With the adoption
of DevOps practices, organizations

1129
01:00:31,238 --> 01:00:32,908
back then, I think Target

1130
01:00:32,908 --> 01:00:38,788
famously had like a dojo, a
devops dojo where they would bring

1131
01:00:38,828 --> 01:00:41,428
teams in with these um devops.

1132
01:00:41,790 --> 01:00:49,210
Subject matter experts take a workload,
and re-architect it and train the

1133
01:00:49,210 --> 01:00:54,040
trainer and get the developers and
the platform engineers up to speed

1134
01:00:54,040 --> 01:00:55,900
around that specific workload.

1135
01:00:56,300 --> 01:00:58,670
migrate it, modernize it, and then.

1136
01:00:59,060 --> 01:01:01,400
Put it through production
and then take the next team.

1137
01:01:01,580 --> 01:01:04,430
And so what you're saying is
this is no different, right?

1138
01:01:04,430 --> 01:01:10,470
AI adoption, developer experience with,
AI tooling, organization, if they want to

1139
01:01:10,470 --> 01:01:15,900
be successful with these tools and really
see some gains, they need to take the same

1140
01:01:15,900 --> 01:01:21,901
approach, of standing up some kind of,
outcome oriented, what do you call it?

1141
01:01:21,901 --> 01:01:24,541
Like a, uh, dojo if you want.

1142
01:01:24,571 --> 01:01:27,881
We could probably call it dojo,
like an AI dojo, for adopting these

1143
01:01:27,881 --> 01:01:32,531
tools to be successful and not just
say, oh, here is an X, Y, Z license.

1144
01:01:32,961 --> 01:01:36,321
report back to me in a quarter
where, you know, how you've been

1145
01:01:36,321 --> 01:01:39,111
30% more productive regardless of

1146
01:01:39,311 --> 01:01:39,821
Laura Tacho: Here's

1147
01:01:39,861 --> 01:01:40,191
Nirmal Mehta: doing.

1148
01:01:40,769 --> 01:01:44,009
Laura Tacho: the bottom line and
maybe a good, um, bottom line to,

1149
01:01:44,069 --> 01:01:48,479
wrap our conversation up, which is if
organizations wanna have organization-wide

1150
01:01:48,479 --> 01:01:52,559
impact increased productivity across
the board, they need to treat AI as

1151
01:01:52,559 --> 01:01:57,059
an organizational tool, not as an
individual productivity speed up.

1152
01:01:57,479 --> 01:02:00,719
And the organizations that do that,
that treat it, like you said, like have

1153
01:02:00,719 --> 01:02:05,879
an AI dojo or treat it with, you know,
enablement in the same way you would with,

1154
01:02:05,909 --> 01:02:09,509
with any other kind of organization-wide
tool, they're seeing big impact.

1155
01:02:09,509 --> 01:02:13,799
And the ones that are just spraying and
praying licenses aren't gonna see that,

1156
01:02:13,799 --> 01:02:19,229
that impact because it's, there's nothing
like, there's nothing new under the sun.

1157
01:02:19,229 --> 01:02:19,619
Right.

1158
01:02:19,649 --> 01:02:20,639
this is a tool.

1159
01:02:20,639 --> 01:02:23,849
As any other developer tool,
no one would've bought.

1160
01:02:24,529 --> 01:02:28,879
Jenkins licenses are like Docker
licenses and just expect like, oh, we're

1161
01:02:28,879 --> 01:02:30,559
gonna modernize our whole application.

1162
01:02:30,559 --> 01:02:31,489
It's gonna run in Docker now.

1163
01:02:31,539 --> 01:02:32,649
here's a license.

1164
01:02:32,799 --> 01:02:35,099
Go figure it out That didn't work then.

1165
01:02:35,099 --> 01:02:36,689
It doesn't work with AI now.

1166
01:02:36,849 --> 01:02:38,739
even though it's sparkly and
shiny and there's a lot of

1167
01:02:38,739 --> 01:02:41,229
hype, like the fundamentals in
the physics haven't changed.

1168
01:02:41,304 --> 01:02:44,574
and I think there's a lot of
level-headed leaders, like both of

1169
01:02:44,574 --> 01:02:48,164
you, Bret and Nirmal, treating this
with the right amount of scrutiny

1170
01:02:48,164 --> 01:02:49,904
and skepticism and optimism.

1171
01:02:49,934 --> 01:02:52,904
leaders who can do that are gonna
be the ones that come out ahead

1172
01:02:53,114 --> 01:02:56,444
than the ones who are, you know,
spraying and praying and following

1173
01:02:56,444 --> 01:02:58,244
victim to the hype, unfortunately.

1174
01:02:59,365 --> 01:03:02,605
Bret: So you said something about two
or three weeks ago, what happened?

1175
01:03:02,605 --> 01:03:03,385
Two or three weeks ago.

1176
01:03:04,440 --> 01:03:08,250
Laura Tacho: this has been a core
concern for so many of our customers

1177
01:03:08,250 --> 01:03:09,750
and so many like CTOs and VPs.

1178
01:03:09,780 --> 01:03:14,610
'cause again, what decisions would
you make if you knew that 60% of your

1179
01:03:14,610 --> 01:03:19,230
developers were using AI for, for AI
assisted coding, but 0% of that code

1180
01:03:19,230 --> 01:03:23,370
made it to production versus if you
had 10% of your developers using.

1181
01:03:23,955 --> 01:03:27,135
AI tools daily, weekly, and
50% of the code going to

1182
01:03:27,135 --> 01:03:28,455
production was written by AI.

1183
01:03:28,485 --> 01:03:31,215
those are two wildly different
scenarios, and it's important

1184
01:03:31,215 --> 01:03:32,035
to figure out where you are.

1185
01:03:32,685 --> 01:03:38,535
we at DX have a tool that runs on the file
system layer so it can track in any IDE

1186
01:03:39,465 --> 01:03:45,735
in the terminal, whatever, code completion
from AI tools as well as copy paste and

1187
01:03:45,735 --> 01:03:50,275
then track the human editing and then do
that on a per line basis, at the commit

1188
01:03:50,275 --> 01:03:52,015
level so that we can look at the ratio.

1189
01:03:52,480 --> 01:03:56,320
Of AI code to human authored code in
a given commit, and then we can track

1190
01:03:56,320 --> 01:03:57,640
that commit all the way to production.

1191
01:03:57,640 --> 01:04:01,630
So there's like pockets of tools that
could do this before, like Windsurf had

1192
01:04:01,630 --> 01:04:06,100
some pretty good telemetry into what
was going on, but before we just sort

1193
01:04:06,100 --> 01:04:11,230
of had acceptance rate, which is, you
know, accepted suggestions and really

1194
01:04:11,230 --> 01:04:16,090
no insight into what happens even like
before the commit, after the acceptance,

1195
01:04:16,090 --> 01:04:19,870
and then definitely after the commit
and into customer facing environment.

1196
01:04:19,870 --> 01:04:20,140
So.

1197
01:04:21,085 --> 01:04:22,015
excited about this.

1198
01:04:22,015 --> 01:04:23,875
It's technically very cool.

1199
01:04:24,325 --> 01:04:24,705
Bret: Is this.

1200
01:04:24,744 --> 01:04:25,494
Nirmal Mehta: scary to me.

1201
01:04:25,501 --> 01:04:25,921
Laura Tacho: industry

1202
01:04:26,244 --> 01:04:26,514
Nirmal Mehta: That,

1203
01:04:26,660 --> 01:04:28,490
Bret: Is this a part of DX?

1204
01:04:28,729 --> 01:04:31,759
is this a part of the DX platform,
or is this an open source project?

1205
01:04:31,809 --> 01:04:32,799
what is this specifically?

1206
01:04:32,810 --> 01:04:33,770
Laura Tacho: it's proprietary,

1207
01:04:33,820 --> 01:04:34,110
Bret: Okay.

1208
01:04:34,165 --> 01:04:34,555
Laura Tacho: the DX

1209
01:04:34,768 --> 01:04:37,798
Nirmal Mehta: sorry, that shot should
not have been my first reaction to that,

1210
01:04:37,828 --> 01:04:39,238
it sounds really cool, but it does.

1211
01:04:39,348 --> 01:04:43,338
So the scary part to me is it sounds very,

1212
01:04:44,088 --> 01:04:44,608
Laura Tacho: Brothery.

1213
01:04:44,937 --> 01:04:48,607
Nirmal Mehta: yeah, like survey and,
you know, we could have another whole

1214
01:04:48,607 --> 01:04:53,375
entire episode on this, but this
kind of comes back to, incentives

1215
01:04:53,375 --> 01:05:00,725
by leadership to use this hype for
ulterior motives, let's just say that.

1216
01:05:01,016 --> 01:05:01,306
Laura Tacho: Yeah.

1217
01:05:01,731 --> 01:05:05,191
Nirmal Mehta: I think it is important
for us to know how much of that AI

1218
01:05:05,191 --> 01:05:07,201
code is out there in production.

1219
01:05:07,231 --> 01:05:11,721
I think that's a very legitimate concern
on all aspects, especially from a platform

1220
01:05:12,197 --> 01:05:12,437
Bret: yeah,

1221
01:05:12,981 --> 01:05:14,601
Nirmal Mehta: and a
DevOps team perspective.

1222
01:05:14,601 --> 01:05:17,121
I think very crucial
information, but at the

1223
01:05:17,207 --> 01:05:17,717
Bret: to be fair,

1224
01:05:18,141 --> 01:05:20,421
Nirmal Mehta: for monitoring
copy and paste from a developer

1225
01:05:20,421 --> 01:05:21,981
like, oh boy, you know.

1226
01:05:22,213 --> 01:05:22,603
Laura Tacho: Yeah.

1227
01:05:22,817 --> 01:05:23,117
Bret: Yeah.

1228
01:05:23,447 --> 01:05:25,827
I mean, it's not actually
uploading the copy, right?

1229
01:05:25,827 --> 01:05:27,657
we're not talking security concerns.

1230
01:05:27,657 --> 01:05:28,437
We're just talking.

1231
01:05:28,478 --> 01:05:29,078
Laura Tacho: no,

1232
01:05:29,939 --> 01:05:33,139
Bret: Because this is, this sounds
way better to me than a decade ago

1233
01:05:33,139 --> 01:05:36,429
when I was freelancing through certain
websites on the internet to random

1234
01:05:36,429 --> 01:05:39,489
companies when I was first starting
my freelancing business 15 years ago.

1235
01:05:39,849 --> 01:05:43,139
back then you were billing per hour
and the tool you were using for

1236
01:05:43,139 --> 01:05:46,829
consulting would screenshot your
computer every five to seven minutes

1237
01:05:46,829 --> 01:05:50,009
randomly and send it to the client to
prove you were working on their tool

1238
01:05:50,009 --> 01:05:51,209
because you were billing per hour.

1239
01:05:51,479 --> 01:05:53,519
That's way more invasive than this.

1240
01:05:53,519 --> 01:05:55,319
I had to put up with that 15 years ago.

1241
01:05:55,319 --> 01:05:57,199
But, anyway, this sounds a lot better.

1242
01:05:57,329 --> 01:06:00,319
Laura Tacho: Well, again, like the
double in the details, so change

1243
01:06:00,319 --> 01:06:03,829
failure rate is one of the four
key DORA metrics that organizations

1244
01:06:03,829 --> 01:06:05,359
have been tracking for a decade.

1245
01:06:05,579 --> 01:06:07,534
in order to get that number.

1246
01:06:08,444 --> 01:06:10,784
We ask a question to individual
developers, this is how it's

1247
01:06:10,784 --> 01:06:12,134
done for the Dora report as well.

1248
01:06:12,134 --> 01:06:15,794
And then accelerate, you know, in
the last month or whatever timeframe,

1249
01:06:15,794 --> 01:06:19,154
how many of the changes you pushed to
production resulted in degraded service.

1250
01:06:19,154 --> 01:06:22,934
so we can have a data point for the
individual and how many of their changes

1251
01:06:22,934 --> 01:06:28,034
resulted in degraded service, but we
don't look at them on an individual basis.

1252
01:06:28,034 --> 01:06:29,354
We look at them in aggregate.

1253
01:06:29,354 --> 01:06:32,513
the point is about measuring the system,
not about measuring the individual.

1254
01:06:32,563 --> 01:06:32,983
Nirmal Mehta: Mm-hmm.

1255
01:06:33,194 --> 01:06:35,084
Laura Tacho: And the same
is true for these AI tools.

1256
01:06:35,084 --> 01:06:35,834
Like we're not.

1257
01:06:36,224 --> 01:06:42,194
Looking at, Bret had 70% AI authored
code and 30% where Nirmal had 10%

1258
01:06:42,194 --> 01:06:44,144
AI authored and 90% human authored.

1259
01:06:44,264 --> 01:06:47,164
We're looking at it in aggregate to
figure out what's the shape of the

1260
01:06:47,164 --> 01:06:48,394
code that's running in production?

1261
01:06:48,994 --> 01:06:50,134
What's the security risk?

1262
01:06:50,134 --> 01:06:51,664
How is AI being adopted?

1263
01:06:51,664 --> 01:06:56,744
Is it being adopted by, you know,
like bearded men are using AI a

1264
01:06:56,764 --> 01:06:59,854
lot more, you know, like, different
attributes and demographics.

1265
01:06:59,934 --> 01:07:03,019
Unfortunately, I have to tell the
audience out there, DX does not

1266
01:07:03,019 --> 01:07:04,459
have the attribute of bearded men.

1267
01:07:04,559 --> 01:07:05,459
Bret: it's not a checkbox.

1268
01:07:05,868 --> 01:07:06,768
Laura Tacho: actually that's not true.

1269
01:07:06,768 --> 01:07:08,208
We can do any custom attribute.

1270
01:07:08,208 --> 01:07:09,368
So, Bret AI

1271
01:07:09,443 --> 01:07:09,923
Bret: Okay,

1272
01:07:10,048 --> 01:07:12,208
Laura Tacho: if we were on our
own team, I would definitely

1273
01:07:12,208 --> 01:07:13,228
have that as an attribute for

1274
01:07:13,263 --> 01:07:13,863
Bret: great.

1275
01:07:13,863 --> 01:07:14,763
Gray beards.

1276
01:07:14,763 --> 01:07:16,293
I want the gray beard checkbox.

1277
01:07:16,443 --> 01:07:16,833
Yeah.

1278
01:07:16,909 --> 01:07:17,243
Laura Tacho: totally.

1279
01:07:17,903 --> 01:07:21,138
but it's all, the same threats
and the same kind of concern

1280
01:07:21,138 --> 01:07:22,788
about Big Brother surveillance.

1281
01:07:23,968 --> 01:07:26,008
measuring the system versus
measuring the individual.

1282
01:07:26,008 --> 01:07:27,538
They all still exist.

1283
01:07:27,538 --> 01:07:30,718
And even more so with AI because
the, the threat of like, my job

1284
01:07:30,718 --> 01:07:34,288
is gonna be at risk because this
technology is gonna replace me.

1285
01:07:34,701 --> 01:07:37,551
I feel like we're in a pressure cooker
and there's pressure coming from all

1286
01:07:37,595 --> 01:07:39,005
Nirmal Mehta: Yeah, that's

1287
01:07:39,316 --> 01:07:39,716
Bret: Yeah.

1288
01:07:40,350 --> 01:07:42,330
Nirmal Mehta: I feel like we're
in a pressure cooker right now.

1289
01:07:42,896 --> 01:07:43,186
Laura Tacho: Yeah.

1290
01:07:43,250 --> 01:07:43,610
Bret: Yeah.

1291
01:07:43,868 --> 01:07:48,408
And so if the audience can feel anything,
we're with you, we feel the pressure,

1292
01:07:48,418 --> 01:07:50,848
you're probably listening to this podcast
'cause you probably have a little bit of

1293
01:07:50,848 --> 01:07:54,328
that pressure that you're feeling to stay
up to date and be on the leading edge.

1294
01:07:54,378 --> 01:07:58,488
part of it's curiosity and
passion around that new tech.

1295
01:07:58,518 --> 01:08:01,398
Part of it's the obligation
that we feel like our jobs are

1296
01:08:01,398 --> 01:08:04,998
somehow maybe at risk someday.

1297
01:08:05,048 --> 01:08:07,808
I mean, there's so much confusion
and news about everything.

1298
01:08:07,858 --> 01:08:11,008
I go up and down in emotions
every day around AI.

1299
01:08:11,378 --> 01:08:15,428
it's probably generated more
frustration and emotion in me than

1300
01:08:16,118 --> 01:08:19,208
anything in recent memory that
I can think of in terms of tech.

1301
01:08:19,249 --> 01:08:19,449
Laura Tacho: Mm-hmm.

1302
01:08:19,598 --> 01:08:23,488
Bret: So, given all that we've talked
about what people can do from here,

1303
01:08:23,568 --> 01:08:25,368
obviously it's a complicated answer.

1304
01:08:25,368 --> 01:08:29,328
I feel like what we're saying is, really
anything in your software life cycle

1305
01:08:29,328 --> 01:08:31,158
potentially could be beneficial with AI.

1306
01:08:31,518 --> 01:08:35,398
But you need to make it a project and you
need to take it seriously and you need

1307
01:08:35,398 --> 01:08:37,168
to train people and do all that stuff.

1308
01:08:37,168 --> 01:08:38,338
I feel like that's the big take.

1309
01:08:38,818 --> 01:08:42,148
But I wanted to switch a little
bit to a slightly adjacent topic

1310
01:08:42,148 --> 01:08:43,678
that we haven't discussed yet.

1311
01:08:44,398 --> 01:08:45,568
Context switching.

1312
01:08:46,588 --> 01:08:52,558
one of the potential hopes I have
and I think others have for AI is

1313
01:08:52,558 --> 01:08:55,848
that it may actually help developers
with all this context switching.

1314
01:08:55,848 --> 01:08:59,538
And you showed a graphic earlier
of like the top 15 activities a

1315
01:08:59,538 --> 01:09:01,068
developer does on a given week.

1316
01:09:01,368 --> 01:09:05,928
Everything from meetings to staring
at Jira to improving documentation to,

1317
01:09:05,928 --> 01:09:08,298
oh yeah, I also get the code right?

1318
01:09:08,348 --> 01:09:08,768
Laura Tacho: Mm-hmm.

1319
01:09:08,778 --> 01:09:10,338
Bret: all of those context switches.

1320
01:09:10,338 --> 01:09:13,458
Nevermind the idea of switching
between one code base and another.

1321
01:09:14,118 --> 01:09:16,578
Um, do you have anything on con?

1322
01:09:16,608 --> 01:09:19,498
Is there, are there, are there
lessons learned from this?

1323
01:09:19,498 --> 01:09:20,668
Is there tools we can look at?

1324
01:09:20,668 --> 01:09:24,688
Because it's one of my top
goals to improve my ability

1325
01:09:24,688 --> 01:09:25,888
to context switch with AI.

1326
01:09:25,938 --> 01:09:26,808
I don't have any good tools.

1327
01:09:26,808 --> 01:09:29,208
don't have any good methods
or any good information on it.

1328
01:09:29,208 --> 01:09:31,248
So I was hoping we could talk about that.

1329
01:09:31,558 --> 01:09:35,568
Laura Tacho: I wanna, just take this
time to highlight that focus time as

1330
01:09:35,568 --> 01:09:39,288
a driver of developer experience is
capturing exactly that, like cognitive

1331
01:09:39,288 --> 01:09:43,668
load context switching kind of tax
that developers pay from like bopping

1332
01:09:43,968 --> 01:09:48,108
around from different tasks, but then
also having their flow interrupted,

1333
01:09:48,108 --> 01:09:49,908
like, you know, I'm coding and then.

1334
01:09:50,623 --> 01:09:51,703
I have to go to a meeting.

1335
01:09:51,943 --> 01:09:55,123
We don't get enough focus time to
really get into the flow state.

1336
01:09:55,303 --> 01:10:00,523
And so that is one of the kind of
hypotheses is that if we use AI and we can

1337
01:10:00,523 --> 01:10:06,193
automate the toil that's interruptive, do
developers actually get more focused time?

1338
01:10:06,493 --> 01:10:10,093
And I wish I had like a more conclusive
answer because I did look at this

1339
01:10:10,093 --> 01:10:13,003
very recently and like the answers are
a bit all over the place right now.

1340
01:10:13,003 --> 01:10:14,113
Even controlling for.

1341
01:10:14,423 --> 01:10:15,743
usage and different things.

1342
01:10:15,743 --> 01:10:22,303
we track it as, a first level, primary
metric on this AI, impact board

1343
01:10:22,303 --> 01:10:24,463
where we can do group comparison.

1344
01:10:24,463 --> 01:10:29,113
So if we wanna compare like all AI users
to non-AI users, we can look at how

1345
01:10:29,113 --> 01:10:32,233
that impacts their focus time, because
that's really important in this case.

1346
01:10:32,473 --> 01:10:37,833
the group that's using AI actually has a
lower amount of, Focus time, which isn't

1347
01:10:38,023 --> 01:10:38,313
Bret: Rent.

1348
01:10:38,673 --> 01:10:43,803
Laura Tacho: isn't great, but we can see
how that trend evolves, across like the

1349
01:10:43,803 --> 01:10:50,103
different usage of like infrequent monthly
to weekly to daily and how the focus

1350
01:10:50,103 --> 01:10:54,733
time is impacted because that is, again,
like the mechanics and the physics of.

1351
01:10:55,678 --> 01:10:59,608
Good devex and like good team
habits, like focus time is still

1352
01:10:59,608 --> 01:11:01,348
important even when we have AI.

1353
01:11:01,378 --> 01:11:05,908
And a lot of companies that I work
with do have the hypothesis that AI

1354
01:11:05,908 --> 01:11:10,308
can automate, toil give more time for
focus and increase the innovation ratio.

1355
01:11:10,308 --> 01:11:13,558
So this percentage of, time spent
on feature development, we see this,

1356
01:11:13,778 --> 01:11:18,428
kind of sample group has 4.8% higher
allocation toward feature development.

1357
01:11:19,058 --> 01:11:20,378
So that's, that's great.

1358
01:11:20,378 --> 01:11:24,488
Their innovation ratio, we're looking
at things like quality, speed, velocity,

1359
01:11:24,518 --> 01:11:29,918
maintainability, and kind of making sure
all those things stay moving together so

1360
01:11:29,918 --> 01:11:32,378
that we don't over index on one metric.

1361
01:11:32,892 --> 01:11:36,582
use AI to preserve context, I leave
little notes for myself in chat.

1362
01:11:36,582 --> 01:11:37,182
GPT.

1363
01:11:37,857 --> 01:11:42,777
And within my five hour Claude code,
session, I find it so helpful to

1364
01:11:42,827 --> 01:11:46,547
not feel the pressure to have to
write a bunch of post-its to myself.

1365
01:11:46,547 --> 01:11:47,057
Like I can just

1366
01:11:47,161 --> 01:11:47,381
Mm.

1367
01:11:47,507 --> 01:11:49,187
from my task and come back to it.

1368
01:11:49,187 --> 01:11:53,117
And I've like really accelerated
the time it takes to get back into

1369
01:11:53,117 --> 01:11:58,067
things just because I have this like
programmer who never gets tired and

1370
01:11:58,067 --> 01:12:02,147
is always there for me, except after
five hours when he magically goes away.

1371
01:12:02,257 --> 01:12:02,587
Nirmal Mehta: Okay.

1372
01:12:03,047 --> 01:12:06,307
as we're wrapping up, for our
listeners, we have individuals trying

1373
01:12:06,307 --> 01:12:11,267
to navigate these rapid changes,
that pressure cooker kind of feeling.

1374
01:12:11,997 --> 01:12:16,167
what's some advice you would
give our audience, that they

1375
01:12:16,167 --> 01:12:19,107
could use today to navigate that?

1376
01:12:21,078 --> 01:12:23,038
Laura Tacho: I think first
thing is sort of a mindset.

1377
01:12:24,033 --> 01:12:30,273
Switch, like average median time savings
per engineer per week from using AI

1378
01:12:30,273 --> 01:12:33,393
tools for AI assisted engineering
is a little under four hours.

1379
01:12:33,613 --> 01:12:36,673
that data comes from more
than 38,000 developers.

1380
01:12:36,943 --> 01:12:41,593
When you see a headline, know that
that's a headline for a reason

1381
01:12:41,593 --> 01:12:44,143
and like get curious about what
might be going on behind it.

1382
01:12:44,143 --> 01:12:47,908
So think that's step one is just to
like identify what hype have a better.

1383
01:12:48,913 --> 01:12:50,983
idea of identifying what type.

1384
01:12:51,433 --> 01:12:52,903
I think the other thing is, it's.

1385
01:12:54,613 --> 01:12:57,763
Really difficult to make an impact
with AI if you can't measure it.

1386
01:12:57,833 --> 01:12:59,903
you know, if you don't know
what's actually happening.

1387
01:12:59,903 --> 01:13:04,883
And so getting instrumentation in place
for measuring things like maintainability,

1388
01:13:04,883 --> 01:13:10,013
DevEx, speed, throughput, quality, these
can be done lightweight with a survey.

1389
01:13:10,173 --> 01:13:15,243
it doesn't need to be this huge robust
ETL process and data lakes everywhere.

1390
01:13:15,243 --> 01:13:19,053
You can do so much for getting
visibility into how your organization is

1391
01:13:19,053 --> 01:13:21,423
operating now and how AI is changing it.

1392
01:13:21,888 --> 01:13:25,148
Do that, easily for very low
cost other than the time it takes

1393
01:13:25,148 --> 01:13:26,708
to do it for you as a person.

1394
01:13:27,518 --> 01:13:31,718
that data is gonna be instrumental
in helping you make arguments for or

1395
01:13:31,718 --> 01:13:33,788
against certain strategic decisions.

1396
01:13:33,868 --> 01:13:36,238
on a leadership level, I
think that's another thing.

1397
01:13:36,598 --> 01:13:41,068
and I think the maybe more tactical
way is if you are feeling the pressure

1398
01:13:41,338 --> 01:13:45,028
for AI more in your organization.

1399
01:13:45,463 --> 01:13:48,703
Look at how AI is being presented.

1400
01:13:48,703 --> 01:13:53,383
Is it an individual productivity
tool where people get, a

1401
01:13:53,383 --> 01:13:56,293
license and then they're sharing
interesting things that they do?

1402
01:13:56,293 --> 01:14:01,063
Or are you doing this AI dojo kind
of scenario where you're looking

1403
01:14:01,063 --> 01:14:04,603
at problems that teams have and
problems that the organization

1404
01:14:04,603 --> 01:14:07,003
has and finding ways that AI can.

1405
01:14:07,363 --> 01:14:10,423
Solve those problems organization
wide, that's where you're

1406
01:14:10,423 --> 01:14:11,593
gonna get the most leverage.

1407
01:14:11,623 --> 01:14:12,793
focus on those things.

1408
01:14:13,073 --> 01:14:16,403
if you do those three together, it's
gonna take some time, but this sort of

1409
01:14:16,463 --> 01:14:18,803
gets you out of the fog a little bit.

1410
01:14:18,853 --> 01:14:22,753
truth be told it doesn't actually matter
if Microsoft is writing 30% of their code.

1411
01:14:23,248 --> 01:14:24,508
Because you're not Microsoft.

1412
01:14:24,758 --> 01:14:29,998
it's really difficult to not get a bit of
FOMO or not feel the pressure, especially

1413
01:14:29,998 --> 01:14:32,848
if you're getting pressure from your
exec team about, well, I saw this in

1414
01:14:32,848 --> 01:14:34,558
the headlines, why aren't we doing that?

1415
01:14:34,918 --> 01:14:37,618
But coming back to them with data
about what our business needs and what

1416
01:14:37,618 --> 01:14:41,278
we're doing, and how I know that we
need to do this and not this next.

1417
01:14:41,458 --> 01:14:44,248
That is what is gonna get
you ahead and stay ahead.

1418
01:14:44,498 --> 01:14:48,983
instead of just chasing metrics, vanity
metrics that you see in a. LinkedIn

1419
01:14:48,983 --> 01:14:51,443
post on a random Tuesday afternoon

1420
01:14:52,266 --> 01:14:53,201
Bret: Unless it's yours,

1421
01:14:54,436 --> 01:14:55,216
Laura Tacho: unless it's mine.

1422
01:14:55,416 --> 01:14:56,041
Bret: unless it's your post.

1423
01:14:56,326 --> 01:14:57,886
Laura Tacho: none of my
metrics are vanity metrics.

1424
01:14:57,936 --> 01:15:02,166
they all have adequately sufficient n
sizes, or I call it when they don't,

1425
01:15:02,196 --> 01:15:03,756
when a median might be unstable.

1426
01:15:04,026 --> 01:15:06,096
But yeah, it is really hard to
differentiate that because not

1427
01:15:06,096 --> 01:15:09,666
everyone, you know, LinkedIn is for
half of a brain and half of a thumb.

1428
01:15:10,066 --> 01:15:14,366
and a lot of, the shiny graphs
get the, most shareable.

1429
01:15:14,546 --> 01:15:17,666
Bret: speaking of that, so, uh,
thanks so much for being here.

1430
01:15:17,666 --> 01:15:20,066
Where can people find out
more about what you're doing?

1431
01:15:20,201 --> 01:15:21,461
where are you chatting?

1432
01:15:21,461 --> 01:15:22,091
On the internet.

1433
01:15:23,107 --> 01:15:25,807
Laura Tacho: I am only
on LinkedIn these days

1434
01:15:25,857 --> 01:15:26,157
Bret: Okay.

1435
01:15:27,027 --> 01:15:28,107
It's an easy place to be.

1436
01:15:28,512 --> 01:15:29,862
Laura Tacho: I tried to do Blue Sky.

1437
01:15:29,862 --> 01:15:31,612
I really did for you,
Bret, and for you Nirmal.

1438
01:15:31,632 --> 01:15:32,982
And I just really couldn't do it.

1439
01:15:33,539 --> 01:15:35,519
Bret: I don't get likes
there, so I'm the same way.

1440
01:15:35,559 --> 01:15:38,999
no one sees and pays attention to my stuff
other than Nirmal and a few other people.

1441
01:15:39,415 --> 01:15:39,895
Laura Tacho: other than.

1442
01:15:40,900 --> 01:15:41,650
That's very cute.

1443
01:15:42,100 --> 01:15:45,010
Um, find me on LinkedIn I'm
the only Laura Tacho out there.

1444
01:15:45,200 --> 01:15:48,600
and you can read about more of like
my research and guides on getdx.com

1445
01:15:48,620 --> 01:15:49,760
if you go to the research tab.

1446
01:15:49,980 --> 01:15:54,820
I'm doing a lot of research and white
papers, and I heard from Nirmal, if you

1447
01:15:54,820 --> 01:15:58,060
Google me, you'll come up with a lot
of interesting, talks that I've done

1448
01:15:58,323 --> 01:15:58,893
Nirmal Mehta: Yes.

1449
01:15:58,950 --> 01:16:01,890
Laura Tacho: I've been, at the
Enterprise Tech Leadership Summit

1450
01:16:01,890 --> 01:16:05,430
in Vegas, talking about booking.com
and how they scaled AI adoption.

1451
01:16:05,910 --> 01:16:08,320
I'm gonna be doing some, other keynotes.

1452
01:16:08,320 --> 01:16:09,610
I'll be at Cloud native Austria.

1453
01:16:09,610 --> 01:16:12,400
Actually, it's not, it's
a CNCF, like adjacent, not

1454
01:16:12,689 --> 01:16:13,259
Bret: Hmm.

1455
01:16:13,670 --> 01:16:14,960
Laura Tacho: but I'm
very excited for that.

1456
01:16:14,990 --> 01:16:16,790
I'll be keynoting there
as well in October.

1457
01:16:16,869 --> 01:16:17,469
Bret: Ooh.

1458
01:16:17,473 --> 01:16:18,703
Laura Tacho: I'll also be at KubeCon.

1459
01:16:18,720 --> 01:16:20,130
I'm gonna be at.

1460
01:16:20,865 --> 01:16:24,675
Platform Engineering Day, talking
about the real adoption rates of

1461
01:16:24,765 --> 01:16:28,875
AI tools across the industry to
help dispel the hype a little bit.

1462
01:16:28,935 --> 01:16:31,335
we'll be at Backstage Con,
and I'll see both of you there

1463
01:16:31,385 --> 01:16:31,595
Nirmal Mehta: Yep.

1464
01:16:31,736 --> 01:16:32,246
Laura Tacho: exciting.

1465
01:16:32,821 --> 01:16:34,111
Bret: that's gonna be an exciting talk.

1466
01:16:34,111 --> 01:16:38,821
we talked the last, KubeCon we were hoping
for more talks, this turnaround on the

1467
01:16:38,821 --> 01:16:43,741
actual using of AI for DevOps and platform
engineering rather than the running of

1468
01:16:43,741 --> 01:16:46,801
AI and GPUs which is what everything's
been focused on for three years.

1469
01:16:46,851 --> 01:16:48,491
it'll be fun to see that kind of stuff.

1470
01:16:48,790 --> 01:16:49,140
Nirmal Mehta: Yep.

1471
01:16:49,161 --> 01:16:49,651
Laura Tacho: totally.

1472
01:16:49,711 --> 01:16:50,821
Bret: where can people find you?

1473
01:16:51,690 --> 01:16:52,370
Nirmal Mehta: with you, Bret.

1474
01:16:52,470 --> 01:16:56,430
so Agentic DevOps fm, our podcast
that you're listening to right now.

1475
01:16:56,680 --> 01:16:58,060
on top of that, I'm on LinkedIn.

1476
01:16:58,090 --> 01:17:04,440
Metaverse hq, on Instagram
and Blue Sky, and then, mostly

1477
01:17:04,440 --> 01:17:06,080
on LinkedIn, Nirmal K Mehta.

1478
01:17:06,150 --> 01:17:06,630
you'll find me.

1479
01:17:06,940 --> 01:17:08,110
Bret: Yeah, we're all on LinkedIn.

1480
01:17:08,111 --> 01:17:08,311
Laura Tacho: kind.

1481
01:17:08,579 --> 01:17:09,029
Nirmal Mehta: Yes.

1482
01:17:09,101 --> 01:17:09,391
Laura Tacho: Yeah.

1483
01:17:09,400 --> 01:17:12,850
Bret: LinkedIn as much as anywhere
else now, so yeah, it is what it is.

1484
01:17:13,290 --> 01:17:14,730
well, Laura, thank you
so much for being here.

1485
01:17:14,730 --> 01:17:17,880
We're gonna have you back, actually,
this is a spoiler that we already are

1486
01:17:17,880 --> 01:17:21,230
planning about maybe another episode,
but we won't talk about that yet.

1487
01:17:21,230 --> 01:17:22,910
We'll talk about when it actually happens.

1488
01:17:23,160 --> 01:17:24,540
I'm excited to have you on this show.

1489
01:17:24,540 --> 01:17:28,140
We're so glad to have you back and
I love this new focus of yours.

1490
01:17:28,284 --> 01:17:28,504
Nirmal Mehta: Yes.

1491
01:17:28,680 --> 01:17:31,550
Bret: Yeah, we were on the container
bandwagon for a while, but I think

1492
01:17:31,600 --> 01:17:34,620
this might just be your moment
and, Yeah, we're here for it.

1493
01:17:34,849 --> 01:17:35,509
Nirmal Mehta: we're here for it.

1494
01:17:35,843 --> 01:17:36,473
Thanks, Laura.

1495
01:17:36,739 --> 01:17:37,159
Bret: All right.

1496
01:17:37,815 --> 01:17:38,365
Laura Tacho: Thanks, Nirmal.

1497
01:17:38,385 --> 01:17:38,935
Thanks, Bret.