AI-Native Engineering Culture from a CTO

1
00:00:00,000 --> 00:00:04,799
Michael Rollins: The big milestones
for us were we started from, you

2
00:00:04,799 --> 00:00:06,889
know, everybody's using AI to code.

3
00:00:06,989 --> 00:00:11,359
We moved into, " Hey, "When's the last
time any of y'all actually looked at

4
00:00:11,410 --> 00:00:16,215
the code?" And all my team was like,
like, everybody didn't wanna get caught.

5
00:00:16,235 --> 00:00:19,245
I was like, "Oh, 'cause I haven't
looked at it in months." Yeah,

6
00:00:19,245 --> 00:00:20,595
and they were like, " Oh, God.

7
00:00:20,615 --> 00:00:21,085
Okay.

8
00:00:21,245 --> 00:00:21,355
Phew.

9
00:00:21,415 --> 00:00:21,895
Thank you.

10
00:00:21,915 --> 00:00:22,355
Thank you."

11
00:00:22,455 --> 00:00:25,735
Bret Fisher: imposter syndrome that
jumps out when the junior-est member

12
00:00:25,735 --> 00:00:29,725
of the team posts something in the
engineering Slack that calls out

13
00:00:29,855 --> 00:00:33,555
something the most advanced engineer's
done, and you're like, " Oh, God, I'm

14
00:00:33,555 --> 00:00:35,365
not even a real engineer." Like, how…

15
00:00:35,395 --> 00:00:38,415
And then you're like, "Oh, wait, you
had Claude look at that, didn't you?"

16
00:00:38,415 --> 00:00:39,655
And they're like, "Actually, no.

17
00:00:39,665 --> 00:00:40,855
It was Codex." You know?

18
00:00:44,614 --> 00:00:48,964
Bret: Welcome back to the Agentic DevOps
podcast, and I'm your host Bret Fisher.

19
00:00:48,974 --> 00:00:53,924
I have my friend Mike Rollins on we went
to lunch, and the way he talked about

20
00:00:53,934 --> 00:00:59,447
his AI-first engineering or agentic
engineering or agentic dev, what-

21
00:00:59,447 --> 00:01:02,704
whatever you wanna call this thing,
where you're throwing tokens mostly

22
00:01:02,704 --> 00:01:05,694
and not writing anything anymore, and
you're not even looking at code anymore.

23
00:01:06,024 --> 00:01:09,604
And he runs a team as a CTO, a small
team in a startup, and I wanted to

24
00:01:09,604 --> 00:01:12,014
talk about what that life is like.

25
00:01:12,124 --> 00:01:15,914
'Cause a lot of us, you know, we-
we're in enterprise or we're, I

26
00:01:15,914 --> 00:01:20,314
would call, like, legacy teams
where, we're toying around with AI.

27
00:01:20,324 --> 00:01:24,434
We're certainly using it for code,
but we're not necessarily riddling

28
00:01:24,444 --> 00:01:28,764
the entire pipeline of the software
life cycle with everything AI so

29
00:01:28,764 --> 00:01:32,134
that we never see code again and that
everything just goes as fast as possible.

30
00:01:32,344 --> 00:01:35,094
Like, most of us are somewhere in there
in the middle or near the beginning,

31
00:01:35,094 --> 00:01:39,294
I think, and certainly with a lot of
the people I talk to, where they aren't

32
00:01:39,344 --> 00:01:43,854
necessarily able to throw out the playbook
that we've had for the last 20 years on

33
00:01:43,864 --> 00:01:49,974
how exactly a team uses Git storage and
automated deployments and test-driven

34
00:01:49,974 --> 00:01:54,604
development and all the things that
maybe, were hot topics before AI, right?

35
00:01:54,804 --> 00:01:57,954
So Mike and I go through a pretty
lengthy conversation, but I wanted to

36
00:01:57,954 --> 00:02:01,794
get as detailed as possible to talk
about the culture, the pros and cons.

37
00:02:01,794 --> 00:02:03,144
Like, what are the downsides of this?

38
00:02:03,284 --> 00:02:08,574
Tell me some horror stories of how this
can go wrong in so many ways and what it's

39
00:02:08,574 --> 00:02:13,414
like trying to push the envelope, being on
the bleeding edge of AI-first development.

40
00:02:13,514 --> 00:02:14,354
So let's get into it

41
00:02:14,374 --> 00:02:15,714
Bret Fisher: Mike, welcome to the show.

42
00:02:15,814 --> 00:02:20,984
t- tell the audience, who you
are and what you do for a living.

43
00:02:21,084 --> 00:02:23,654
Michael Rollins: So, uh,
my name is Mike Rollins.

44
00:02:23,884 --> 00:02:26,384
the majority of my career
has been in Silicon Valley.

45
00:02:26,454 --> 00:02:30,864
back in '22, Our company was getting
bought by Twitter, and then this

46
00:02:30,864 --> 00:02:35,154
dude named Elon jumps in, and we were
no longer being bought by Twitter.

47
00:02:35,254 --> 00:02:42,004
And as a result, I went on this multi-year
escapade of, I worked for a healthcare

48
00:02:42,004 --> 00:02:46,884
startup, and then after the health- I left
the healthcare startup, or I was asked

49
00:02:46,894 --> 00:02:50,364
to leave the healthcare startup, because
it turns out if you ask very difficult

50
00:02:50,394 --> 00:02:52,794
questions pointedly, people don't like it.

51
00:02:52,894 --> 00:02:58,294
I ended up, going o- out on my
own, and it was terrifying, scary.

52
00:02:58,394 --> 00:03:03,504
I mean, from about 2023 until really
November of last year, I was a

53
00:03:03,504 --> 00:03:05,744
contractor, and I did a bunch of stuff.

54
00:03:05,744 --> 00:03:08,274
I was the founding engineer
for multiple projects.

55
00:03:08,294 --> 00:03:11,624
I was this… I've been the CTO
now of three different companies,

56
00:03:11,634 --> 00:03:13,574
though mostly fractional.

57
00:03:13,674 --> 00:03:19,894
And I have at some point, the internal
clock in my head is always ticking, and

58
00:03:19,894 --> 00:03:22,414
I feel this constant need to go faster.

59
00:03:22,514 --> 00:03:28,854
And so as I was building things as
typically the only engineer, the clock in

60
00:03:28,854 --> 00:03:31,434
my head is just faster, faster, faster.

61
00:03:31,434 --> 00:03:38,004
And so when AI became, like, a thing,
meaning back in the latter half

62
00:03:38,004 --> 00:03:42,504
of 2024 and then especially at the
beginning of 2025, myself and a few

63
00:03:42,504 --> 00:03:46,714
other folks, a guy named Thanos, who
he and I have a podcast together that

64
00:03:46,714 --> 00:03:50,284
we're kicking back off called Will
AI Snap That, we both just dove in.

65
00:03:50,384 --> 00:03:55,414
And what's funny is that since
about really the entry of Claude

66
00:03:55,414 --> 00:03:59,664
Code into the market, I have
been doing AI first engineering.

67
00:03:59,764 --> 00:04:06,264
And what I had noticed is that by the
time we hit about May or June of last

68
00:04:06,264 --> 00:04:08,974
year, I was no longer writing code.

69
00:04:09,074 --> 00:04:13,934
And then right around November
of this year, I realized I was

70
00:04:13,934 --> 00:04:18,074
no longer reading code And it
was this, shocking realization.

71
00:04:18,174 --> 00:04:25,764
And so I basically dove headfirst into AI
j- and, you know, high risk, high reward.

72
00:04:25,784 --> 00:04:28,794
The risk here is that my
family can't eat, right?

73
00:04:28,894 --> 00:04:33,354
The reward is that I figure out a way
to do engineering that other people

74
00:04:33,354 --> 00:04:34,854
don't know, and it's what we did.

75
00:04:34,954 --> 00:04:38,604
And so nowadays I'm at this
company called, Relefy and

76
00:04:38,604 --> 00:04:39,874
then also Family Digital.

77
00:04:39,884 --> 00:04:41,014
I'm CTO of both.

78
00:04:41,114 --> 00:04:45,824
And I have this team of four other
engineers, and really truthfully,

79
00:04:45,874 --> 00:04:49,784
only one of them, well, one of them
is a designer that we've turned into

80
00:04:49,784 --> 00:04:54,064
a front-end engineer, and none of
us read code, none of us write code.

81
00:04:54,164 --> 00:04:58,694
we ship… We've built this agentic
system that's built from the ground

82
00:04:58,694 --> 00:05:03,454
up for sharing, and it is kind of,
it's built at right angles to the way

83
00:05:03,454 --> 00:05:08,024
everybody else is doing things, and we
are moving so fast, it's terrifying.

84
00:05:08,124 --> 00:05:12,294
But the clock continues to tick, and I
continue to think, "How do we go faster,

85
00:05:12,294 --> 00:05:17,124
faster, faster, faster?" But I, dude,
never in my life… I got to write mobile.

86
00:05:17,224 --> 00:05:22,724
In 2011, I started writing code for iOS
and then Android, and mobile felt amazing.

87
00:05:22,734 --> 00:05:24,214
It felt like a freaking rocket ship.

88
00:05:24,314 --> 00:05:29,034
Everything changed so fast because there
was a new phone about every three months.

89
00:05:29,134 --> 00:05:34,034
And now it's dude, didn't we just see a
new Frontier model a week and a half ago?

90
00:05:34,094 --> 00:05:36,814
'Cause we did, and it's just unbelievable.

91
00:05:36,874 --> 00:05:39,324
So anyway, that's how I ended up here.

92
00:05:39,424 --> 00:05:40,644
Bret Fisher: Yeah, that's a great story.

93
00:05:40,654 --> 00:05:43,554
And, I was on the hardware side
of mobile, not the software side.

94
00:05:43,554 --> 00:05:47,104
So, yeah, it has become very apparent to
me in the last few years that it's like

95
00:05:47,204 --> 00:05:50,814
AI, that's a good analogy of like AI feels
like it's kind of replaced mobile as the

96
00:05:50,814 --> 00:05:53,044
exciting tech where that was really…

97
00:05:53,054 --> 00:05:55,924
I mean, mobile kind of came around a
little bit a few years after cloud,

98
00:05:55,934 --> 00:05:59,814
but cloud I feel like was pretty well
understood within four or five years and

99
00:05:59,814 --> 00:06:02,984
then we had mobile and that was exciting,
and then there was like this lull.

100
00:06:03,134 --> 00:06:06,994
I kind of feel like 2018 to 2022
was kind of this lull of, we

101
00:06:07,042 --> 00:06:07,712
Michael Rollins: Yes, it was.

102
00:06:07,812 --> 00:06:09,232
Bret Fisher: yeah, we were
all sort of exhibiting the

103
00:06:09,320 --> 00:06:10,180
Michael Rollins: was
supposed to be crypto.

104
00:06:10,280 --> 00:06:10,870
Bret Fisher: Oh, that's true.

105
00:06:10,880 --> 00:06:13,380
Yeah, the vacuum of crypto, which,
you know, and I don't know anyone

106
00:06:13,380 --> 00:06:14,620
that touches it with a 10-foot pole.

107
00:06:14,650 --> 00:06:18,410
So, yeah, that was a thing, but
not necessarily in my ecosystem.

108
00:06:18,410 --> 00:06:20,860
So we all were talking about
how isn't it great that Docker

109
00:06:20,860 --> 00:06:21,780
and Kubernetes are boring?

110
00:06:21,780 --> 00:06:24,860
Isn't it great that like microservices
and distributed services and cloud

111
00:06:24,860 --> 00:06:28,810
tech are all kind of boring and, we can
just focus on the work and, enhancing

112
00:06:28,810 --> 00:06:30,090
our craft and getting better at it?

113
00:06:30,090 --> 00:06:32,950
And then, you know, ChatGPT shows
up and then we're all copy-pasting.

114
00:06:33,050 --> 00:06:36,080
So I wanted to get into some of
the nerdy details of… it sounds

115
00:06:36,080 --> 00:06:39,420
like it's a relatively small team,
so a total of five it sounds like.

116
00:06:39,420 --> 00:06:40,430
There's a total of five of you.

117
00:06:40,430 --> 00:06:44,610
So does that mean you have five people
making PRs into, presumably GitHub?

118
00:06:45,542 --> 00:06:45,812
Michael Rollins: Mm-hmm.

119
00:06:45,912 --> 00:06:48,202
Bret Fisher: And how do you go
about even starting something?

120
00:06:48,202 --> 00:06:51,122
Is it like an individual
thing where you start an idea?

121
00:06:51,122 --> 00:06:55,042
you mentioned this idea of concepts first
before ideas or something like that.

122
00:06:56,104 --> 00:06:58,584
Bret AI July 2025: Have you been
adding text-to-speech in your apps?

123
00:06:58,754 --> 00:07:01,874
One of my late-night projects
this year was creating a personal

124
00:07:01,874 --> 00:07:05,914
podcast feed so I could listen to
the changelog of my favorite open

125
00:07:05,914 --> 00:07:07,594
source projects while I do my chores.

126
00:07:07,994 --> 00:07:10,214
I needed a text-to-speech API to do that.

127
00:07:10,624 --> 00:07:13,964
I also heard from a colleague this
week who built an app for his kids to

128
00:07:13,974 --> 00:07:18,244
turn their e-books into audiobooks, and
he's using the Speechify AI developer

129
00:07:18,244 --> 00:07:20,764
platform to do that text-to-speech

130
00:07:20,764 --> 00:07:24,754
Speechify AI is a developer platform
for real-time text-to-speech, and

131
00:07:24,754 --> 00:07:26,274
they make their own voice models.

132
00:07:26,504 --> 00:07:31,504
Their Simba 3.2 model ranked number one
on the Artificial Analysis Benchmark

133
00:07:31,734 --> 00:07:33,324
for conversational voice quality.

134
00:07:33,574 --> 00:07:37,984
They've got a free plan, streaming
and REST APIs, TypeScript and Python

135
00:07:38,014 --> 00:07:42,884
SDKs, and if you wanna get fancy, you
can use their SSML metadata support

136
00:07:42,894 --> 00:07:44,954
to add emotion into the speech output

137
00:07:45,868 --> 00:07:49,698
Whether you're shipping a weekend project
or production voice agents at work,

138
00:07:49,718 --> 00:07:52,328
you can start free at speechify.ai.

139
00:07:52,648 --> 00:07:53,778
The link is in the show notes.

140
00:07:54,248 --> 00:07:56,728
Thanks so much to Speechify AI
for sponsoring this episode.

141
00:07:57,690 --> 00:07:59,800
Michael Rollins: well, let
me start at the end for you

142
00:07:59,900 --> 00:08:00,220
Bret Fisher: Okay

143
00:08:00,320 --> 00:08:03,650
Michael Rollins: because this is typically
the framing… This is the framing I've

144
00:08:03,650 --> 00:08:05,780
been aiming for, and we're almost there.

145
00:08:05,880 --> 00:08:12,440
What does it take for me to allow
an agent to pick up something, build

146
00:08:12,440 --> 00:08:17,090
it, test it, commit it, open the PR,
make sure everything runs, listen to

147
00:08:17,090 --> 00:08:20,890
all the feedback, merge it to main,
and deploy it, and a human is never

148
00:08:20,890 --> 00:08:24,320
involved, and as a matter of fact, a
human may never know that it happened?

149
00:08:24,470 --> 00:08:27,080
What do you have to do to make that real?

150
00:08:27,750 --> 00:08:30,990
And risk is really the big asterisk there.

151
00:08:31,090 --> 00:08:36,550
And I, you know, as the clock is
ticking in my head, I, I make decisions

152
00:08:36,570 --> 00:08:38,270
based on a risk assessment, right?

153
00:08:38,320 --> 00:08:43,720
What is the likelihood of failure
versus what is the outcome if we fail?

154
00:08:43,820 --> 00:08:48,890
And what you find is that even if the
likelihood of failure is high, but the

155
00:08:48,890 --> 00:08:54,380
risk associated with the actual failure
is low, you may choose to take the gamble.

156
00:08:54,480 --> 00:09:00,680
Conversely, if the likelihood of
failure is very low, but the risk

157
00:09:00,780 --> 00:09:04,920
is extraordinarily high, you're
pro- you may not take that leap.

158
00:09:05,020 --> 00:09:06,050
It's not a gamble.

159
00:09:06,090 --> 00:09:10,889
It's almost a guarantee, but you know
that if you hit the Enter on this

160
00:09:10,890 --> 00:09:15,699
select statement in the production
database, and it goes sideways, like

161
00:09:15,709 --> 00:09:19,770
the failure mode could be the database
disappears, and then so does the company.

162
00:09:19,870 --> 00:09:23,019
And that's one of those things
where it's like I've made that

163
00:09:23,019 --> 00:09:26,059
decision and had it go bad, right?

164
00:09:26,059 --> 00:09:28,829
I've made… I have wiped
the company database

165
00:09:28,929 --> 00:09:31,979
Bret Fisher: Oh, are you the one that's
like on the tip of the spear just so the

166
00:09:31,979 --> 00:09:33,689
rest of us can learn from your failures?

167
00:09:33,789 --> 00:09:36,519
Michael Rollins: when I was at Outlier,
we did machine learning, over time

168
00:09:36,519 --> 00:09:39,179
series, and this was in 2018 to 2022.

169
00:09:39,279 --> 00:09:41,369
That was the company that was
gonna get bought by, Twitter.

170
00:09:41,469 --> 00:09:41,769
Bret Fisher: Yeah

171
00:09:41,869 --> 00:09:45,669
Michael Rollins: I was on call for three
month- for three year straight, and…

172
00:09:45,729 --> 00:09:50,279
I wrote direct queries in production
databases constantly, ' cause it

173
00:09:50,279 --> 00:09:53,209
was the only way that we could keep
the system up and running sometimes.

174
00:09:53,309 --> 00:09:56,929
And I would wake up every morning and I
would rebuild the system, and there were

175
00:09:56,929 --> 00:10:00,189
a couple times that, boy, did I that up.

176
00:10:00,289 --> 00:10:05,209
even in a couple, not the current company,
but the previous one, there were a

177
00:10:05,209 --> 00:10:07,689
couple times I just, I wasn't thinking.

178
00:10:07,789 --> 00:10:12,489
I knew I was doing something a little
bit risky, and I did it anyways, and

179
00:10:12,499 --> 00:10:17,039
had to go desperately hope that we
had a recovery copy stashed away.

180
00:10:17,139 --> 00:10:17,779
and we did.

181
00:10:17,879 --> 00:10:21,019
And so I have legitimately,
I've blown it all up.

182
00:10:21,119 --> 00:10:24,920
I'm very risk-tolerant in that aspect,
but the reason I'm risk-tolerant is

183
00:10:25,109 --> 00:10:30,530
I fixed it And so when I look at…
let me give this as an example.

184
00:10:30,630 --> 00:10:34,550
we had an incident occur a few
weeks ago, and all of a sudden I

185
00:10:34,550 --> 00:10:38,910
realized that we could no longer
upload files to our system.

186
00:10:38,910 --> 00:10:41,149
And now keep in mind,
it's an agentic system.

187
00:10:41,180 --> 00:10:42,779
You create a bunch of agents.

188
00:10:42,779 --> 00:10:44,190
They can all talk to each other.

189
00:10:44,199 --> 00:10:47,970
It's a lot like OpenClaw, except
we were doing it first, and we

190
00:10:47,970 --> 00:10:49,970
built it to be enterprise safe.

191
00:10:50,070 --> 00:10:54,218
And part of that is, you have to upload
things, and they get shoved into the RAG

192
00:10:54,218 --> 00:10:58,068
database, and they get shoved into the
S3 file system, and I tried to move a

193
00:10:58,068 --> 00:10:59,808
bunch of things around, and I couldn't.

194
00:10:59,908 --> 00:11:04,058
I was like, "What is going on?" Now,
I'm at my daughter's dance competition.

195
00:11:04,158 --> 00:11:06,198
I don't even have my computer with me.

196
00:11:06,298 --> 00:11:09,958
I'm the only engineer that
can fix this at this moment.

197
00:11:10,058 --> 00:11:14,338
And so I'm actually on my phone,
so I swipe over to Claude Code.

198
00:11:14,438 --> 00:11:17,168
I have a session running
on my machine at work.

199
00:11:17,168 --> 00:11:21,468
I tap into it, and I say, "I want
you to you know, clear your context.

200
00:11:21,498 --> 00:11:22,828
I want you to pull main.

201
00:11:22,908 --> 00:11:26,488
I want you to get to a new working
tree, and I want you to help me diagnose

202
00:11:26,488 --> 00:11:31,628
what this problem is." Within about
four minutes, I knew that the disks

203
00:11:31,648 --> 00:11:37,748
that were, serving the ECS, the EC2
or ECS images were full, ' cause we

204
00:11:37,748 --> 00:11:43,008
had not been pruning them after we
would deploy. And I had Claude go in,

205
00:11:43,038 --> 00:11:48,228
'cause it also has my AWS credentials
as part of it, and said, " Go fix it."

206
00:11:48,358 --> 00:11:49,558
It went ahead and fixed it.

207
00:11:49,658 --> 00:11:51,398
I didn't even have to restart the system.

208
00:11:51,398 --> 00:11:54,698
I had the whole thing back up and
running correctly in five minutes.

209
00:11:54,798 --> 00:11:55,158
Bret Fisher: Yeah.

210
00:11:55,294 --> 00:11:56,194
Michael Rollins: The cost

211
00:11:56,338 --> 00:11:58,088
Bret Fisher: basically, or
what- whatever, whatever

212
00:11:58,088 --> 00:12:00,988
Michael Rollins: yeah, whatever the
container service equivalent was.

213
00:12:00,988 --> 00:12:02,088
But that was it, right?

214
00:12:02,188 --> 00:12:05,228
And well, the risk was actually
pretty low, and the likelihood of

215
00:12:05,228 --> 00:12:10,038
failure was almost minuscule because
I knew that I could do everything

216
00:12:10,048 --> 00:12:13,888
from the command line on my machine
through the app that was on my phone.

217
00:12:13,988 --> 00:12:17,248
And so the risk profiles of
everything are just different now.

218
00:12:17,348 --> 00:12:21,628
so when I sit down and I think about
what does it take to allow an agent

219
00:12:21,728 --> 00:12:25,928
to write code and ship it without
me ever knowing, there's an inherent

220
00:12:25,948 --> 00:12:31,068
assumption built in that any failure
state is something I can recover from

221
00:12:31,068 --> 00:12:32,838
in a relatively short amount of time.

222
00:12:32,938 --> 00:12:37,428
And now there's an extraordinary number
of assumptions that are involved there,

223
00:12:37,528 --> 00:12:40,468
and it's not like this is YOLO, right?

224
00:12:40,568 --> 00:12:44,688
Our build infrastructure, our GitHub
actions, everything that happens

225
00:12:44,708 --> 00:12:48,568
post-commit is methodical, it's intense.

226
00:12:48,668 --> 00:12:51,618
going back to what was your
initial question, we don't

227
00:12:51,628 --> 00:12:54,468
do tickets, we do features.

228
00:12:54,568 --> 00:12:58,788
And oftentimes because of the
speed with which we move, we…

229
00:12:58,888 --> 00:13:00,838
There's not even time to build a UI.

230
00:13:00,938 --> 00:13:03,608
Our designer is often
catching up to engineering.

231
00:13:03,618 --> 00:13:06,708
We have officially broken the
entire organization because

232
00:13:06,708 --> 00:13:08,538
engineering moves faster than sales.

233
00:13:08,638 --> 00:13:12,198
And that is a wild thing to
deal with on a business level.

234
00:13:12,298 --> 00:13:12,548
like

235
00:13:12,648 --> 00:13:14,038
Bret Fisher: that's when
people get fired, right?

236
00:13:14,038 --> 00:13:17,738
Because there's too many engineers for the
sales team and… But that so rarely ever

237
00:13:17,738 --> 00:13:19,258
happened 'cause there was always a gap.

238
00:13:19,308 --> 00:13:19,648
Michael Rollins: Yeah.

239
00:13:19,768 --> 00:13:23,848
I mean, the, typically the lead time
from sales to engineering is six months.

240
00:13:23,948 --> 00:13:24,348
Bret Fisher: Yeah.

241
00:13:24,448 --> 00:13:25,328
What do I tell the customer?

242
00:13:25,328 --> 00:13:25,738
We need them,

243
00:13:25,750 --> 00:13:27,080
Michael Rollins: Yeah, what
do I tell the customer?

244
00:13:27,180 --> 00:13:28,130
Bret Fisher: we need the feature." Yeah.

245
00:13:28,230 --> 00:13:28,470
Michael Rollins: yeah.

246
00:13:28,570 --> 00:13:30,580
And I'm like, " It's already there.

247
00:13:30,600 --> 00:13:33,590
Just sign the contract." they
were like, "It wasn't there two

248
00:13:33,590 --> 00:13:35,380
days ago." I was like, "I know.

249
00:13:35,380 --> 00:13:36,320
We shipped it."

250
00:13:36,420 --> 00:13:37,400
Bret Fisher: They're so not used to that.

251
00:13:37,400 --> 00:13:40,190
Yeah, especially if they're old school
and they're used to the giant lags

252
00:13:40,290 --> 00:13:40,590
Michael Rollins: Yeah.

253
00:13:40,690 --> 00:13:43,640
Well, and, you know, so the, one
thing to say is the mentality of

254
00:13:43,640 --> 00:13:45,120
engineers has to be totally different.

255
00:13:45,130 --> 00:13:49,540
My engineers have to understand
the business, and I don't let my

256
00:13:49,540 --> 00:13:52,000
engineers just focus on tickets.

257
00:13:52,030 --> 00:13:54,530
As a matter of fact, that's
an amazing wayl- way to fail

258
00:13:54,530 --> 00:13:55,870
catastrophically at this point.

259
00:13:55,970 --> 00:14:00,600
instead, I say, " Here's a thing,
and I want you to understand

260
00:14:00,600 --> 00:14:01,810
it and I want you to do it."

261
00:14:01,910 --> 00:14:05,180
And the engineers that are working for
me at this point, which by the way, none

262
00:14:05,180 --> 00:14:08,750
of them have professional experience
before they started working with me,

263
00:14:08,850 --> 00:14:12,860
these folks are making really strong
judgment calls, and that's all I want.

264
00:14:12,960 --> 00:14:14,750
I want good judgment, right?

265
00:14:14,800 --> 00:14:18,110
The model can, and the harness can
figure out how to write the code.

266
00:14:18,210 --> 00:14:21,030
Bret Fisher: So is there a large
part, is a portion of your training

267
00:14:21,040 --> 00:14:25,200
or cross-training in the team
helping everyone with judgment?

268
00:14:25,300 --> 00:14:27,090
is that become an increased thing?

269
00:14:27,100 --> 00:14:30,140
'cause since you're going so much faster,
you, there's less time to talk about

270
00:14:30,150 --> 00:14:32,550
things, so people have to be able to
make more decisions on their own, which

271
00:14:32,560 --> 00:14:34,360
means they need more agency, which, yeah.

272
00:14:34,460 --> 00:14:34,780
Yeah.

273
00:14:34,904 --> 00:14:37,094
Michael Rollins: There's one amazing
way to let people build better

274
00:14:37,094 --> 00:14:38,534
judgment, and you let them fail,

275
00:14:38,634 --> 00:14:38,884
Bret Fisher: Mm-hmm.

276
00:14:38,944 --> 00:14:41,984
Michael Rollins: we shipped something
this morning, and, I had actually

277
00:14:41,984 --> 00:14:45,594
started work on it and then had
to move on to something different.

278
00:14:45,594 --> 00:14:50,964
And so Miko, who's a official… He would
be classified a junior engineer, right?

279
00:14:51,064 --> 00:14:55,664
Miko will have been out of college for
approximately a year in the next few days.

280
00:14:55,764 --> 00:14:59,884
Miko's making decisions at a senior level,
but he shipped something this morning,

281
00:14:59,984 --> 00:15:02,904
and he overlooked something pretty major.

282
00:15:03,004 --> 00:15:03,774
Pretty major.

283
00:15:03,794 --> 00:15:03,814
Yeah.

284
00:15:03,914 --> 00:15:07,114
And I halfway knew it was gonna
happen, and I let it ship anyways.

285
00:15:07,214 --> 00:15:10,624
And, that is one of the things
that I allow my team to fail.

286
00:15:10,724 --> 00:15:16,164
I will let the problem hit production, is
once again, it's a risk assessment, right?

287
00:15:16,194 --> 00:15:18,734
Because you've done it and I've done it.

288
00:15:18,834 --> 00:15:22,964
There's nothing quite exhil- as
exhilarating as realizing you

289
00:15:22,994 --> 00:15:25,674
just shipped a nasty bug, man.

290
00:15:25,734 --> 00:15:29,934
everything is being destroyed,
and it's your fault.

291
00:15:30,034 --> 00:15:33,244
Bret Fisher: It, that is a
permanently ingrained memory that

292
00:15:33,244 --> 00:15:35,094
you cannot escape and forget.

293
00:15:35,114 --> 00:15:39,094
it becomes a part of who you are, of
your identity whenever you do that,

294
00:15:39,144 --> 00:15:41,194
Michael Rollins: and so my
leadership style actually

295
00:15:41,254 --> 00:15:42,884
relies pretty heavily on that.

296
00:15:42,954 --> 00:15:47,274
I know that I'm tossing people
in over their heads, and I do.

297
00:15:47,374 --> 00:15:51,684
And, but I also tell them, " Hey, listen,
you're probably gonna fuck this up.

298
00:15:51,784 --> 00:15:52,214
Bret Fisher: Yeah

299
00:15:52,340 --> 00:15:53,610
Michael Rollins: And when
you do, I'll be here.

300
00:15:53,710 --> 00:15:54,590
You just come find me.

301
00:15:54,700 --> 00:15:55,490
There's no blame.

302
00:15:55,590 --> 00:15:57,070
I know you're in over your head.

303
00:15:57,180 --> 00:16:00,950
You know you're in over your head, and
I happen to know that there's sharks

304
00:16:00,950 --> 00:16:03,230
and they have lasers and you don't yet."

305
00:16:03,330 --> 00:16:03,750
Bret Fisher: Yeah.

306
00:16:03,830 --> 00:16:04,210
Michael Rollins: so,

307
00:16:04,210 --> 00:16:05,040
Bret Fisher: it's challenging.

308
00:16:05,040 --> 00:16:08,730
so yeah, you're giving… You're
challenging people to grow and rather

309
00:16:08,730 --> 00:16:12,840
than coddling them through every
mistake that they'll never remember,

310
00:16:12,840 --> 00:16:15,960
because I'm sure if you told them,
walked by the desk or, the equivalent,

311
00:16:16,010 --> 00:16:18,720
the analogy of walking by the desk
and going, "Oh, what you working on?

312
00:16:18,820 --> 00:16:21,110
Oh yeah, you probably wanna check
this so that doesn't fail 'cause that

313
00:16:21,110 --> 00:16:23,590
would've totally broke production."
They may not even remember that in

314
00:16:23,590 --> 00:16:25,160
five months or, you know, two months.

315
00:16:25,160 --> 00:16:27,730
So, but basically the boss just told
them a rule, but that's, they don't

316
00:16:27,730 --> 00:16:29,030
really maybe know the deep why.

317
00:16:29,130 --> 00:16:29,920
I'm gonna ask real quick.

318
00:16:29,980 --> 00:16:34,360
is the assumption here that you have
solid, It sounds like it got past

319
00:16:34,370 --> 00:16:38,320
testing in this scenario, so that
means that the testing was lacking

320
00:16:38,320 --> 00:16:39,640
for that particular assumption.

321
00:16:39,740 --> 00:16:40,450
And then

322
00:16:40,534 --> 00:16:40,924
Michael Rollins: it wasn't,

323
00:16:41,024 --> 00:16:43,614
Bret Fisher: also assume, by the
way, that production can, fail back?

324
00:16:43,644 --> 00:16:46,614
do you need… In order to give you
the confidence that this isn't going

325
00:16:46,614 --> 00:16:50,294
to ruin the company, do you have
to have, some other structures to

326
00:16:50,304 --> 00:16:52,024
be able to roll back automatically?

327
00:16:52,024 --> 00:16:54,124
Or, is there stuff like that
in play that you're relying on?

328
00:16:54,224 --> 00:16:56,344
Michael Rollins: So in this
instance, it actually wasn't a bug.

329
00:16:56,444 --> 00:16:58,794
It was a missed part of the feature.

330
00:16:58,894 --> 00:17:00,904
so this was a little bit unique.

331
00:17:00,954 --> 00:17:03,194
But honestly, I mean, that's okay.

332
00:17:03,294 --> 00:17:04,144
Here's the interesting thing.

333
00:17:04,164 --> 00:17:07,284
It forced us into the decision of
whether or not we wanted to address

334
00:17:07,284 --> 00:17:08,904
it, and we decided not to address it.

335
00:17:09,004 --> 00:17:12,224
And yeah, you end up ha- you
find yourself in these situations

336
00:17:12,234 --> 00:17:17,214
sometimes where because you can't
necessarily cover all your bases,

337
00:17:17,214 --> 00:17:20,304
you find yourself making decisions
you might not have made initially.

338
00:17:20,404 --> 00:17:23,664
In this instance, we were basically
changing the ownership of a certain

339
00:17:23,684 --> 00:17:25,454
type of object within an agent.

340
00:17:25,464 --> 00:17:29,084
So previously, it had belonged
to the user, and we realized it

341
00:17:29,084 --> 00:17:30,394
needed to belong to the agent.

342
00:17:30,494 --> 00:17:32,564
And so we fixed that, right?

343
00:17:32,604 --> 00:17:34,684
Now all of a sudden these
things belong to the agent.

344
00:17:34,694 --> 00:17:38,804
But there were a bunch of them, and they
previously belonged to the user, so we had

345
00:17:38,814 --> 00:17:41,684
to make the user's core agent own them.

346
00:17:41,784 --> 00:17:46,114
But if you were using a bunch of
these things in another agent, they

347
00:17:46,114 --> 00:17:50,334
were no longer there And it's "Oh,
there's no elegant solution for this.

348
00:17:50,434 --> 00:17:55,594
So how do we get things from the
PR to production, I think is a

349
00:17:55,594 --> 00:17:57,394
great place to really hone in.

350
00:17:57,494 --> 00:18:01,994
And the first thing you gotta know is
that whenever our, in our environment,

351
00:18:02,094 --> 00:18:06,074
whenever you issue a PR, we stand
up a copy of prod for the whole PR.

352
00:18:06,174 --> 00:18:09,974
… it is a somewhat expensive way to
run things, and it also catches

353
00:18:09,974 --> 00:18:11,464
so many things immediately.

354
00:18:11,564 --> 00:18:15,014
now at the same time that we're
standing up that copy of production,

355
00:18:15,114 --> 00:18:16,944
and this includes the database.

356
00:18:17,134 --> 00:18:18,624
This is everything, right?

357
00:18:18,624 --> 00:18:20,294
It is a true copy of production.

358
00:18:20,394 --> 00:18:26,554
we have just a boatload of tests
and checks and all kinds of things.

359
00:18:26,554 --> 00:18:30,844
I think last time I looked,
there's 15 or 16 different

360
00:18:30,874 --> 00:18:32,894
workflows that get kicked off.

361
00:18:32,994 --> 00:18:35,434
we've got multiple, AI, audits.

362
00:18:35,454 --> 00:18:40,414
We've got, s- I mean, look, we have tests
running, I don't even know what they are.

363
00:18:40,514 --> 00:18:41,374
They just run.

364
00:18:41,474 --> 00:18:44,564
Like, Claude was like, "I don't know, we
should do this." I was like, "Excellent.

365
00:18:44,664 --> 00:18:48,204
Let's make that happen." now it took
me about a week and a half to literally

366
00:18:48,204 --> 00:18:50,984
tear our old testing infrastructure
out and put this new one in.

367
00:18:51,084 --> 00:18:52,184
but it's ratcheting.

368
00:18:52,194 --> 00:18:55,934
We have rough running to make
sure that things continue to get

369
00:18:55,964 --> 00:18:57,554
more and more, you know, covered.

370
00:18:57,564 --> 00:19:01,304
We have unit tests, we have end-to-end
tests, we have Playwright tests.

371
00:19:01,304 --> 00:19:06,154
We have, a bunch of, we have linters that
run, even though I think they're useless.

372
00:19:06,254 --> 00:19:11,914
and so, we catch a lot of things
before they even hit, before the

373
00:19:11,924 --> 00:19:14,554
production, before the, that we
call them PR environments, before

374
00:19:14,554 --> 00:19:16,034
the PR environments are built.

375
00:19:16,134 --> 00:19:18,154
We know there are problems often.

376
00:19:18,154 --> 00:19:21,014
One of them is, we double-check
that we don't end up with a,

377
00:19:21,104 --> 00:19:22,874
too many Alembic heads, right?

378
00:19:22,904 --> 00:19:23,744
I hate Alembic.

379
00:19:23,744 --> 00:19:26,934
It is literally one of the most
awful pieces of technology on the

380
00:19:26,934 --> 00:19:28,574
planet if you don't use it right.

381
00:19:28,674 --> 00:19:30,834
And so we've been forced to use it right.

382
00:19:30,934 --> 00:19:33,104
Alembic does our database
migrations, by the way.

383
00:19:33,204 --> 00:19:35,474
So that's the first
line of defense, right?

384
00:19:35,674 --> 00:19:39,604
now I will take one step back and say
that every single engineer can also

385
00:19:39,604 --> 00:19:42,994
run the entire stack on their machine
That's-- You have to start there.

386
00:19:43,064 --> 00:19:45,314
If the engineer cannot…
Yeah, table stakes.

387
00:19:45,314 --> 00:19:50,464
you know, everything in a way,
prod-production is a copy of our local

388
00:19:50,464 --> 00:19:54,374
environment because even though we don't
have the data, like I'm not gonna let

389
00:19:54,374 --> 00:19:59,114
you put production environment data on
your environment locally, you have, like,

390
00:19:59,224 --> 00:20:03,844
all the Docker images that get built on
your local also get built with different

391
00:20:03,844 --> 00:20:05,554
environment variables in production.

392
00:20:05,654 --> 00:20:08,264
And so you have to be able
to run everything locally.

393
00:20:08,274 --> 00:20:12,524
We also make sure that the model knows,
"Hey, you need to make sure that there's

394
00:20:12,534 --> 00:20:15,884
unit test coverage for everything you
do. You need to make sure that you're

395
00:20:15,884 --> 00:20:20,974
covering, you know, Playwright end-to-end
tests." I'm very, very big on… Now,

396
00:20:21,004 --> 00:20:25,364
this is because we're primarily a web app,
but I've also done it with mobile now.

397
00:20:25,464 --> 00:20:30,344
I want the model to run the actual
app locally and double-check that

398
00:20:30,344 --> 00:20:32,204
everything it built actually works.

399
00:20:32,304 --> 00:20:36,504
And so it's not just a set of tests
locally, it's also, " Hey, I want you

400
00:20:36,504 --> 00:20:41,644
to fire up an, a Playwright MCP session,
and I want you to click around and

401
00:20:41,644 --> 00:20:45,034
double-check that this actually does
what you say it does." hallucinations

402
00:20:45,034 --> 00:20:48,774
are real, and the only way to really
catch them so that they don't become

403
00:20:48,774 --> 00:20:53,064
catastrophic is to literally make the
model prove that it's not a hallucination.

404
00:20:53,164 --> 00:20:55,254
so that's our testing infrastructure.

405
00:20:55,354 --> 00:20:56,724
it's the best I've ever seen

406
00:20:56,824 --> 00:20:59,814
Bret Fisher: what are these, what are the,
are these tests, are you using Makefiles?

407
00:20:59,864 --> 00:21:02,344
are you g- you got, are you using
testing frameworks that are,

408
00:21:02,354 --> 00:21:05,284
not frameworks necessarily, but
trust testing tooling that is, y-

409
00:21:05,334 --> 00:21:08,824
Michael Rollins: we use Jest for
TypeScript, and then we use, I don't know,

410
00:21:08,824 --> 00:21:12,344
one of the Python… Our back- our entire
system is just TypeScript and Python.

411
00:21:12,444 --> 00:21:13,674
Bret Fisher: I'm a big GitHub Actions fan.

412
00:21:13,674 --> 00:21:14,994
I teach GitHub Actions courses.

413
00:21:15,034 --> 00:21:15,704
Michael Rollins: GitHub Actions.

414
00:21:15,704 --> 00:21:16,804
Everything's based on the GitHub Actions.

415
00:21:16,856 --> 00:21:19,696
Bret Fisher: and GitHub Actions
locally is often problematic

416
00:21:19,696 --> 00:21:22,806
and difficult, and I generally
don't use it even though we have,

417
00:21:22,906 --> 00:21:24,756
Michael Rollins: we don't,
I don't run actions locally.

418
00:21:24,856 --> 00:21:26,936
Yeah, I, yeah, we don't
run actions locally.

419
00:21:27,036 --> 00:21:29,546
Bret Fisher: I tend to throw stuff in
the agents file, like always run these

420
00:21:29,556 --> 00:21:32,896
three linters and then always make sure
they're in the, make a make file so that

421
00:21:32,906 --> 00:21:34,626
everything you do is through a make file.

422
00:21:34,626 --> 00:21:37,776
Because I don't know if this is
advice or a, the way that I do it, and

423
00:21:37,776 --> 00:21:39,906
I'd love for you to… your opinion
on a lot of the things I'm doing.

424
00:21:40,006 --> 00:21:44,106
because often, if I go fast
enough, it's making so much stuff.

425
00:21:44,106 --> 00:21:47,416
I built a Mac app over the last month, a
recording app because I'm so particular

426
00:21:47,416 --> 00:21:51,006
that the thousand different options for
recording screen and video is not good

427
00:21:51,006 --> 00:21:52,156
enough for me, so I had to make my own.

428
00:21:52,168 --> 00:21:53,248
Michael Rollins: I agree 100%.

429
00:21:53,641 --> 00:21:55,901
Bret Fisher: and as a part of it,
it started to have all sorts, like

430
00:21:55,901 --> 00:21:58,221
I need to, it needs to do a dev
build, and it's got a prod build,

431
00:21:58,221 --> 00:21:59,691
and it copies it into my, this place.

432
00:21:59,691 --> 00:22:02,511
And so I started to say, "Hey, look,
you're just writing a lot of commands

433
00:22:02,511 --> 00:22:04,791
for me that I don't even know what
you're doing. Could you make sure that

434
00:22:04,801 --> 00:22:06,231
all that is always in a make file?"

435
00:22:06,231 --> 00:22:08,611
And I made that a rule in my agents
file, and it started doing that.

436
00:22:08,611 --> 00:22:11,291
And then I said, "Okay, always m- run
these linters when you're done at time,

437
00:22:11,351 --> 00:22:14,101
at every time you edit, always run these
linters, always use this make file."

438
00:22:14,201 --> 00:22:17,891
And 'cause I started to realize that I
couldn't do anything without it, right?

439
00:22:17,891 --> 00:22:20,271
I couldn't sit at the command
line and know how to build my app

440
00:22:20,281 --> 00:22:21,681
because I'm not a SwiftUI guy.

441
00:22:21,781 --> 00:22:22,101
Michael Rollins: Mm-hmm.

442
00:22:22,201 --> 00:22:24,021
Bret Fisher: and so I've started
to the point where I, those are

443
00:22:24,021 --> 00:22:27,121
almost table stakes enforcements
for all of my projects, where I

444
00:22:27,131 --> 00:22:30,101
want the agent locally to behave
a certain way, like running tests.

445
00:22:30,201 --> 00:22:33,225
So the challenge is, because I'm using
GitHub Actions, I end up with it just

446
00:22:33,325 --> 00:22:36,875
writing, whether it's a compose file
or a make file or, you know, whatever.

447
00:22:36,935 --> 00:22:40,150
I'm always curious what other teams
are doing since I feel like there's

448
00:22:40,150 --> 00:22:43,610
this pull to pull the testing
into wherever the agent's living.

449
00:22:43,610 --> 00:22:46,370
So if you're running a harness locally,
you probably need to run the test locally

450
00:22:46,370 --> 00:22:49,130
because the agent needs to run them,
unless you plan on going to get a new

451
00:22:49,150 --> 00:22:52,126
coffee and you just tell the agent,
please run all these in GitHub Actions

452
00:22:52,126 --> 00:22:53,386
and let me know when they're done."

453
00:22:53,486 --> 00:22:57,066
And then you have to wait 20 minutes
because, or 30 minutes or longer depending

454
00:22:57,066 --> 00:23:00,176
on your test, harnesses and stuff and,
So I was just curious what you do.

455
00:23:00,276 --> 00:23:03,536
Michael Rollins: I actually have, it's
publicly available on my GitHub, if

456
00:23:03,536 --> 00:23:05,856
you wanna go see how I start a project.

457
00:23:05,966 --> 00:23:06,636
this is it.

458
00:23:06,676 --> 00:23:08,916
It even has a skill that you can use

459
00:23:08,916 --> 00:23:10,976
so I call this the DeltaV Launchpad.

460
00:23:11,016 --> 00:23:14,806
it has a Claude skill that'll
basically set this up for you.

461
00:23:14,906 --> 00:23:15,976
it has a Make file.

462
00:23:16,076 --> 00:23:17,516
n- I do everything with Make.

463
00:23:17,586 --> 00:23:23,186
I'm like you, I got completely just
dead tired of Okay, go do this again."

464
00:23:23,186 --> 00:23:27,406
Goes-- Finally, it's just "Go use the
Make file." Make files, you know, the

465
00:23:27,406 --> 00:23:31,236
Docker configs are all, compose files.

466
00:23:31,336 --> 00:23:33,496
it's all pretty standard stuff.

467
00:23:33,596 --> 00:23:35,726
this is all-- None of
this is rocket science.

468
00:23:35,726 --> 00:23:37,656
we all know how this is built.

469
00:23:37,756 --> 00:23:38,106
Bret Fisher: Yeah

470
00:23:38,206 --> 00:23:39,336
Michael Rollins: And so
I just picked something

471
00:23:39,436 --> 00:23:43,046
Bret Fisher: yeah, I think a lot of
the people in the my guild, the Agentic

472
00:23:43,046 --> 00:23:46,736
DevOps Guild, th-there, you know, there's
a wide range of skillsets in there.

473
00:23:46,736 --> 00:23:48,556
There's people that are,
like, very new to this.

474
00:23:48,656 --> 00:23:49,436
They're dipping their toes in

475
00:23:49,436 --> 00:23:51,666
the Copilot or Cursor, maybe Claude Code.

476
00:23:51,766 --> 00:23:54,456
there's people all on the, all the
way on the other end that are, you

477
00:23:54,456 --> 00:23:57,056
know, they're using s- they're using
plugin marketplaces to manage their

478
00:23:57,056 --> 00:24:01,566
skill deployments to their teams and
centralizing the skill management

479
00:24:01,586 --> 00:24:05,806
as, while at the same time they're l-
adding more and more agents to their

480
00:24:05,806 --> 00:24:11,076
post-commit workflows to self-improve
things, on occasion, which i-is a topic

481
00:24:11,076 --> 00:24:12,396
I wanna get into at some point about.

482
00:24:12,496 --> 00:24:15,846
the idea that once you've sort
of matured yourself to, you know,

483
00:24:15,846 --> 00:24:17,856
you've got your agents reviewing PRs.

484
00:24:17,856 --> 00:24:20,826
You've got sort of like the t- the
table stakes of agents post-commit,

485
00:24:20,836 --> 00:24:23,096
where you've got it, you've
got agents looking at your PRs.

486
00:24:23,106 --> 00:24:25,756
You've got agents, maybe you've,
you're lucky enough and you've got

487
00:24:25,856 --> 00:24:26,426
Michael Rollins: Why, but…

488
00:24:26,564 --> 00:24:27,844
Bret Fisher: you troubleshoot or something

489
00:24:27,944 --> 00:24:30,744
Michael Rollins: Yeah, but here's one of
the questions everybody should really ask.

490
00:24:30,844 --> 00:24:33,284
Do we have a way to do this in code?

491
00:24:33,384 --> 00:24:34,934
' Cause if we do, we should do it in code.

492
00:24:35,034 --> 00:24:35,384
Bret Fisher: Yeah.

493
00:24:35,484 --> 00:24:37,384
Michael Rollins: It's funny, I talk to
people sometimes, they're like, "I wanna

494
00:24:37,384 --> 00:24:38,644
build an app." I'm like, "All right.

495
00:24:38,644 --> 00:24:41,924
Tell me about it." And this one
guy told me about, he runs his pool

496
00:24:41,924 --> 00:24:43,514
business on an Excel spreadsheet.

497
00:24:43,614 --> 00:24:46,274
He was like, "Do you think I can
turn this into an app?" I was like,

498
00:24:46,344 --> 00:24:50,364
"Absolutely." He goes, "Where should
I add AI?" And I was like, " Nowhere."

499
00:24:50,464 --> 00:24:52,524
it runs on an Excel spreadsheet.

500
00:24:52,524 --> 00:24:53,714
It's deterministic.

501
00:24:53,814 --> 00:24:56,974
Do you have anywhere in this
process where you want a

502
00:24:56,984 --> 00:24:59,364
non-deterministic choice to be made?

503
00:24:59,464 --> 00:25:02,574
' Cause if the answer is
no, just run the code.

504
00:25:02,674 --> 00:25:07,024
but yeah, I mean, look, I tend to
drive certain types of engineers

505
00:25:07,064 --> 00:25:09,844
insane because I kind of don't care.

506
00:25:09,944 --> 00:25:11,764
You know, you're like, "Well,
what are you using for your

507
00:25:11,764 --> 00:25:13,034
testing?" I'm like, "I don't know.

508
00:25:13,034 --> 00:25:14,014
We're using something.

509
00:25:14,114 --> 00:25:15,084
I know there's tests

510
00:25:15,084 --> 00:25:18,214
there." Right Who's looked at your
tests to make sure that they're

511
00:25:18,214 --> 00:25:21,894
valid?" " Nobody." " How do you know
your tests are good?" "Well, I have

512
00:25:21,894 --> 00:25:24,524
about 80 other things that check." Right?

513
00:25:24,624 --> 00:25:28,224
And so, I don't dictate to
my team the tools, right?

514
00:25:28,264 --> 00:25:30,394
I do say, "You're gonna
use Docker," right?

515
00:25:30,404 --> 00:25:33,064
"You're gonna use Docker Compose.
You're gonna use this Makefile."

516
00:25:33,164 --> 00:25:35,814
But that's more just
for, efficiency's sake.

517
00:25:35,914 --> 00:25:40,174
you know, one of my engineers, he's
way smarter than I am, he uses Codex.

518
00:25:40,204 --> 00:25:43,604
I think it's a terrible idea, but
he uses Codex, and we fight about

519
00:25:43,604 --> 00:25:47,894
whether or not ChatGPT or Claude are
better on a daily basis, and I tell

520
00:25:47,894 --> 00:25:50,124
him he's still wrong, but whatever.

521
00:25:50,224 --> 00:25:52,704
though it is funny, we get to watch
the models leapfrog each other.

522
00:25:52,804 --> 00:25:56,764
I really don't care what the tools are
as long as they get the job done, and

523
00:25:56,944 --> 00:25:59,054
that means I also change the tools often.

524
00:25:59,154 --> 00:26:01,844
Bret Fisher: Well, no, but I mean,
it brings up a good point that,

525
00:26:01,884 --> 00:26:04,404
sometimes the reason I ask these
questions is because it, in the, in

526
00:26:04,404 --> 00:26:06,664
the question of w- you know, think
of some of the things you're using in

527
00:26:06,664 --> 00:26:10,464
CI and just rattle off some toolings
because we can be vague and hand-wavy

528
00:26:10,464 --> 00:26:11,714
about a lot of these concepts that…

529
00:26:11,814 --> 00:26:13,424
But I always like to get
into the nerd details.

530
00:26:13,424 --> 00:26:17,084
But the reason I ask that is because
often the answer is, yeah, Docker Compose

531
00:26:17,094 --> 00:26:19,821
files and Makefiles and like in other
words, there's nothing new in there.

532
00:26:19,921 --> 00:26:21,921
There, it is exactly what it always was.

533
00:26:22,021 --> 00:26:24,791
I personally, don't, I don't know
if this is the general advice you

534
00:26:24,791 --> 00:26:29,841
would give, but I personally advise
a team to guide their AI to make the

535
00:26:29,841 --> 00:26:33,824
choices on the toolings that they
know rather than just let it make…

536
00:26:33,834 --> 00:26:37,844
'Cause in, in theory, it, if you
asked an AI before you greenfield

537
00:26:37,844 --> 00:26:41,324
something what language it thinks
it would use, it might, it probably

538
00:26:41,324 --> 00:26:43,074
will prefer Python or TypeScript.

539
00:26:43,074 --> 00:26:46,259
It, but it might make a bold
choice, and that doesn't mean

540
00:26:46,269 --> 00:26:47,409
it's right for your team.

541
00:26:47,449 --> 00:26:51,309
And so people I think are sometimes,
maybe not so much anymore, but I think

542
00:26:51,309 --> 00:26:53,909
they were looking at this at "Is this
an opportunity for us to reinvent

543
00:26:53,909 --> 00:26:55,119
the wheel in a lot of our tooling?"

544
00:26:55,119 --> 00:26:57,869
And my argument to them is n-
no, because at any point, look,

545
00:26:57,869 --> 00:27:00,079
the model uptimes aren't 100%.

546
00:27:00,179 --> 00:27:04,569
So if at any point you're stuck and you
have to read something, it, wouldn't

547
00:27:04,569 --> 00:27:08,579
it be nice if it was in a thing you
knew in- instead of some completely

548
00:27:08,579 --> 00:27:11,989
foreign… Like, I'm not gonna have
my AI building a Podman interface to

549
00:27:11,989 --> 00:27:13,939
my containers when I'm the Docker guy.

550
00:27:14,039 --> 00:27:17,569
So if it starts to do that, I'm
gonna guide it back to the places,

551
00:27:17,579 --> 00:27:20,449
even though I may never, ever look
at that Dockerfile after the first

552
00:27:20,449 --> 00:27:21,989
creation of a, the, of the PR.

553
00:27:22,089 --> 00:27:24,799
and I don't know if you have standards
like that, but I've always been imagining

554
00:27:24,799 --> 00:27:28,189
that teams need to actually start, even
polyglot teams, they need to kind of

555
00:27:28,199 --> 00:27:30,939
settle in and say, "Look, these are
the languages we would really prefer

556
00:27:30,939 --> 00:27:34,299
you stick with in AI and not just,
don't be COBOLing our production."

557
00:27:34,399 --> 00:27:37,559
Michael Rollins: I've got this Instagram
channel that seems to be blowing up,

558
00:27:37,709 --> 00:27:41,429
and I piss engineers off all the time
' cause I say things like, "You shouldn't

559
00:27:41,429 --> 00:27:44,629
be reading code, and you shouldn't be
writing code." And s- there's quite a few

560
00:27:44,629 --> 00:27:46,349
engineers that are very, very literal.

561
00:27:46,449 --> 00:27:50,619
the other, another one I had recently
is like, "Nah, just make sure your tests

562
00:27:50,619 --> 00:27:54,459
run. It kind of doesn't matter what they
are." And, but I had a few people get in,

563
00:27:54,459 --> 00:27:57,319
like really get nasty arguing with me,

564
00:27:57,437 --> 00:27:57,817
Bret Fisher: Yeah.

565
00:27:57,917 --> 00:27:59,157
Messing with their identity, bro.

566
00:27:59,257 --> 00:28:03,277
Michael Rollins: No, no, they were right."
They were right because my assumption

567
00:28:03,287 --> 00:28:07,277
was different. So I've had a couple
people come in and say, "This is insane.

568
00:28:07,287 --> 00:28:08,207
How can you do this?

569
00:28:08,217 --> 00:28:09,857
It's too risky," da, da, da, da, da.

570
00:28:09,857 --> 00:28:11,427
And I've had a lot of
people come in and say that.

571
00:28:11,681 --> 00:28:15,431
Well, then it turns out that
they're building embedded systems.

572
00:28:15,531 --> 00:28:15,811
Bret Fisher: Hmm.

573
00:28:15,911 --> 00:28:19,061
Michael Rollins: They're in healthcare,
they're in banking, they're in

574
00:28:19,061 --> 00:28:20,931
some kind of high security field.

575
00:28:21,031 --> 00:28:23,421
I'm like, " Yeah, don't listen to me."

576
00:28:23,521 --> 00:28:23,831
Bret Fisher: Right

577
00:28:23,949 --> 00:28:28,359
Michael Rollins: But if you're building
web apps, if you're building mobile that

578
00:28:28,359 --> 00:28:30,549
is just an extension of the web app,

579
00:28:30,722 --> 00:28:31,092
Bret Fisher: Yeah.

580
00:28:31,242 --> 00:28:32,622
Michael Rollins: it's already been done.

581
00:28:32,722 --> 00:28:34,372
Just use the standard.

582
00:28:34,472 --> 00:28:36,602
You don't have to reinvent that wheel,

583
00:28:36,702 --> 00:28:38,922
Bret Fisher: you have a very valid
point about your assumptions because,

584
00:28:39,022 --> 00:28:42,032
you know, you both of us op- have
been operating… I mean, I've been

585
00:28:42,032 --> 00:28:43,752
in cloud native for over a decade.

586
00:28:43,762 --> 00:28:46,242
you know, the h- the things that we
learned about distributed computing,

587
00:28:46,342 --> 00:28:50,412
microservice architecture, you know, API
contracts, like all the sort of the things

588
00:28:50,412 --> 00:28:53,422
that kind of evolved over the last 15
years, like I've been living and breathing

589
00:28:53,422 --> 00:28:55,052
that at the near tip of the spear.

590
00:28:55,152 --> 00:28:58,992
So when I meet a team that they have 3%
test coverage, they're not using a type

591
00:28:58,992 --> 00:29:03,632
safe or memory safe language, they may
be YOLOing some old PHP stuff and they've

592
00:29:03,632 --> 00:29:07,512
got a very monolithic approach, and then
they're like, " How-- We can't do this.

593
00:29:07,512 --> 00:29:09,782
We're having all kinds of production
failures." and I look at them and I

594
00:29:09,782 --> 00:29:12,742
would evaluate it and say, "Yeah, that's
because…" I love this phrase lately,

595
00:29:12,742 --> 00:29:18,042
is that AI AI just amplifies the team
culture, and your team culture is of 2010.

596
00:29:18,142 --> 00:29:21,752
You need to modernize a lot of
your culture before you go fast.

597
00:29:21,762 --> 00:29:24,372
Like you can't go fast without
the safety, and the safety

598
00:29:24,372 --> 00:29:25,602
was never there to begin with.

599
00:29:25,702 --> 00:29:27,782
And so I do run into that
on occasion with teams.

600
00:29:27,792 --> 00:29:31,392
And there's surprisingly a number
of teams, maybe not in embedded or

601
00:29:31,392 --> 00:29:34,762
healthcare, but like a surprising
number of teams that have very large,

602
00:29:34,762 --> 00:29:39,142
important apps that I get c- to consult
with, and I cannot believe the, I

603
00:29:39,142 --> 00:29:44,212
would call it the level of immaturity
that the code base and the team has.

604
00:29:44,399 --> 00:29:47,099
And 'cause we don't have accreditations
in, in, in our business, right?

605
00:29:47,099 --> 00:29:48,629
We don't have plumber certifications.

606
00:29:48,729 --> 00:29:51,689
So like I can't look at them and assume
that they know all these things and

607
00:29:51,689 --> 00:29:54,039
that they're doing all these behaviors
that I've been doing for so long.

608
00:29:54,139 --> 00:29:56,149
Michael Rollins: I have been
able to look under the hood at

609
00:29:56,149 --> 00:29:58,109
multiple Silicon Valley giants.

610
00:29:58,209 --> 00:29:58,569
Bret Fisher: Hmm.

611
00:29:58,669 --> 00:30:00,429
Michael Rollins: It's all
shit all the way down.

612
00:30:00,519 --> 00:30:04,659
Like every single engineering team
is "We have the worst stack ever.

613
00:30:04,759 --> 00:30:08,889
All of our code is crap." And it's
"No, your code's fine." there's really

614
00:30:08,889 --> 00:30:12,879
only one measure that matters when it
comes to talking about your product.

615
00:30:12,979 --> 00:30:13,809
Does it work?

616
00:30:13,909 --> 00:30:16,569
A follow-up question is,
does it work all the time?

617
00:30:16,669 --> 00:30:16,899
Bret Fisher: Mm-hmm.

618
00:30:17,017 --> 00:30:21,457
Michael Rollins: If the answer to those
two things is yes, it doesn't matter.

619
00:30:21,557 --> 00:30:25,247
Now, there's a third implication now
recently, especially with a bunch of

620
00:30:25,257 --> 00:30:27,647
the Mythos stuff, which is it secure?

621
00:30:27,747 --> 00:30:31,337
And that's always been important,
and it's always been relatively easy

622
00:30:31,367 --> 00:30:33,567
to exploit sloppy stacks, right?

623
00:30:33,667 --> 00:30:36,207
Bret Fisher: By the way, Sloppy Stack
is the name of a future podcast.

624
00:30:36,307 --> 00:30:37,517
Michael Rollins: Oh my God, yes.

625
00:30:37,557 --> 00:30:39,217
That's, that's really… Yeah.

626
00:30:39,317 --> 00:30:43,137
I love, I love accidentally, yeah,
yeah, break down of your sloppy stack.

627
00:30:43,137 --> 00:30:44,157
Now we're gonna enter it.

628
00:30:44,157 --> 00:30:45,667
It doesn't, it also sounds really gross.

629
00:30:45,767 --> 00:30:49,797
whether or not your architecture sucks,
whether or not your stack sucks, whether

630
00:30:49,797 --> 00:30:54,077
or not anything sucks, there's only
one measure that you have to apply

631
00:30:54,077 --> 00:30:55,897
to it, which is does it work, right?

632
00:30:55,937 --> 00:30:59,427
And now that d- decomposes
fast into tiny little things.

633
00:30:59,527 --> 00:31:03,787
but if it works and it's built
on ancient PHP, maybe there's

634
00:31:03,787 --> 00:31:05,627
not a reason to fix it, right?

635
00:31:05,727 --> 00:31:07,077
Bret Fisher: I have
another, follow-on question.

636
00:31:07,167 --> 00:31:10,167
So we, we're seeing a lot in the
news around, the general degradation

637
00:31:10,177 --> 00:31:13,947
of code quality as people onboard
AI, and I understand how it's

638
00:31:13,947 --> 00:31:17,647
happening, and I don't understand
why it's a mystery on how to fix it.

639
00:31:17,747 --> 00:31:23,127
Because it feels, at least in my
experience, if I have all these review

640
00:31:23,127 --> 00:31:28,407
agents and PRs and I'm using I've got
a series of skills, that are auditing,

641
00:31:28,407 --> 00:31:32,687
security audits, uh concurrency audits,
performance audits, you know, these

642
00:31:32,687 --> 00:31:36,297
are things that used to be very hard
for engineers to do because it required

643
00:31:36,297 --> 00:31:40,537
a lot of manpower, and we would often
skip them, especially when budgets were

644
00:31:40,537 --> 00:31:42,237
tight and project deadlines were close.

645
00:31:42,337 --> 00:31:46,117
and continual documentation updates, I'm
just thinking, going down the list of the

646
00:31:46,117 --> 00:31:50,877
things that I can now have agents burn a
bunch of tokens on, to me, it doesn't make

647
00:31:50,877 --> 00:31:53,117
sense that the code quality would go down.

648
00:31:53,127 --> 00:31:57,557
It feels like to me that, with enough
gates that I'm triggering AIs in, that

649
00:31:57,557 --> 00:32:01,607
the code quality would, in terms of
maybe not my impression of how the

650
00:32:01,607 --> 00:32:05,047
good code quality is, but at least
the way it performs and behaves in

651
00:32:05,047 --> 00:32:07,767
production, feels like it should at
least stay the same, if not go up.

652
00:32:07,867 --> 00:32:09,247
Is that what you're seeing?

653
00:32:09,257 --> 00:32:11,497
Do you see that or do
you see the opposite?

654
00:32:11,497 --> 00:32:12,097
am I wrong?

655
00:32:12,277 --> 00:32:12,517
Or

656
00:32:12,617 --> 00:32:13,557
Michael Rollins: I don't read the code.

657
00:32:13,557 --> 00:32:14,137
Why do I care?

658
00:32:14,237 --> 00:32:16,167
Bret Fisher: Yeah, well, but
you see the outcome of the code.

659
00:32:16,177 --> 00:32:16,857
So obviously,

660
00:32:16,911 --> 00:32:18,071
Michael Rollins: see
the outcome of the code.

661
00:32:18,171 --> 00:32:20,961
Bret Fisher: So if your production,
if you were five nines before and

662
00:32:20,961 --> 00:32:24,251
you're four nines now, you would notice
that, assuming you're measuring it.

663
00:32:24,281 --> 00:32:24,551
But,

664
00:32:24,653 --> 00:32:26,743
Michael Rollins: yeah, I mean,
listen, we're not five nines.

665
00:32:26,873 --> 00:32:28,113
Like I was… We're close.

666
00:32:28,143 --> 00:32:30,703
we're much closer now than
we were two months ago.

667
00:32:30,803 --> 00:32:33,333
you know, it used to be I had to
take about 10 to 15 minutes of

668
00:32:33,333 --> 00:32:37,463
downtime to d- per any release
because of the way our stack is

669
00:32:37,463 --> 00:32:39,703
actually built, and because
of the way that we release.

670
00:32:39,733 --> 00:32:41,633
But I've since fixed that.

671
00:32:41,663 --> 00:32:42,973
We don't have downtime.

672
00:32:43,073 --> 00:32:46,323
as a matter of fact, I mean, we probably
are close to something like five nines.

673
00:32:46,423 --> 00:32:49,353
But the question is like, Well, let
me, let me turn this back onto you.

674
00:32:49,453 --> 00:32:51,573
Why does code quality
matter, and what is it?

675
00:32:51,673 --> 00:32:52,053
Bret Fisher: Well,

676
00:32:52,243 --> 00:32:53,263
Michael Rollins: Why are you focused

677
00:32:53,333 --> 00:32:54,053
Bret Fisher: that's a question.

678
00:32:54,093 --> 00:32:58,603
I would say, I- what I'm specifically
talking about is failure rates

679
00:32:58,603 --> 00:33:00,763
in production as a DORA metric.

680
00:33:00,863 --> 00:33:01,943
that's-- and this is the problem.

681
00:33:01,943 --> 00:33:05,103
This is my assumption, is when I
think of code quality, it isn't

682
00:33:05,123 --> 00:33:06,273
whether you tabbed or spaced.

683
00:33:06,273 --> 00:33:10,903
It isn't whether you followed the black
linter on your, you know, whatever, spec.

684
00:33:10,913 --> 00:33:14,393
it is about when you got to production,
in an enterprise there will be a

685
00:33:14,393 --> 00:33:17,913
security team that does a production
analysis of what's going on, and they'll

686
00:33:17,913 --> 00:33:22,103
find more flags than previously, or
you'll start to have more outages

687
00:33:22,113 --> 00:33:23,973
or customer complaints than before.

688
00:33:23,973 --> 00:33:27,713
And so these metrics, according to the
mark- the news, the media, you know, these

689
00:33:27,713 --> 00:33:32,283
things are going up for teams that are
going too fast in AI, and that's, that--

690
00:33:32,383 --> 00:33:36,783
Some are deriving that as the conclusion
means humans still have to read the code.

691
00:33:36,883 --> 00:33:39,963
I don't necessarily take that as
an outcome of that assumption.

692
00:33:39,963 --> 00:33:44,593
I take it as, no, you need to put more,
you need to put more rigor into the AI

693
00:33:44,613 --> 00:33:48,633
process, not back up to the world where
we were all reading every line of code.

694
00:33:48,643 --> 00:33:50,053
So I was curious on your take.

695
00:33:50,153 --> 00:33:51,973
Michael Rollins: it would be
interesting to go back to when

696
00:33:51,973 --> 00:33:53,433
the Agile Manifesto was written.

697
00:33:53,533 --> 00:33:56,563
Everybody had done Waterfall to that
point, and all of a sudden somebody comes

698
00:33:56,563 --> 00:34:00,493
out and they say, "This is stupid. We
should, change as fast as we can." Right?

699
00:34:00,523 --> 00:34:03,363
"We shouldn't plan for
a whole freaking year.

700
00:34:03,373 --> 00:34:04,393
That's insanity.

701
00:34:04,493 --> 00:34:08,023
What we should do instead is we should,
take stock of where we are and take

702
00:34:08,023 --> 00:34:12,493
stock of where we wanna be in two weeks
and get there." I am willing to bet that

703
00:34:12,523 --> 00:34:16,823
the performance of Agile at the very
beginning was much worse than Waterfall.

704
00:34:16,923 --> 00:34:18,033
Nobody had done it.

705
00:34:18,133 --> 00:34:18,473
Bret Fisher: Right

706
00:34:18,573 --> 00:34:20,253
Michael Rollins: Everybody
was trying to figure it out.

707
00:34:20,283 --> 00:34:22,053
This is worse, right?

708
00:34:22,153 --> 00:34:23,783
Bret Fisher: This is,
this is that on steroids.

709
00:34:23,863 --> 00:34:24,073
Michael Rollins: Yeah.

710
00:34:24,173 --> 00:34:30,503
I look at the things we are doing now
and on my team that we weren't doing

711
00:34:30,553 --> 00:34:32,583
two months ago, and I sweat Right?

712
00:34:32,683 --> 00:34:33,593
But you know what?

713
00:34:33,693 --> 00:34:39,313
My release rate, the amount of code that
has landed is hockey sticking, and I've

714
00:34:39,313 --> 00:34:41,493
been forced into what I'm doing now.

715
00:34:41,593 --> 00:34:45,633
I would be willing to bet
that we are shipping more bugs

716
00:34:45,643 --> 00:34:46,853
than I wish we were shipping.

717
00:34:46,953 --> 00:34:51,723
I don't think we're shipping the same
bug twice, at least not one that,

718
00:34:51,823 --> 00:34:53,483
like… Have we had regressions?

719
00:34:53,653 --> 00:34:54,393
Absolutely.

720
00:34:54,493 --> 00:34:58,403
have we had regressions in this changeover
period from the old way we were doing it

721
00:34:58,403 --> 00:35:00,113
until the new way we're, we've done it?

722
00:35:00,173 --> 00:35:01,173
Yes, 100%.

723
00:35:01,273 --> 00:35:04,583
I don't know that we've shipped
a regression since we started on

724
00:35:04,583 --> 00:35:07,323
whatever this new thing is that
we're s- that we've been doing.

725
00:35:07,423 --> 00:35:09,893
Now granted, we've only been
doing it for three weeks.

726
00:35:09,993 --> 00:35:10,223
Bret Fisher: Yeah.

727
00:35:10,323 --> 00:35:13,913
Is there like a hot tip on exactly,
do you add these to a skill?

728
00:35:13,923 --> 00:35:16,493
Do you add just, do you add
new tests for every failure?

729
00:35:16,513 --> 00:35:16,843
Like

730
00:35:17,037 --> 00:35:20,547
Michael Rollins: we have to be
absolutely obsessive about tests.

731
00:35:20,647 --> 00:35:20,887
Bret Fisher: Yeah.

732
00:35:20,967 --> 00:35:23,157
Michael Rollins: Yeah … I hated tests.

733
00:35:23,217 --> 00:35:23,947
I hate tests.

734
00:35:24,097 --> 00:35:25,667
I hate tests as an engineer.

735
00:35:25,767 --> 00:35:28,517
people that are like, "Test-driven
development," I would always be

736
00:35:28,517 --> 00:35:30,897
like, "You are out of your…
That is the stupidest thing."

737
00:35:30,997 --> 00:35:35,097
I could not think of a better way to grind
an engineering team to a halt, right?

738
00:35:35,137 --> 00:35:36,557
we gotta ship this fast.

739
00:35:36,657 --> 00:35:38,367
Bret Fisher: I'm the same way
with spec-driven development

740
00:35:38,367 --> 00:35:39,097
that I am with test-driven.

741
00:35:39,157 --> 00:35:42,587
I'm a shiny person and none of those
things give me the dopamine hit of shiny.

742
00:35:42,587 --> 00:35:45,247
Which is why no one pays me
professionally to develop full-time.

743
00:35:45,347 --> 00:35:46,137
Michael Rollins: Yeah, but you know what?

744
00:35:46,237 --> 00:35:48,787
I get test-driven
development for free now.

745
00:35:48,887 --> 00:35:49,247
Bret Fisher: Yeah.

746
00:35:49,347 --> 00:35:50,257
Michael Rollins: I don't have to worry

747
00:35:50,327 --> 00:35:50,717
Bret Fisher: the pain

748
00:35:50,817 --> 00:35:51,177
Michael Rollins: Yeah.

749
00:35:51,217 --> 00:35:52,607
let me throw this at you.

750
00:35:52,707 --> 00:35:57,647
I was working on 15 different bugs
at the same time the other day, okay?

751
00:35:59,563 --> 00:36:03,613
was doing so much that I locked
my machine up by filling up

752
00:36:03,653 --> 00:36:07,873
the disk, because apparently if
you run Node, the, what is it?

753
00:36:07,883 --> 00:36:12,420
The npm install and the type, you know,
the Python equivalent, that can be a

754
00:36:12,420 --> 00:36:16,430
lot of space, and I only have a half a
terabyte hard drive on my work machine,

755
00:36:16,430 --> 00:36:18,640
'cause I decided that's a fine thing.

756
00:36:18,720 --> 00:36:20,410
Bret Fisher: that code
is small and that, you d-

757
00:36:20,510 --> 00:36:22,350
Michael Rollins: And I was
building Docker images, yeah.

758
00:36:22,530 --> 00:36:25,960
and, next thing I know,
my machine just stops.

759
00:36:26,060 --> 00:36:28,340
not slows down, it just stops, right?

760
00:36:28,440 --> 00:36:31,350
But in the process, I had
also opened up 15 PRs.

761
00:36:31,350 --> 00:36:33,910
I locked up the entire, stack.

762
00:36:34,010 --> 00:36:38,560
I accidentally flooded the GitHub
Actions, overloaded the Lambda that

763
00:36:38,560 --> 00:36:42,890
runs our, database cloning, where
we clone production into the PR

764
00:36:42,890 --> 00:36:46,130
environment, and found a bug in the
process, which was very nice to find.

765
00:36:46,230 --> 00:36:48,820
but it was not the first time that
it happened, where I had basically

766
00:36:48,820 --> 00:36:53,320
launched this clone storm that locked
our entire PR infrastructure up.

767
00:36:53,420 --> 00:36:56,530
Everything was still running,
but you couldn't do a new thing.

768
00:36:56,630 --> 00:37:01,430
I've never tried to land 15
PRs at the same time before.

769
00:37:01,530 --> 00:37:02,560
Have you ever done that?

770
00:37:02,660 --> 00:37:02,880
Bret Fisher: Not even

771
00:37:02,992 --> 00:37:05,832
Michael Rollins: Has that… Have
you ever heard of anybody doing that?

772
00:37:05,932 --> 00:37:08,942
Bret Fisher: Yeah, I mean, there's
the whole new, what's the PR gating

773
00:37:08,952 --> 00:37:09,892
thing where you combine them?

774
00:37:09,992 --> 00:37:13,262
Michael Rollins: as, PO it's like
merge, merge streams or something.

775
00:37:13,362 --> 00:37:15,202
Bret Fisher: yeah, merge stacks or

776
00:37:15,302 --> 00:37:18,012
Michael Rollins: yeah, it's basically
what GitHub just totally blew up.

777
00:37:18,112 --> 00:37:20,702
Oh man, GitHub sloppy stacked this hard.

778
00:37:20,802 --> 00:37:25,462
so, but listen, yes, of course it's, it
may, yeah, maybe it is degrading, right?

779
00:37:25,552 --> 00:37:27,332
but nobody's ever done this before, man.

780
00:37:27,432 --> 00:37:31,652
Look, we were doing it before
Anthropic and before OpenAI.

781
00:37:31,752 --> 00:37:37,172
We were doing this style of
engineering back in July of last year.

782
00:37:37,272 --> 00:37:37,292
Yeah.

783
00:37:37,292 --> 00:37:39,492
Anthropic and OpenAI did not adopt it.

784
00:37:39,582 --> 00:37:40,742
nobody's done this, man.

785
00:37:40,882 --> 00:37:44,082
And so, yeah, are we finding
things are degrading a little bit?

786
00:37:44,112 --> 00:37:44,642
Sure.

787
00:37:44,742 --> 00:37:47,842
you know how long it takes
me at this point to put a fix

788
00:37:47,842 --> 00:37:49,582
into production if I notice it?

789
00:37:49,682 --> 00:37:53,072
20 minutes, 30 minutes, assuming
I have to build the images.

790
00:37:53,172 --> 00:37:56,452
Bret Fisher: I recorded a very similar
episode to this topic yesterday with,

791
00:37:56,562 --> 00:37:59,592
someone from GitHub Next and then
someone from Microsoft Research, and

792
00:37:59,592 --> 00:38:02,752
they're both working on the agentic
workflows feature set for GitHub,

793
00:38:02,782 --> 00:38:05,222
which is currently, I believe, in
beta, and I'm very excited about it.

794
00:38:05,222 --> 00:38:08,092
I thought it was very weird at first,
but after spending an hour and a half

795
00:38:08,092 --> 00:38:11,502
with them, I'm all on board and I'm very
excited about spending more time with it.

796
00:38:11,602 --> 00:38:16,822
He talked about that he's also working
across 22 projects with multiple PRs

797
00:38:16,832 --> 00:38:21,402
in each project, and that he does
everything that-- because they're

798
00:38:21,402 --> 00:38:24,772
GitHub, they do everything through
Copilot, everything in GitHub.

799
00:38:24,812 --> 00:38:28,452
And so he spends most of his day
as a researcher that's, or as a

800
00:38:28,452 --> 00:38:33,982
developer researcher, I should say,
prompting through github.com/, what-

801
00:38:34,182 --> 00:38:37,202
whatever the little agentic, page
is where you can prompt like you're

802
00:38:37,202 --> 00:38:39,302
in Co- Code GUI or Claude Code GUI.

803
00:38:39,402 --> 00:38:41,652
and he was saying, you know, he does it,
He spends a lot of time on his mobile,

804
00:38:41,752 --> 00:38:45,692
doing it, and, sort of talked me through
his day to day and I was like, "Man, I

805
00:38:45,692 --> 00:38:50,552
feel so antiquated," 'cause I'm still got
this habit of if I think of something, I

806
00:38:50,552 --> 00:38:53,992
go and write it down on my phone, but I
think I'll do that when I get back to my

807
00:38:53,992 --> 00:38:59,612
computer because I'm just so not driven
to use Dispatch or I'm just not there yet.

808
00:38:59,612 --> 00:38:59,892
Yeah.

809
00:38:59,922 --> 00:39:03,962
I'm-- I know I have it and I just don't
default to it yet as, "Oh, I can kick, I

810
00:39:03,962 --> 00:39:07,102
can kick this off right now. I don't need
to wait till I'm back at my computer."

811
00:39:07,102 --> 00:39:08,612
Michael Rollins: So this
is, this is what I do.

812
00:39:08,712 --> 00:39:10,562
I have two machines that I write code on.

813
00:39:10,662 --> 00:39:13,772
one is this laptop here, and
one is my machine back at work.

814
00:39:13,822 --> 00:39:16,312
I typically use my machine
back at work for this.

815
00:39:16,412 --> 00:39:21,238
I have remote sessions or remote
environments or something, turn on

816
00:39:21,258 --> 00:39:25,418
automatically in Claude Code, and then I
also have Auto Mode turned on by default.

817
00:39:25,518 --> 00:39:25,848
Bret Fisher: Right.

818
00:39:25,952 --> 00:39:26,592
Safety off.

819
00:39:26,692 --> 00:39:27,032
Michael Rollins: no.

820
00:39:27,132 --> 00:39:27,932
I leave safety on.

821
00:39:28,032 --> 00:39:29,182
Auto mode is not YOLO

822
00:39:29,282 --> 00:39:29,622
Bret Fisher: oh, right.

823
00:39:29,622 --> 00:39:29,882
Sorry.

824
00:39:29,922 --> 00:39:31,012
Auto safety mode.

825
00:39:31,042 --> 00:39:31,272
Right.

826
00:39:31,272 --> 00:39:31,582
I'm sorry.

827
00:39:31,582 --> 00:39:32,262
I'm th- I was thinking of

828
00:39:32,462 --> 00:39:36,002
Michael Rollins: it's legitimately
called, there's planning, except that

829
00:39:36,002 --> 00:39:40,612
it's whatever the default, normal default
state is, and now they added Auto Mode.

830
00:39:40,732 --> 00:39:44,552
Auto mode actually has a safety
classifier in it, and so there's

831
00:39:44,552 --> 00:39:48,882
effectively another agent that's over here
running that says, "Is this dangerous?

832
00:39:48,922 --> 00:39:49,802
Is this dangerous?

833
00:39:49,822 --> 00:39:52,282
Is this dan-?" No longer
can you do that, right?

834
00:39:52,382 --> 00:39:52,782
Bret Fisher: Right.

835
00:39:52,950 --> 00:39:53,920
Michael Rollins: it'll just smack it down.

836
00:39:54,020 --> 00:39:55,580
Bret Fisher: Sorry, you don't,
real quick, you don't enable,

837
00:39:55,630 --> 00:39:57,010
sandboxing in the Claude Code?

838
00:39:57,110 --> 00:39:59,930
Michael Rollins: actually Auto mode may be
some kind of sandboxing in the background.

839
00:39:59,930 --> 00:40:02,650
Bret Fisher: You could call it that,
but yeah, 'cause it's gating the tool.

840
00:40:02,670 --> 00:40:04,210
it's gating tool access basically, right?

841
00:40:04,210 --> 00:40:07,860
Like it's a programmatic deterministic
gate, at least as maybe not auto,

842
00:40:07,860 --> 00:40:10,450
maybe part of it's a little bit of
an, a mini LLM or something in there.

843
00:40:10,500 --> 00:40:11,070
Michael Rollins: I'm fairly…

844
00:40:11,170 --> 00:40:12,200
Bret Fisher: deterministic way to do it.

845
00:40:12,300 --> 00:40:15,260
Michael Rollins: Yeah, but then there's
also, then there's your unsafe mode.

846
00:40:15,360 --> 00:40:16,170
Bret Fisher: un- hands-on.

847
00:40:16,310 --> 00:40:16,560
Michael Rollins: Okay.

848
00:40:16,660 --> 00:40:20,520
So the danger with unsafe mode is
actually, auto mode and unsafe mode

849
00:40:20,520 --> 00:40:24,420
are very similar, but there are
certain things that auto mode will

850
00:40:24,420 --> 00:40:30,110
not allow the agent to do, and one of
them is just go find a module to use.

851
00:40:30,210 --> 00:40:32,680
and that is super-duper
important right now.

852
00:40:32,690 --> 00:40:37,220
the number of tool chain attacks that
are currently in flight are insane.

853
00:40:37,320 --> 00:40:41,840
And the security landscape is, it's
actually one of the things that,

854
00:40:41,840 --> 00:40:43,260
keeps me up thinking about it.

855
00:40:43,270 --> 00:40:47,530
Did we ship something that has a
security vulnerability that was

856
00:40:47,530 --> 00:40:51,930
intentionally put there so that our
stack would be exploitable later?

857
00:40:52,030 --> 00:40:53,400
Bret Fisher: it sleeping on my machine?

858
00:40:53,500 --> 00:40:55,140
Michael Rollins: And is
it sleeping on my machine?

859
00:40:55,312 --> 00:40:55,912
Bret Fisher: waiting to take my

860
00:40:56,012 --> 00:40:56,322
Michael Rollins: Yeah.

861
00:40:56,422 --> 00:41:00,832
but I have these remote sessions running,
and I, I'll be sitting… I used to not

862
00:41:00,832 --> 00:41:04,692
sit on the couch and watch TV with my
wife because I felt like I needed to go be

863
00:41:04,692 --> 00:41:06,862
doing something, and I'd have these ideas.

864
00:41:06,872 --> 00:41:09,832
I don't love watching TV, but I like
being with my wife, and it was like,

865
00:41:09,832 --> 00:41:11,152
okay, well let me go sit with her.

866
00:41:11,192 --> 00:41:13,612
But then I'd have these ideas, and
I'd be like, "I need to go do this."

867
00:41:13,712 --> 00:41:16,912
Well, now I just pull my phone
out and I tap a few things on into

868
00:41:16,912 --> 00:41:18,952
Claude, and I just put my phone away.

869
00:41:19,052 --> 00:41:22,562
And it goes and it tests everything
and does the idea, and then it

870
00:41:22,562 --> 00:41:25,712
comes back and it's "Okay, yeah,
it's ready to ship. Are you ready?"

871
00:41:25,812 --> 00:41:26,952
I mean, it's a wild world.

872
00:41:27,052 --> 00:41:29,872
Bret Fisher: you can't start a new
project from the phone, can you?

873
00:41:29,972 --> 00:41:31,252
On Claude Code specifically?

874
00:41:31,352 --> 00:41:31,792
Michael Rollins: Yes, you can.

875
00:41:31,892 --> 00:41:32,242
Bret Fisher: Okay.

876
00:41:32,342 --> 00:41:34,122
Michael Rollins: when you say a
new project, what do you mean?

877
00:41:34,222 --> 00:41:37,862
Bret Fisher: for example, if I don't
have remote enabled on a, on an

878
00:41:37,862 --> 00:41:43,432
existing repo that I've cloned, or I
wanna start a new code project and I

879
00:41:43,432 --> 00:41:48,342
don't have a repo yet or a folder, I
can't get it to initiate a local new

880
00:41:48,352 --> 00:41:50,312
session on my Mac remotely, right?

881
00:41:50,312 --> 00:41:53,272
I kinda, like you said, you had
remote on always, or you have it, I

882
00:41:53,272 --> 00:41:54,142
didn't even know you could do this.

883
00:41:54,152 --> 00:41:56,982
It's actually a pro tip is, just turn
that thing on for everyone, all of

884
00:41:56,982 --> 00:42:00,362
them, 'cause I always forget to, to
slash enable remote in my sessions.

885
00:42:00,434 --> 00:42:02,434
Michael Rollins: All right, so first
off, you have to give Claude Code

886
00:42:02,434 --> 00:42:03,994
the access to your GitHub repo.

887
00:42:04,094 --> 00:42:07,044
If it doesn't have a GitHub
repo, you can't use it.

888
00:42:07,144 --> 00:42:11,224
Now, if you wanna start a new project,
you go create the new repo in the GitHub

889
00:42:11,224 --> 00:42:14,504
app, and then you come back to Claude
Code, and you give it permission on your

890
00:42:14,504 --> 00:42:16,994
phone to just use that, and then it can.

891
00:42:17,094 --> 00:42:20,794
And you can start the, you can start
it in the, there's like a cloud

892
00:42:20,894 --> 00:42:22,964
version that'll run and make all the

893
00:42:23,297 --> 00:42:25,387
Bret Fisher: And you're using, the
connections feature that actually

894
00:42:25,479 --> 00:42:27,249
Michael Rollins: Yeah, I don't
even remember what it's called.

895
00:42:27,349 --> 00:42:29,829
Bret Fisher: Yeah, it's an
automatic connection to GitHub

896
00:42:29,839 --> 00:42:31,089
from the Claude Code web UI.

897
00:42:31,189 --> 00:42:32,079
Michael Rollins: Yes, that's correct.

898
00:42:32,139 --> 00:42:37,339
And now because of the way our stack
works, because I can issue a PR and stand

899
00:42:37,349 --> 00:42:41,509
up the copy of production, I actually
don't have to have something running

900
00:42:41,609 --> 00:42:43,949
on my local, but I can continue it.

901
00:42:44,049 --> 00:42:48,329
And so if there's something going on
and I'm like, "Huh, I would-- I need

902
00:42:48,339 --> 00:42:52,489
to go walk away for two hours, but I
wanna make sure this thing progresses,"

903
00:42:52,589 --> 00:42:53,869
I'll just get a notification.

904
00:42:53,869 --> 00:42:57,369
I can look at it when I'm able and
say, "Okay, go do this, that, and the

905
00:42:57,369 --> 00:42:59,709
other." So that is, that is a pro tip.

906
00:42:59,809 --> 00:43:00,019
Yeah.

907
00:43:00,169 --> 00:43:05,009
you can, if you want, be running through
First Landing Park, with 25 pounds of

908
00:43:05,009 --> 00:43:09,669
computer equipment on you, and pull your
phone out and advance your project again.

909
00:43:09,769 --> 00:43:10,099
You know?

910
00:43:10,159 --> 00:43:12,639
Bret Fisher: I don't see a lot of,
those weight v- exercise weight vests

911
00:43:12,639 --> 00:43:15,569
with laptop sleeves in them, but, you
know, maybe that's the nerd equivalent.

912
00:43:15,669 --> 00:43:20,259
So if there was… we've talked a lot
of high-level stuff, and I really,

913
00:43:20,309 --> 00:43:23,479
I think this is gonna be really good
for people to s- sort of hear a team

914
00:43:23,479 --> 00:43:27,299
that's… I w- I wanna make a point here
that I feel like one of your secrets is

915
00:43:27,299 --> 00:43:30,139
that you're in tune with the business,
which we, you kind of started with that.

916
00:43:30,239 --> 00:43:34,409
Your team is aware of the business
goals, sales objectives, like in, in

917
00:43:34,409 --> 00:43:37,379
terms of what the product needs to be,
where we're headed, why does this thing

918
00:43:37,379 --> 00:43:39,209
exist, what do our customers look like?

919
00:43:39,309 --> 00:43:42,954
Because that allows them more agency and
allows them to… Also, there's a metric

920
00:43:42,974 --> 00:43:46,404
of judging risk there, because like you
said, if you were a healthcare, if you

921
00:43:46,404 --> 00:43:52,154
were a fin- fin healthcare company that
was, that, you may not have the same

922
00:43:52,154 --> 00:43:57,574
posture because you're appropriately
dialing up the risk tolerance to the level

923
00:43:57,574 --> 00:44:00,994
of the organization that you're in, and
that is different for every organization.

924
00:44:00,994 --> 00:44:03,694
So I j- I feel it's like a, feels
like it, this bears worth repeating

925
00:44:03,794 --> 00:44:07,854
that if you're not… Do not try
to shove this round peg in a square

926
00:44:07,864 --> 00:44:11,714
hole of, you're in a high safety,
you have, you're on a DARPA contract.

927
00:44:11,804 --> 00:44:16,084
you probably, yeah, you probably
should be doing old school co-pilot

928
00:44:16,184 --> 00:44:20,964
code completion with the AI at best,
where every line is sweated over

929
00:44:20,994 --> 00:44:22,694
and you're just saving keystrokes.

930
00:44:22,734 --> 00:44:24,934
Michael Rollins: I've always been
the engineer that was connected

931
00:44:24,934 --> 00:44:26,404
to the entire organization.

932
00:44:26,474 --> 00:44:27,914
It's just my nature.

933
00:44:27,964 --> 00:44:31,184
I like talking to people, and I'm
curious about how the business works.

934
00:44:31,284 --> 00:44:35,764
However, that is a skill that
engineers are gonna have to have in

935
00:44:35,764 --> 00:44:37,554
the future, at least to some degree.

936
00:44:37,654 --> 00:44:42,004
When we started this, the very
first thing we realized was that our

937
00:44:42,004 --> 00:44:43,714
infrastructure was the first gate.

938
00:44:43,814 --> 00:44:46,834
So we went ahead and we fixed our
infrastructure, and we thought,

939
00:44:46,844 --> 00:44:50,404
"Great, now we can move forward."
But then we broke product.

940
00:44:50,504 --> 00:44:52,124
Product could not keep up with us.

941
00:44:52,244 --> 00:44:54,074
There were only three
engineers at this point.

942
00:44:54,174 --> 00:44:55,994
Product could not keep up.

943
00:44:56,094 --> 00:44:57,524
Then we broke design.

944
00:44:57,624 --> 00:45:03,314
Design is consistently behind in our
system, and it's not design's fault.

945
00:45:03,324 --> 00:45:08,154
It's that there's now four other
engineers plus our designer who

946
00:45:08,204 --> 00:45:09,804
actually commits code, right?

947
00:45:09,904 --> 00:45:13,284
Because that's the only way for
him to keep s- keep up with us.

948
00:45:13,384 --> 00:45:15,624
Then we broke sales, right?

949
00:45:15,724 --> 00:45:21,384
I would have to get on sales calls and
consistently during the sales call would

950
00:45:21,394 --> 00:45:24,994
have one of our sales guys be like,
"Yeah, we're gonna have that in, you

951
00:45:24,994 --> 00:45:29,664
know, a couple months." And I'll say,
"No, it's in the product right now."

952
00:45:29,764 --> 00:45:30,124
Right?

953
00:45:30,164 --> 00:45:33,864
I have closed deals because it
was already in the product and

954
00:45:33,864 --> 00:45:35,264
I was the only one that knew,

955
00:45:35,811 --> 00:45:36,171
Bret Fisher: Yeah.

956
00:45:36,179 --> 00:45:36,719
Michael Rollins: right?

957
00:45:36,999 --> 00:45:37,888
Um, we

958
00:45:37,931 --> 00:45:38,241
Bret Fisher: fresh

959
00:45:38,403 --> 00:45:38,723
Michael Rollins: Yeah.

960
00:45:38,823 --> 00:45:42,373
We, we don't have much marketing, so I
don't know if we've broken marketing.

961
00:45:42,373 --> 00:45:43,443
I suspect we have.

962
00:45:43,543 --> 00:45:47,013
But when you're in a position where
the engineering organization is now the

963
00:45:47,013 --> 00:45:53,043
fastest organization in the company,
the only way for you to move safely

964
00:45:53,083 --> 00:45:57,673
as a business is for your engineers
to know how the business works, right?

965
00:45:57,773 --> 00:46:02,413
So before we ship things, the engineers
that work with me know, " Hey,

966
00:46:02,513 --> 00:46:04,423
do we have any demos coming up?"

967
00:46:04,523 --> 00:46:05,493
' Cause that's a real question.

968
00:46:05,493 --> 00:46:06,033
We're small.

969
00:46:06,083 --> 00:46:08,073
We have to ask that question, right?

970
00:46:08,173 --> 00:46:10,453
Are there any important
things happening today?

971
00:46:10,553 --> 00:46:12,583
Because that's something
else we need to know.

972
00:46:12,683 --> 00:46:15,063
And so these are the questions
that engineers typically don't

973
00:46:15,093 --> 00:46:16,393
ask, and they're gonna have to.

974
00:46:16,493 --> 00:46:20,323
Engineering is different now because
the speed at which engineering can

975
00:46:20,323 --> 00:46:25,043
progress is effectively putting
pressure and force on the rest of the

976
00:46:25,053 --> 00:46:27,413
business and breaking it in the process.

977
00:46:27,513 --> 00:46:31,703
And I've never seen anything like
this, and that's how I know it works.

978
00:46:31,803 --> 00:46:32,113
Bret Fisher: Yeah.

979
00:46:32,213 --> 00:46:37,213
Y- you're describing this as what I
would maybe label as a pro-engineering

980
00:46:37,213 --> 00:46:42,513
culture, where a lot of the messaging
we see in the media is around cost

981
00:46:42,543 --> 00:46:45,113
optimization of reducing engineering.

982
00:46:45,213 --> 00:46:49,313
And what I'm hearing, the the language
out of you, which is clearly part of

983
00:46:49,313 --> 00:46:53,323
the business culture of your company,
isn't looking to save the money by…

984
00:46:53,423 --> 00:46:55,983
Because you're so efficient,
let's reduce you down to two

985
00:46:55,983 --> 00:46:57,663
engineers instead of five, right?

986
00:46:57,663 --> 00:47:00,303
Let's get rid of the designer and
burden someone else with that workflow.

987
00:47:00,403 --> 00:47:03,273
I don't hear that out of you,
and that sound- I'm assuming

988
00:47:03,273 --> 00:47:04,203
that's pretty deliberate.

989
00:47:04,243 --> 00:47:08,043
It's a deliberate strategy to not
starve the engineering organization.

990
00:47:08,193 --> 00:47:11,043
Michael Rollins: Well, I mean, to
be clear, The idea that what we've

991
00:47:11,053 --> 00:47:15,113
built is what we've built with just
four or five engineers is absurd.

992
00:47:15,213 --> 00:47:19,123
There's no way I could've presided
over this thing being built with

993
00:47:19,123 --> 00:47:22,993
the number of engineers that we
have with the, without these tools.

994
00:47:23,093 --> 00:47:25,033
Is AI a cost savings?

995
00:47:25,093 --> 00:47:29,253
Only insofar as it is an
accelerator to your individual team.

996
00:47:29,353 --> 00:47:31,153
AI is not going to be cheap.

997
00:47:31,253 --> 00:47:33,683
Boy, have we discovered that recently.

998
00:47:33,683 --> 00:47:34,103
Yeah.

999
00:47:34,103 --> 00:47:39,663
I'm like, look, man, the minute Claude
Max and the $200 equivalent for OpenAI

1000
00:47:39,693 --> 00:47:43,023
goes away, small companies like mine
are gonna be in a world of hurt.

1001
00:47:43,123 --> 00:47:43,983
It's too expensive.

1002
00:47:44,083 --> 00:47:48,503
So, but the option is, but we can't
afford to do it any differently either, so

1003
00:47:48,603 --> 00:47:48,963
Bret Fisher: Yeah.

1004
00:47:49,063 --> 00:47:50,603
That leads to a question around cost.

1005
00:47:50,613 --> 00:47:54,943
Do you, um… I have this diagram
which I wanna show you real quick.

1006
00:47:55,633 --> 00:47:59,893
I'm trying to explain in my courses
to people that are, you know, not

1007
00:47:59,903 --> 00:48:03,763
near as, as advanced as you all
are yet and, or as experienced.

1008
00:48:03,863 --> 00:48:08,063
And I'm taking the project management
triangle, which was traditionally

1009
00:48:08,113 --> 00:48:10,743
it was the rule was there's
three thing- levers you can pull.

1010
00:48:10,883 --> 00:48:13,593
You get to control two, you
meaning management, and you don't

1011
00:48:13,593 --> 00:48:17,293
get to control all three, and
that is quality, cost, and time.

1012
00:48:17,393 --> 00:48:21,313
And so the whole adage was you might
be able to, you know, you might be

1013
00:48:21,313 --> 00:48:24,903
able to reduce the cost and, and
resources, but that's gonna cause

1014
00:48:24,903 --> 00:48:26,433
the time- timeline to stretch.

1015
00:48:26,433 --> 00:48:29,213
So you move two parts of the triangle,
the third one's gonna be affected.

1016
00:48:29,313 --> 00:48:32,943
And I'm starting to correlate this
to what I'm maybe calling like the AI

1017
00:48:32,943 --> 00:48:36,713
engineer project management triangle,
which is a sort of an internal metric

1018
00:48:36,713 --> 00:48:40,413
that we… Assuming there's not unlimited
budget, which some teams are very lucky

1019
00:48:40,413 --> 00:48:44,213
and they can just token max to the,
all day long, that we really kinda get

1020
00:48:44,213 --> 00:48:45,983
three new choices we didn't have before.

1021
00:48:45,993 --> 00:48:49,763
That is model choice, harness
choice, and where or when to use AI.

1022
00:48:50,273 --> 00:48:53,863
Because obviously you could have AI
robot, you could put an agent into

1023
00:48:53,863 --> 00:48:58,373
production and have it Ralph looping
every second of the day, constantly

1024
00:48:58,373 --> 00:49:01,483
looking at stats in production to
look for any potential failures, and

1025
00:49:01,483 --> 00:49:05,033
it will burn thousands of dollars of
tokens a day if you let it, right?

1026
00:49:05,033 --> 00:49:09,983
So not everyone gets to choose max AI in
every part of their software life cycle.

1027
00:49:10,083 --> 00:49:14,453
So if we get to choose model, which
effectively not only is a time thing

1028
00:49:14,463 --> 00:49:18,533
because Haiku is way faster than Opus,
but it's also very much a cost thing.

1029
00:49:18,633 --> 00:49:22,813
And then which harness choice, which
is maybe more around functionality

1030
00:49:22,833 --> 00:49:26,103
and, ease of use because Claude, I
feel like, is so, so far ahead of

1031
00:49:26,103 --> 00:49:29,003
everyone else that it leads to a lot
of new functionality that we don't

1032
00:49:29,003 --> 00:49:30,263
yet have on some of the other ones.

1033
00:49:30,363 --> 00:49:33,543
And then when and where to use AI,
which some organizations maybe don't

1034
00:49:33,543 --> 00:49:36,423
even let you put it into production
yet for helping with troubleshooting.

1035
00:49:36,623 --> 00:49:39,783
And then there's these esoteric
things of the environmental impact,

1036
00:49:39,793 --> 00:49:42,733
the privacy impact, and then the
political impact of certain things.

1037
00:49:42,833 --> 00:49:47,123
I have members of my community that are
in Turkey, for example, where they're,

1038
00:49:47,173 --> 00:49:51,533
relegated to not being able to access
all the models, or they have privacy

1039
00:49:51,533 --> 00:49:54,663
rules where they're not l- allowed
to move this data outside their data

1040
00:49:54,663 --> 00:49:57,953
center, so they have to then be able
to somehow house local models that they

1041
00:49:57,953 --> 00:49:59,393
have to run open weighted or whatever.

1042
00:49:59,493 --> 00:50:03,183
I was curious if there's anything in
here of value that you would think

1043
00:50:03,283 --> 00:50:07,753
that, my idea has merit in the fact
that this is a new paradigm that we've

1044
00:50:07,753 --> 00:50:11,253
never in the past, I don't recall the
last time I had to worry about cost

1045
00:50:11,353 --> 00:50:15,893
in my daily activities that might blow
my budget as an engineering manager.

1046
00:50:15,993 --> 00:50:16,863
This is a new thing.

1047
00:50:16,963 --> 00:50:19,803
Like maybe cloud, but that
wasn't happening in a day, right?

1048
00:50:19,863 --> 00:50:22,223
Michael Rollins: So first off, the most
important part of this is the harness.

1049
00:50:22,323 --> 00:50:25,933
There was a Stanford paper, and
the Stanford paper came out and

1050
00:50:25,933 --> 00:50:30,653
basically said, " You can improve the
performance of the model by 6X if the

1051
00:50:30,653 --> 00:50:36,033
harness is correct." And so harness
is really what you need to focus on.

1052
00:50:36,093 --> 00:50:41,163
And while Claude Code is one of the
best harnesses in the world, if not the

1053
00:50:41,163 --> 00:50:44,253
best, there's still an extraordinary
amount of room for optimization.

1054
00:50:44,353 --> 00:50:49,003
So our models, our harness, so we
actually, yes, I code in Claude

1055
00:50:49,003 --> 00:50:53,313
Code, but in our production
environment, we deploy agents.

1056
00:50:53,313 --> 00:50:55,863
This is what we sell, is
this agent infrastructure.

1057
00:50:55,963 --> 00:50:58,703
And our agents write code.

1058
00:50:58,803 --> 00:51:01,943
Now, our agents do not get tool calls.

1059
00:51:02,043 --> 00:51:06,463
I don't let the LLM touch
many tool calls at all.

1060
00:51:06,563 --> 00:51:12,973
What I do instead is I package, we package
the tool calls into an agent SDK, and

1061
00:51:13,113 --> 00:51:17,933
the only way for the model to touch the
tool is through writing Python code.

1062
00:51:18,033 --> 00:51:20,633
Now, what does that end up doing for us?

1063
00:51:20,683 --> 00:51:25,783
Well, number one, most models are really,
really good at writing Python code.

1064
00:51:25,883 --> 00:51:29,383
You can actually deploy small
models to write Python code, right?

1065
00:51:29,483 --> 00:51:34,353
But we realized we have this im- immense
cost saving right now that we have

1066
00:51:34,353 --> 00:51:35,943
not leveraged, and we're going to.

1067
00:51:36,043 --> 00:51:40,983
Because if our agents are writing
code to express themselves constantly,

1068
00:51:41,083 --> 00:51:42,463
well, I know how to save code.

1069
00:51:42,563 --> 00:51:46,703
As long as I can find that code
again, I can actually go reuse it.

1070
00:51:46,783 --> 00:51:49,443
I don't have to spend the
tokens recreating it, right?

1071
00:51:49,603 --> 00:51:54,603
And all of this, I've built this Instagram
agent, that I've been using for m- my

1072
00:51:54,723 --> 00:52:00,243
Instagram account, and I actually built
my own Instagram integration in Claude

1073
00:52:00,243 --> 00:52:04,103
Code, and I moved it into my agent,
and I said, "Whenever you do anything

1074
00:52:04,103 --> 00:52:05,473
with Instagram, go use this code."

1075
00:52:05,573 --> 00:52:10,003
And so what I found is that it has
been using it and improving it daily.

1076
00:52:10,103 --> 00:52:10,373
Bret Fisher: Mm-hmm.

1077
00:52:10,473 --> 00:52:14,793
Michael Rollins: And I realized, oh,
I think this agent is now using less

1078
00:52:14,803 --> 00:52:18,833
tokens, ' cause it just goes and
runs the code it's already built.

1079
00:52:18,933 --> 00:52:21,773
And so that's one of those things
where it's oh, well, if the harness

1080
00:52:21,793 --> 00:52:26,013
can be used to make the model better,
the harness can also be used to

1081
00:52:26,013 --> 00:52:29,923
reduce the number of tokens that are
necessary to run on a daily basis.

1082
00:52:30,023 --> 00:52:32,863
And so putting the harness at the
center of that thing is actually

1083
00:52:32,863 --> 00:52:34,203
the way to look at it, right?

1084
00:52:34,223 --> 00:52:38,993
The central connector between the
cost of the LLM, the time that the

1085
00:52:38,993 --> 00:52:44,403
LLM takes to run, and whether or
not it's safe to run is the harness.

1086
00:52:44,503 --> 00:52:48,833
And in point of fact, you can even
structure your harness such that

1087
00:52:48,933 --> 00:52:54,373
you can observe the inputs and
outputs to the harness without the…

1088
00:52:54,523 --> 00:52:56,783
ever leaking secrets to the LLM.

1089
00:52:56,883 --> 00:52:56,893
Bret Fisher: Mm-hmm.

1090
00:52:57,037 --> 00:52:59,517
Michael Rollins: Because one of
the other things we do is we proxy

1091
00:52:59,517 --> 00:53:02,827
everything in and out of the harness
that's running on our remote system.

1092
00:53:02,927 --> 00:53:06,287
And since we're proxying it, I
can actually inspect the traffic,

1093
00:53:06,387 --> 00:53:08,607
and I can do that with code.

1094
00:53:08,707 --> 00:53:10,557
I don't have to do that with an LLM.

1095
00:53:10,657 --> 00:53:12,167
And- Yeah … it's fast.

1096
00:53:12,277 --> 00:53:13,077
I know how to do it.

1097
00:53:13,107 --> 00:53:14,567
I know the things I'm looking for.

1098
00:53:14,597 --> 00:53:18,197
And so now if I start seeing the
harness producing stuff that it should

1099
00:53:18,197 --> 00:53:20,977
not produce, then I can smack it down.

1100
00:53:21,037 --> 00:53:25,377
And conversely, I don't have to
give the harness all the secrets.

1101
00:53:25,477 --> 00:53:29,407
I can inspect the traffic on the way
out and insert the secrets in flight

1102
00:53:29,507 --> 00:53:31,527
without ever exposing them to an LLM.

1103
00:53:31,537 --> 00:53:35,137
So the real the center of
that triangle is the harness.

1104
00:53:35,237 --> 00:53:37,267
It is the most important
thing in what we're doing.

1105
00:53:37,297 --> 00:53:39,217
Now, are the models, amazing?

1106
00:53:39,277 --> 00:53:39,737
Yes.

1107
00:53:39,837 --> 00:53:42,127
But now the models are good enough.

1108
00:53:42,207 --> 00:53:43,187
Go focus on the harness.

1109
00:53:43,287 --> 00:53:43,597
Bret Fisher: Yeah.

1110
00:53:43,797 --> 00:53:45,767
Is that, are you using
LiteLLM as your proxy?

1111
00:53:45,867 --> 00:53:46,027
Michael Rollins: No.

1112
00:53:46,029 --> 00:53:46,359
Bret Fisher: Do you know?

1113
00:53:46,459 --> 00:53:47,639
Michael Rollins: No, we rolled our own.

1114
00:53:47,739 --> 00:53:48,059
Bret Fisher: Oh, okay.

1115
00:53:48,159 --> 00:53:51,829
Michael Rollins: So in terms of bringing
in the LLM, we just bring the LLM in

1116
00:53:51,839 --> 00:53:54,459
through the, um I don't like LiteLLM.

1117
00:53:54,559 --> 00:53:57,679
There were at least some significant
issues with it previously.

1118
00:53:57,809 --> 00:54:02,689
We use the LangSmith LangGraph library,
and we just make sure that our LLMs

1119
00:54:02,709 --> 00:54:05,059
conform to that particular structure.

1120
00:54:05,159 --> 00:54:07,559
Bret Fisher: We've got teams that I'm
working with that are trying to figure

1121
00:54:07,559 --> 00:54:11,029
out when it comes to local human agent
harnesses, that, you know, they're

1122
00:54:11,029 --> 00:54:16,919
trying to understand usage per person
and trying to figure out, give model

1123
00:54:16,919 --> 00:54:21,499
access as well as maybe inject things
at some of the sandboxing tools are

1124
00:54:21,499 --> 00:54:25,309
also learning how to inject things,
in the, on the fly in the HTTPS.

1125
00:54:25,409 --> 00:54:28,729
so the last question I was gonna see
if I could wrap this up with the last

1126
00:54:28,729 --> 00:54:31,634
question, and I'm gonna show you a
graph again 'cause I have another one I

1127
00:54:31,634 --> 00:54:34,539
spend, I try to spend some of my time…
I'm trying to, and I'm really bad at

1128
00:54:34,539 --> 00:54:35,609
it, so I'm forcing myself to do it.

1129
00:54:35,709 --> 00:54:38,119
so what I have on screen is something
that I'm working on and calling the AI

1130
00:54:38,119 --> 00:54:42,969
Engineer Maturity Model, but it comes
from a different aspect than every single

1131
00:54:42,989 --> 00:54:45,909
other one that I see on the internet,
and I spend a lot of time looking at what

1132
00:54:45,909 --> 00:54:49,249
everyone else is talking about, of like,
how do I onboard my team to the skill,

1133
00:54:49,249 --> 00:54:53,739
the human skills they need to really
complete the life, the software life

1134
00:54:53,739 --> 00:54:56,649
cycle of where all the AIs belong, right?

1135
00:54:56,699 --> 00:54:58,869
It's not just about
Claude Code on my desktop.

1136
00:54:58,869 --> 00:55:01,599
There's lots of other opportunities, and
of course, the industry is now starting

1137
00:55:01,599 --> 00:55:04,379
to figure out that if you speed up the
number of PRs you're shipping, but you're

1138
00:55:04,379 --> 00:55:07,319
not improving the things after that,
then you're just gonna have a giant

1139
00:55:07,349 --> 00:55:10,939
bottleneck in your PR stack, and your
production won't actually get faster.

1140
00:55:10,949 --> 00:55:15,919
But this screen that I'm looking at is
a fictional screen that on it is the

1141
00:55:15,919 --> 00:55:20,449
idea of over time, as you implement
different human workflows or different

1142
00:55:20,459 --> 00:55:25,109
human skill sets that the team ramps
up on, we assume that your overall

1143
00:55:25,109 --> 00:55:28,529
team productivity improves, and there
will be times, like you said, when you

1144
00:55:28,529 --> 00:55:31,539
first are doing certain activities that
team productivity actually declines.

1145
00:55:31,549 --> 00:55:34,529
Like the day that you implement Claude
Code for the whole team and they're all

1146
00:55:34,529 --> 00:55:36,219
using it, that's a rough week, right?

1147
00:55:36,279 --> 00:55:39,439
That's a really unique thing you need to
give people time to onboard, so there's

1148
00:55:39,439 --> 00:55:41,259
probably gonna be a lower productivity.

1149
00:55:41,269 --> 00:55:45,009
And then there's other points at where
I see teams that maybe they've, they

1150
00:55:45,009 --> 00:55:49,109
fully adopt agent harnesses locally, but
they really haven't done much in CI yet.

1151
00:55:49,209 --> 00:55:52,029
They're barely even
doing PR review agents.

1152
00:55:52,039 --> 00:55:53,599
They maybe not even using that yet.

1153
00:55:53,699 --> 00:55:56,739
And they realize now that they have
this CI backlog, so I would argue

1154
00:55:56,739 --> 00:55:59,429
that becomes a team productivity
dip because now they're spending all

1155
00:55:59,429 --> 00:56:02,639
their time staring at code that AI
built, and they haven't really kind

1156
00:56:02,639 --> 00:56:04,179
of leapfrogged to the next level.

1157
00:56:04,189 --> 00:56:09,189
So I was looking at this like imaginary
list of things you can do in a team

1158
00:56:09,209 --> 00:56:12,269
to improve the overall productivity
of your software deployments.

1159
00:56:12,369 --> 00:56:15,799
Do you see like activities or
certain things that sort of rise up

1160
00:56:15,899 --> 00:56:19,949
as the obvious things besides just
using Claude Code on your desktop?

1161
00:56:20,029 --> 00:56:24,229
Is there activities that you think are a
big tell on whether the team is mature or

1162
00:56:24,229 --> 00:56:26,862
not in their AI, s- software life cycle?

1163
00:56:27,012 --> 00:56:30,132
Michael Rollins: Um, I
hadn't thought much about it.

1164
00:56:30,352 --> 00:56:33,332
Uh, bec- partially because I
haven't encountered teams that are

1165
00:56:33,332 --> 00:56:34,622
doing it the way we're doing it.

1166
00:56:35,352 --> 00:56:38,222
Um, I b- I promise you, I've
had a lot of people look at me

1167
00:56:38,222 --> 00:56:39,762
directly and tell me I'm insane.

1168
00:56:39,862 --> 00:56:40,902
They're like, "This is insane.

1169
00:56:40,922 --> 00:56:44,092
This isn't real software engineering."
And then they come back, they're like,

1170
00:56:44,152 --> 00:56:45,822
"Oh." Here's what I can tell you.

1171
00:56:45,872 --> 00:56:49,772
The end of that graph looked like
this gentle line that just went up.

1172
00:56:49,792 --> 00:56:49,852
Yeah.

1173
00:56:49,952 --> 00:56:54,662
And there's a point in there where
that gentle line just becomes vertical

1174
00:56:54,762 --> 00:56:59,792
And that's the point where you do have
agents fixing things without human input.

1175
00:56:59,892 --> 00:57:04,852
The big milestones for us were
we started from, you know,

1176
00:57:04,922 --> 00:57:06,782
everybody's using AI to code.

1177
00:57:06,882 --> 00:57:10,352
We moved into, " Hey, "When's
the last time any of y'all

1178
00:57:10,352 --> 00:57:11,892
actually looked at the code?"

1179
00:57:11,992 --> 00:57:15,462
And all my team was like, like,
everybody didn't wanna get caught.

1180
00:57:15,482 --> 00:57:18,492
I was like, "Oh, 'cause I haven't
looked at it in months." Yeah,

1181
00:57:18,492 --> 00:57:19,842
and they were like, " Oh, God.

1182
00:57:19,862 --> 00:57:20,332
Okay.

1183
00:57:20,492 --> 00:57:20,602
Phew.

1184
00:57:20,662 --> 00:57:21,142
Thank you.

1185
00:57:21,162 --> 00:57:21,602
Thank you."

1186
00:57:21,796 --> 00:57:23,326
Bret Fisher: They, They didn't know
where you were going with it, right?

1187
00:57:23,336 --> 00:57:23,896
Like, you were, you

1188
00:57:24,010 --> 00:57:24,280
yeah.

1189
00:57:24,376 --> 00:57:24,646
yeah.

1190
00:57:24,696 --> 00:57:28,406
Oh, by the way, imposter syndrome
that jumps out when the junior-est

1191
00:57:28,426 --> 00:57:32,766
member of the team posts something in
the engineering Slack that calls out

1192
00:57:32,896 --> 00:57:36,596
something the most advanced engineer's
done, and you're like, " Oh, God, I'm

1193
00:57:36,596 --> 00:57:38,406
not even a real engineer." Like, how…

1194
00:57:38,436 --> 00:57:41,456
And then you're like, "Oh, wait, you
had Claude look at that, didn't you?"

1195
00:57:41,456 --> 00:57:42,696
And they're like, "Actually, no.

1196
00:57:42,706 --> 00:57:43,940
It was Codex." You know?

1197
00:57:44,040 --> 00:57:46,980
But it is everybody using AI full time?

1198
00:57:47,080 --> 00:57:50,460
The next step is when was the
last time anybody read code?

1199
00:57:50,560 --> 00:57:52,270
Not just write it, read it.

1200
00:57:52,370 --> 00:57:54,200
Do you have full CI/CD yet?

1201
00:57:54,300 --> 00:57:58,200
And then beyond that, it's as simple
as, you know, I'm sure that you

1202
00:57:58,200 --> 00:58:01,440
could throw some milestones in there,
you know, what's your… How fast

1203
00:58:01,440 --> 00:58:03,720
is it to sh- to fix a bug, right?

1204
00:58:03,860 --> 00:58:07,900
Because in lieu of a perfect
release, how fast can you fix

1205
00:58:07,900 --> 00:58:09,530
the release if it goes sideways?

1206
00:58:09,530 --> 00:58:13,190
By the way, you did ask one question:
Do you have a single button rollback?

1207
00:58:13,290 --> 00:58:15,760
We do, but it's only good for 10 minutes.

1208
00:58:15,860 --> 00:58:16,270
Okay?

1209
00:58:16,370 --> 00:58:17,380
So we gotta be real careful.

1210
00:58:17,480 --> 00:58:20,990
but then finally is can the
agents just do your job for

1211
00:58:20,990 --> 00:58:22,880
you, and under what constraint?

1212
00:58:22,980 --> 00:58:28,180
And that is when I think you're at
full AI maturity, that when it is safe

1213
00:58:28,180 --> 00:58:32,610
enough to let agents move in your system
without getting human interaction, you

1214
00:58:32,956 --> 00:58:37,996
are now scaling yourself infinitely,
because every step change that you

1215
00:58:38,076 --> 00:58:43,576
add to those autonomous agents is an
infinite scaler for the rest of the team.

1216
00:58:43,676 --> 00:58:47,136
One thing I will point out, though,
and we can close on this, is when

1217
00:58:47,136 --> 00:58:51,836
you make mistakes in this style
of engineering, they're big.

1218
00:58:51,936 --> 00:58:54,306
I'm not talking about shipping bugs, okay?

1219
00:58:54,406 --> 00:58:58,186
I'm talking about a misstep
in the architecture.

1220
00:58:58,286 --> 00:59:04,686
I'm talking about a misstep in, holy
shit, how did we manage to ship that

1221
00:59:04,706 --> 00:59:06,916
migration order on the database?

1222
00:59:07,016 --> 00:59:12,276
When you make a mistake like that in
this style of engineering, it's not

1223
00:59:12,276 --> 00:59:15,016
a self-wound, it is a decapitation.

1224
00:59:15,116 --> 00:59:18,336
And so every engineering team
will hit the point where they're

1225
00:59:18,336 --> 00:59:23,166
like, " Oh my God, what have we
done?" And we've done it twice.

1226
00:59:23,266 --> 00:59:28,156
And both times we ended up having to
regress the system by two weeks, which

1227
00:59:28,156 --> 00:59:32,046
doesn't sound like a lot until you think
about the fact that we have commits

1228
00:59:32,056 --> 00:59:35,076
that have been 60 commits behind.

1229
00:59:35,116 --> 00:59:39,036
We've had PRs 60 commits behind
because they were five days old.

1230
00:59:39,136 --> 00:59:41,936
Michael Rollins: So the cost
of the catastrophic failures is

1231
00:59:41,936 --> 00:59:44,656
higher, and that's the one thing
you do have to think through, so

1232
00:59:44,756 --> 00:59:47,246
Bret Fisher: Do you think about that
in terms of are there different tiers

1233
00:59:47,246 --> 00:59:51,896
of risk in the way you approach AI in
different repos or different parts of a

1234
00:59:51,896 --> 00:59:55,226
repo where you're like, "Well, this one's
dealing with the database migration, so

1235
00:59:55,276 --> 00:59:57,356
we have extra safety involved with that.

1236
00:59:57,426 --> 00:59:59,936
Michael Rollins: Yeah, so we
do have-- We use an owner file.

1237
01:00:00,036 --> 01:00:01,736
we need to be using an owner file.

1238
01:00:01,746 --> 01:00:03,316
I just keep forgetting to put it in.

1239
01:00:03,416 --> 01:00:06,436
But there are very specific
files that if we- Say again?

1240
01:00:06,536 --> 01:00:07,476
Bret Fisher: Get confession hour.

1241
01:00:07,476 --> 01:00:08,186
This is confession

1242
01:00:08,194 --> 01:00:08,594
Michael Rollins: Yeah.

1243
01:00:08,654 --> 01:00:10,104
Oh, believe me, we're not perfect.

1244
01:00:10,184 --> 01:00:13,684
Everything I've told you has been
relatively true, and maybe I've

1245
01:00:13,684 --> 01:00:15,564
waved my hands once or twice, right?

1246
01:00:15,664 --> 01:00:17,864
but the-- Yeah, we need
to put the owner file in.

1247
01:00:17,884 --> 01:00:20,184
If our system prompt changes,
I need to know about it.

1248
01:00:20,214 --> 01:00:23,264
If our list of tools changes,
I need to know about it.

1249
01:00:23,284 --> 01:00:27,714
If our, look, we ship multiple database
migrations in almost every single PR.

1250
01:00:27,814 --> 01:00:32,724
I mean, I cannot explain how many
times we have metamorphized the

1251
01:00:32,724 --> 01:00:34,694
database, in just a few days.

1252
01:00:34,804 --> 01:00:38,304
but there are files that if they
change, I want to know about it.

1253
01:00:38,314 --> 01:00:42,974
And by the way, one of those is going
to be, like, what are your imports?

1254
01:00:43,164 --> 01:00:45,104
What node modules are you bringing in?

1255
01:00:45,124 --> 01:00:47,274
What Python libraries are you bringing in?

1256
01:00:47,304 --> 01:00:48,414
'Cause those are important as well.

1257
01:00:48,514 --> 01:00:48,884
Bret Fisher: Yeah.

1258
01:00:48,984 --> 01:00:52,164
I can see how that would trigger
different, additional safety checks

1259
01:00:52,184 --> 01:00:56,084
in GitHub Actions where… I mean,
I do at times make directory-scoped

1260
01:00:56,124 --> 01:00:59,774
actions that are… It's also
really good just for any action.

1261
01:00:59,784 --> 01:01:03,124
This is a pro tip with GitHub Actions is
a lot of times your actions that run all

1262
01:01:03,124 --> 01:01:06,384
the time don't need to run on s- they only
need to run on certain types of changes.

1263
01:01:06,484 --> 01:01:10,064
And GitHub Actions, turns out they c-
actually have a lot of filters involved

1264
01:01:10,074 --> 01:01:14,684
so that you can say, "Hey, don't run the
action linter unless the action file's

1265
01:01:14,684 --> 01:01:17,904
changed," and that's only in this one
directory, so there's no even reason

1266
01:01:17,904 --> 01:01:19,554
to even fire it off unless it hits

1267
01:01:19,606 --> 01:01:22,426
Michael Rollins: I probably could,
if I probably paid you for an

1268
01:01:22,426 --> 01:01:26,636
afternoon, we I could get so many more
efficiencies out of our system, dude.

1269
01:01:26,636 --> 01:01:27,106
It's insane.

1270
01:01:27,116 --> 01:01:29,046
Look, I'm not DevOps,
I'm not infrastructure,

1271
01:01:29,220 --> 01:01:30,800
Bret Fisher: Yeah, at this
point I'm just a skill.

1272
01:01:30,860 --> 01:01:31,520
I'm just a skill.

1273
01:01:31,520 --> 01:01:33,240
I just have a GitHub
Actions workflow skill.

1274
01:01:33,290 --> 01:01:37,060
My new rule is every new conversation
I start with an AI needs a skill.

1275
01:01:37,070 --> 01:01:38,330
If it's a one-off question, fine.

1276
01:01:38,330 --> 01:01:40,810
But if I'm planning on this gonna take
an hour or two, we're gonna have a

1277
01:01:40,810 --> 01:01:45,190
conversation about it, I'm looking for
a skill to give that agent to enhance

1278
01:01:45,290 --> 01:01:48,510
its ability, because I personally have
noticed on my projects in the last two

1279
01:01:48,510 --> 01:01:54,290
months that the skill in every case has
always improved the outcomes, or at least

1280
01:01:54,390 --> 01:01:58,260
got me faster to the outcome, whether
it's making sure the skill is teaching

1281
01:01:58,260 --> 01:02:00,970
it the latest version of Go, making
sure that it's adhering to the latest Go

1282
01:02:00,970 --> 01:02:04,870
patterns, or whether it's the front-end
design skill that everybody's using, or

1283
01:02:04,870 --> 01:02:08,380
whether it's a GitHub Action skill that
enforces certain linters and certain

1284
01:02:08,530 --> 01:02:10,520
security patterns that I behave with.

1285
01:02:10,580 --> 01:02:12,740
I just feel like there's always a
good, good reason to have at least

1286
01:02:12,740 --> 01:02:14,160
one skill in your, conversations.

1287
01:02:14,260 --> 01:02:14,590
Pro tip.

1288
01:02:14,690 --> 01:02:14,860
Michael Rollins: Yeah.

1289
01:02:14,960 --> 01:02:15,370
Bret Fisher: Hey, man.

1290
01:02:15,470 --> 01:02:17,900
So what's the one thing you're
gonna excited to do next?

1291
01:02:17,950 --> 01:02:20,620
What have you not done that's, on your
to-do list of, I wanna try this out"?

1292
01:02:20,720 --> 01:02:21,520
Michael Rollins: Turn agents loose.

1293
01:02:21,620 --> 01:02:24,340
Get them to fix things end to
end without me ever knowing.

1294
01:02:24,440 --> 01:02:25,700
Bret Fisher: Continual self-improvement?

1295
01:02:25,800 --> 01:02:29,660
Michael Rollins: To be able to have
an agent pick up a report from Slack,

1296
01:02:29,760 --> 01:02:34,410
figure out if it's real, create the
issue, fix the issue, commit the code,

1297
01:02:34,510 --> 01:02:38,720
build the PR, test everything, merge it
to main, and ship it, and I never know.

1298
01:02:38,820 --> 01:02:42,480
When that starts happening, our
engineering team of five will look

1299
01:02:42,480 --> 01:02:43,960
like an engineering team of 80.

1300
01:02:44,060 --> 01:02:47,990
Bret Fisher: Check out agentic workflows
in GitHub if you haven't seen that yet.

1301
01:02:48,050 --> 01:02:51,740
it's a-- That's the actual term,
agentic workflows, because it writes…

1302
01:02:51,750 --> 01:02:55,880
it's like a full safety version of
putting an AI prompt in a workflow

1303
01:02:55,890 --> 01:02:58,310
without all the risky behaviors
that everybody's using right now.

1304
01:02:58,410 --> 01:03:00,540
There's like sandboxing, there's
all these different extractions.

1305
01:03:00,720 --> 01:03:02,960
And one of the things that it
can do is you can put it on

1306
01:03:02,960 --> 01:03:04,300
like a, kinda like a Ralph loop.

1307
01:03:04,400 --> 01:03:07,540
According to what I heard yesterday, was
they designed their system so that it's

1308
01:03:07,540 --> 01:03:11,460
not all Monday at 9:00 AM, and like they,
they, they distribute these things out.

1309
01:03:11,480 --> 01:03:16,620
It has inherent, rules inside it, so
if you ask it to be an issue reviewer,

1310
01:03:16,720 --> 01:03:18,530
it won't do all the issues right now.

1311
01:03:18,630 --> 01:03:21,782
Like it's not gonna l- overload you with
all of this work, which is what a lot

1312
01:03:21,782 --> 01:03:23,560
of people I think do, and initially the

1313
01:03:23,726 --> 01:03:24,226
Michael Rollins: I will

1314
01:03:24,326 --> 01:03:25,166
Bret Fisher: them a bunch of work.

1315
01:03:25,266 --> 01:03:28,466
Michael Rollins: I will say this, if
Copilot is my only choice, I won't use it.

1316
01:03:28,566 --> 01:03:28,886
Bret Fisher: Yeah.

1317
01:03:28,986 --> 01:03:30,156
Well, none of this has to do with Copilot.

1318
01:03:30,166 --> 01:03:30,316
You

1319
01:03:30,316 --> 01:03:30,586
Michael Rollins: Okay.

1320
01:03:30,686 --> 01:03:30,956
good.

1321
01:03:30,976 --> 01:03:31,326
'Cause this,

1322
01:03:31,356 --> 01:03:32,346
Bret Fisher: All of this with Claude Code.

1323
01:03:32,536 --> 01:03:33,886
Yeah, you pick your own agent.

1324
01:03:34,022 --> 01:03:34,832
Michael Rollins: Okay.

1325
01:03:34,862 --> 01:03:36,452
Bret Fisher: framework is agent agnostic,

1326
01:03:36,608 --> 01:03:36,978
Michael Rollins: Excellent.

1327
01:03:37,078 --> 01:03:38,088
Then yes, I may try it.

1328
01:03:38,188 --> 01:03:38,418
So

1329
01:03:38,518 --> 01:03:40,308
Bret Fisher: yeah, I think the,
the push button features in

1330
01:03:40,308 --> 01:03:41,618
GitHub are all Copilot, right?

1331
01:03:41,618 --> 01:03:43,378
You turn on the PR
review in your settings.

1332
01:03:43,378 --> 01:03:45,568
that's, th- these are just,
features that happen to use Copilot.

1333
01:03:45,578 --> 01:03:48,678
This is more of a hobby kit approach,
where you're at the command line

1334
01:03:48,678 --> 01:03:50,368
with this stuff and using YAML files.

1335
01:03:50,468 --> 01:03:52,028
I'll show it to you someday if
you don't ever get a chance.

1336
01:03:52,128 --> 01:03:53,908
Mike, thanks so much for being here.

1337
01:03:53,958 --> 01:03:56,858
Can you give us your full
name, and title, and company?

1338
01:03:56,958 --> 01:04:00,888
Michael Rollins: Mike Rollins,
CTO of Relify and Family Digital.

1339
01:04:00,988 --> 01:04:04,588
You can also find me at rollins.io online.

1340
01:04:04,638 --> 01:04:09,258
Instagram is rollins.io, and uh, I
think that's the major points there

1341
01:04:09,258 --> 01:04:12,378
Bret Fisher: You talk about AI on
Instagram, so if people wanna hear

1342
01:04:12,378 --> 01:04:15,638
more thoughts from you, I would
recommend Instagram, Awesome, man.

1343
01:04:15,638 --> 01:04:18,118
Well, thank you so much for being here,
and we'll, I'm gonna definitely have you

1344
01:04:18,118 --> 01:04:21,568
back on 'cause we're gonna talk about
you pushing the envelope even further, so

1345
01:04:21,668 --> 01:04:22,068
Michael Rollins: Let's do it.

1346
01:04:22,401 --> 01:04:24,741
Bret AI July 2025: Thanks for joining
us, and I'll see you in the next episode.