Rob Zuber is seeing the gap widen

WEBVTT
00:00.000 --> 00:03.052
So we actually came up with a new metric
which we call merge efficiency.
00:04.088 --> 00:07.770
How many builds on a branch does it take
before you can merge something to main?
00:07.770 --> 00:13.974
And the really successful companies are
seeing the same sort
00:13.974 --> 00:16.995
of merge rate or higher as they get
better,
00:16.995 --> 00:20.947
right? And everything is increasing
because they're applying AI,
00:20.947 --> 00:23.309
they're creating more PRs, they're
creating them faster,
00:23.309 --> 00:25.300
they're creating even bigger ones,
00:25.300 --> 00:28.542
yet still faster and still merging faster.
00:28.542 --> 00:32.704
The sort of median company is creating a
lot.
00:32.856 --> 00:37.201
more builds on a branch because their work
is broken and it fails and
00:37.201 --> 00:41.245
it fails and it fails and it fails and
then they finally get it merged into main
00:41.525 --> 00:44.118
or whatever your default branch is that we
assume goes to production.
00:43.556 --> 00:44.556
Sure.
00:44.483 --> 00:45.483
Yep.
00:46.186 --> 00:49.757
And your final productivity, you're doing
a lot more work.
00:49.757 --> 00:52.678
You're using more tokens and more energy
and whatever.
00:52.678 --> 00:55.879
But then the final output is kind of the
same.
00:55.879 --> 00:56.879
And so that's
00:56.356 --> 00:57.356
Mm.
00:56.839 --> 01:00.010
a really interesting gap. We've been
trying to help people solve the for that
gap,
01:00.010 --> 01:03.051
right? Like, how do you get more
confidence earlier?
01:03.051 --> 01:06.392
How do you steer your agent? Assuming
you're using an agent to build
01:06.412 --> 01:11.033
How do you steer your tools so that you're
getting good outcomes earlier?
01:11.033 --> 01:14.955
Rob Zuber is a veteran software leader
with over two decades of startup
experience,
01:14.955 --> 01:18.658
currently serving as the chief technology
officer at CircleCI.
01:18.658 --> 01:23.221
He joined the company back in 2014
following Circle CI's acquisition of
Distiller,
01:23.221 --> 01:26.283
a continuous integration and deployment
platform that he co-founded.
01:26.283 --> 01:30.366
Prior to that, Rob established a strong
track record as a serial entrepreneur,
01:30.366 --> 01:33.178
co-founding technology companies such as
Copius and Yoohoot
01:34.012 --> 01:38.135
Currently, he oversees product innovation
and leads CircleCI's globally distributed
01:38.135 --> 01:42.388
engineering team. And today, Rob Zuber is
our guest on the Engineering with
01:42.388 --> 01:43.388
AI podcast.
02:00.939 --> 02:03.120
cool. Rob, thank you so much for coming on
the show today.
02:03.120 --> 02:04.311
We really appreciate having you on.
02:05.315 --> 02:06.315
Yeah, thanks for having me, Kyle.
02:06.239 --> 02:07.239
I'm excited to be here.
02:07.523 --> 02:12.664
Yeah, awesome. Well it's I'm I'm really
keen to to find out more about your
specific
02:12.664 --> 02:14.895
position at Circle CI. ~ you know,
02:14.895 --> 02:18.536
I'm I'm on teams that use it. So I'm I'm
curious to know like
02:17.456 --> 02:18.456
Mm-hmm. Awesome.
02:18.536 --> 02:20.346
what you know about like what all's going
on.
02:20.346 --> 02:25.218
~ but let's start with you. you you caught
the little bit of a bio that I read back.
02:25.218 --> 02:27.449
what would you add to that? what do you
think about when you think about your
02:27.449 --> 02:28.680
career? What story would you tell?
02:28.680 --> 02:29.680
Yeah.
02:29.830 --> 02:31.631
Yeah, it's it's a great question.
02:31.631 --> 02:34.223
I mean I despite being forced to talk
about it all the time,
02:34.223 --> 02:35.223
I don't think about it that much.
02:35.173 --> 02:38.486
But I think, you know, w when what one of
the things that happens to me
02:38.486 --> 02:40.087
a lot is people ask me, you know,
02:40.087 --> 02:42.849
what advice do you have for me as sort of
someone entering software?
02:42.373 --> 02:43.373
Mm. Mm-hmm.
02:42.849 --> 02:43.849
And there's
02:43.519 --> 02:45.390
two reasons I have no advice for them,
02:45.390 --> 02:47.192
or I could say this is what I did,
02:47.192 --> 02:48.993
but you know, maybe I don't know if that's
gonna help you.
02:48.993 --> 02:52.335
One, obviously AI is changing everything
and I'm just having this conversation
02:51.291 --> 02:52.313
Yeah. Mm-hmm.
02:52.335 --> 02:55.117
with juniors all the time of like I don't
even know what you should
02:55.117 --> 02:56.568
be thinking about learning. Like,
02:56.568 --> 02:57.568
let's learn together.
02:57.842 --> 02:58.842
Which is really
02:57.927 --> 02:58.927
Mm-hmm.
02:58.362 --> 02:59.913
interesting and different. I can't just,
02:59.913 --> 03:04.366
you know, my my sage advice from 25 years
ago is totally useless at this point,
03:04.366 --> 03:09.270
right? And the other is I I came into
software after graduating college.
03:09.270 --> 03:11.341
So I didn't study, I didn't study computer
science.
03:11.341 --> 03:14.093
~ I studied engineering physics for
whatever that's for.
03:14.093 --> 03:15.464
So I mean I technical background,
03:15.464 --> 03:16.684
but but ~
03:16.085 --> 03:17.085
Yeah.
03:16.965 --> 03:21.057
sort of it was the late 90s, it was the
original dot-com boom.
03:21.057 --> 03:25.550
I mean, anybody kind of that was
interested in finding a job was was moving
into
03:25.572 --> 03:26.572
~ tech.
03:27.323 --> 03:28.984
But then continued that pattern.
03:28.984 --> 03:30.985
I spent some time doing product
management,
03:30.985 --> 03:32.495
some time doing business development,
03:32.495 --> 03:36.667
some time teaching. And so I think a lot
of that exploration,
03:36.667 --> 03:39.528
~ you know, I always came back to
software,
03:39.528 --> 03:40.528
~
03:40.177 --> 03:41.177
Mm.
03:40.529 --> 03:45.511
but helped me build a really good
understanding as an executive of what
03:45.511 --> 03:50.584
my my partners and my colleagues need from
me and a lot more empathy.
03:50.584 --> 03:51.974
Like I think it's pretty classic
03:51.548 --> 03:52.548
Mm.
03:51.974 --> 03:53.775
as a software engineer to be like,
03:54.331 --> 03:55.872
What do these salespeople even do?
03:55.872 --> 03:56.872
We build
03:56.336 --> 03:57.336
Yeah.
03:56.422 --> 03:57.422
everything,
03:57.023 --> 03:58.582
they just get paid for it. You know,
03:58.582 --> 04:04.617
like this kind of nonsense. Spend five
minutes watching a salesperson do their
job,
04:04.617 --> 04:08.179
and you'll be like, I wouldn't have the
slightest clue how to make this work.
04:08.179 --> 04:09.179
Right. Like I
04:08.453 --> 04:09.453
Yeah, yeah, yeah.
04:09.140 --> 04:11.301
would be terrified of every single part of
this.
04:11.301 --> 04:12.882
And I I I remember a few moments,
04:12.882 --> 04:16.685
both as an engineer supporting salespeople
in the field,
04:16.685 --> 04:19.028
and then as someone again doing business
development and sort
04:19.028 --> 04:20.267
of sales-related activities.
04:20.987 --> 04:23.168
~ just being like, no, I know what I'm
good at,
04:23.168 --> 04:26.179
and it's not this. And I'm so thankful
there are people who are good at this,
04:24.653 --> 04:27.117
Yeah. Yeah, yeah. They they do this all
the time.
04:26.179 --> 04:28.469
right? Like, cause then because then when
you go and build companies,
04:27.117 --> 04:28.696
Yeah. Yeah.
04:29.450 --> 04:31.720
you like early, you have to do all those
jobs,
04:31.720 --> 04:34.651
which is another great way to get that
rounded experience to allow you
04:34.651 --> 04:36.462
to become an executive that isn't just
like,
04:36.462 --> 04:37.632
I know how to write software now,
04:37.632 --> 04:38.632
how do I run a company
04:37.851 --> 04:38.851
Yeah, yeah.
04:38.422 --> 04:44.094
sort of thing? ~ but you're so thankful
when people show up with like tons
04:44.094 --> 04:46.795
of experience, you know, doing that job.
04:45.157 --> 04:47.480
Right. Know how it's actually done.
04:46.795 --> 04:47.825
~
04:48.314 --> 04:49.914
Yeah, exactly. Like, you know,
04:49.914 --> 04:52.266
ev even at Circle CI where I I joined,
04:52.266 --> 04:56.648
we were about ~ I think 14 people or
something when when three of
04:56.648 --> 04:59.630
us joined through that acquisition ~ that
you mentioned of Distiller,
04:59.630 --> 05:03.952
you know, we were doing sales for a long
time.
05:03.952 --> 05:06.874
And when the first, you know, proper
salespeople came in,
05:06.874 --> 05:09.185
I was like, thank you for being here.
05:09.185 --> 05:11.066
I'm so glad that I don't you know,
05:11.066 --> 05:14.158
like I I I'm okay. I'm not afraid to talk
to customers,
05:14.158 --> 05:15.939
but there's just, you know,
05:16.453 --> 05:18.204
The process, the steps you go through,
05:18.204 --> 05:20.945
the repeatability, all of it is just not
what what you do.
05:20.945 --> 05:23.726
Right. But I I do think dabbling,
05:23.726 --> 05:25.527
getting some experience,
05:24.335 --> 05:25.484
Mm-hmm. Breath.
05:25.527 --> 05:27.318
you know, playing a salesperson on TV,
05:27.318 --> 05:28.869
whatever, like it gives you enough,
05:28.869 --> 05:32.180
again, empathy and then understanding of
what those folks
05:29.713 --> 05:32.002
Yeah. Yep.
05:32.180 --> 05:35.932
need from you. Cause I think technical
people really get into like,
05:35.932 --> 05:39.984
I explain my technical details and it's
very logical.
05:39.984 --> 05:41.875
So I don't understand why you don't
understand.
05:40.913 --> 05:42.453
Mm. ~
05:42.247 --> 05:43.641
Do you know what mean? And learning
05:42.453 --> 05:43.470
right, right. Mm-hmm.
05:43.641 --> 05:46.751
how other people think and what they're
listening for and what they need
05:46.751 --> 05:49.048
out of a conversation or about out of your
information,
05:48.103 --> 05:49.784
It was different. Yeah.
05:49.048 --> 05:50.048
I think is is really
05:50.465 --> 05:51.916
Yeah. Yeah. For sure. For sure.
05:51.916 --> 05:53.307
I I I love that you mentioned that.
05:53.307 --> 05:57.489
I will never forget the day working as
this is when I was at Bindley,
05:57.489 --> 05:59.267
a little software as a service company.
05:59.267 --> 06:00.732
We ended up getting acquired by Workday.
06:00.732 --> 06:05.323
And ~ I I was sort of on the front line of
a team that had I
06:05.495 --> 06:09.878
you know involved somewhat in the sale and
was part of the conversations about what
06:09.878 --> 06:12.179
we did and didn't have and what they did
and didn't need.
06:12.179 --> 06:14.641
And and then later you get in there and
you realize,
06:14.641 --> 06:17.343
there there was things you didn't know you
needed to tell us.
06:17.595 --> 06:18.806
You know, like they're they're
06:18.236 --> 06:19.236
Mm-hmm.
06:18.806 --> 06:22.478
thinking about the current solution and it
pisses me off in the following two ways.
06:22.478 --> 06:23.898
So you better have it better than that.
06:23.898 --> 06:26.030
But they don't tell you the hundred ways
in which it's already fine,
06:26.030 --> 06:27.280
you know? And so they don't
06:26.768 --> 06:27.768
Right.
06:27.280 --> 06:31.643
necessarily think to check and and you
don't think to ask and yeah,
06:31.643 --> 06:34.524
it's all just like we're all just sort of
crashing into each other in terms
06:34.524 --> 06:37.526
of how all this works. ~ and we're all
doing our best.
06:36.337 --> 06:37.337
Yeah. Yeah.
06:37.526 --> 06:41.008
And I can remember being on the other side
when you're not involved with that
06:41.008 --> 06:42.269
and you're just like, What on earth?
06:42.269 --> 06:43.950
Who said yes to this deal? You know?
06:43.950 --> 06:44.960
And you learn some
06:44.165 --> 06:45.165
Right. Right.
06:44.960 --> 06:46.311
empathy of like, okay, actually.
06:46.531 --> 06:48.112
It's not as easy as it sounds,
06:48.112 --> 06:51.686
right? It it feels like you ought to just
be able to check all these details,
06:51.686 --> 06:53.457
but you kinda can't in the end.
06:53.457 --> 06:58.639
It's yeah, it's I mean, the kind of
asymmetry of hindsight,
06:58.639 --> 06:59.639
meaning
06:59.242 --> 07:00.242
Yeah.
06:59.979 --> 07:00.979
one
07:00.229 --> 07:03.771
person looked at it without the hindsight
and then another person comes along
07:03.771 --> 07:05.401
and looks at it, you know, after the fact.
07:05.401 --> 07:06.401
And
07:05.611 --> 07:06.611
Right.
07:06.081 --> 07:11.083
it's like it's every engineer walking into
someone else's code base is,
07:11.083 --> 07:13.980
you know, you don't even have to take it
to like selling a deal or whatever.
07:11.113 --> 07:13.097
Yeah, yeah. Everything's terrible.
07:13.097 --> 07:16.503
I hate all of this. Yeah. Yeah,
07:13.980 --> 07:16.685
You're like, why would anyone have ever
done this this way?
07:16.503 --> 07:17.503
yeah.
07:16.685 --> 07:17.685
But
07:17.846 --> 07:20.787
but someone very rational, right?
07:20.799 --> 07:21.799
Did. Yeah.
07:21.823 --> 07:24.066
Who would happily look at you some of your
stuff and be like,
07:24.066 --> 07:26.309
what on earth? Right. They they
07:24.681 --> 07:26.086
Yeah, for sure.
07:26.309 --> 07:29.583
had a different context. They were they
had half the information that you now
have,
07:29.583 --> 07:32.036
all those things, right? And ~ I yeah,
07:32.036 --> 07:33.548
I think there's a huge amount in there,
07:33.548 --> 07:36.272
even within our own domain, but then
especially across domains,
07:36.272 --> 07:40.167
to just say, wow, like there's so many
things about this that I don't understand.
07:36.755 --> 07:38.976
Hundred percent. Well
07:40.577 --> 07:42.477
and I like how you mentioned AI is
changing everything.
07:42.477 --> 07:44.628
That I mean that is partially what we're
here to talk about.
07:44.628 --> 07:45.919
And it is true, right?
07:44.970 --> 07:45.970
Yeah. Yeah.
07:45.919 --> 07:49.360
Like this this feels like one of the
biggest changes in my career.
07:49.360 --> 07:51.061
And it sounds like our careers are about
as long.
07:51.061 --> 07:54.143
~ so so let's get into that. Like when
you're using
07:53.707 --> 07:54.707
Mm-hmm.
07:54.143 --> 07:57.044
these tools, when you're sitting down to
do technical work,
07:57.044 --> 07:58.464
if you're keeping your skills sharp,
07:58.464 --> 08:01.276
~ or or you know, organizing, you know,
08:01.276 --> 08:04.087
strategy or anything else, like what what
tools are you reaching for
08:04.760 --> 08:07.562
~ as part of your day to day like
practitioner work these days.
08:07.562 --> 08:08.562
Yeah.
08:08.265 --> 08:11.297
Yeah. And ~ just before I talk about
specific tools,
08:11.297 --> 08:13.579
it's worth saying, because you you kind of
said if this or that,
08:13.579 --> 08:18.862
like I I am. I I'm thinking strategically
we're a software developer tool company.
08:18.862 --> 08:24.576
In order to do that, I can't not I for the
way my brain works and the
08:24.576 --> 08:26.207
way that I think about what our customer
needs,
08:26.207 --> 08:30.811
I can't not use the tools. if I'm just
reading blog posts and saying this
08:30.811 --> 08:32.452
is probably the direction we should go as
a company,
08:32.452 --> 08:33.972
I feel disconnected from my people.
08:33.972 --> 08:35.443
I just feel disconnected from my
customers.
08:35.443 --> 08:37.075
So I am a very hands-on learner.
08:37.772 --> 08:41.714
~ and so I'd say i the basics,
08:41.714 --> 08:48.264
right? I mean, I spend a decent amount of
time with both Claude and ChatGPT.
08:48.264 --> 08:49.266
It kind of depends on the day.
08:49.266 --> 08:50.327
I have funny stories about
08:49.941 --> 08:50.941
Mm-hmm.
08:50.327 --> 08:52.868
how they interact with me and their
personas that some days I'm just
08:52.868 --> 08:54.528
not comfortable with one of them versus
the other,
08:54.528 --> 08:57.039
but it doesn't matter. So a bunch of just
like,
08:57.039 --> 08:58.626
how would I go about this? You know,
08:58.626 --> 09:02.411
what what would be what tools are
available to me to try to achieve this
thing?
09:02.411 --> 09:03.562
What does good look like in this case?
09:03.562 --> 09:06.283
And I always end up in this weird
situation where then.
09:06.655 --> 09:09.797
You know, if I'm using one of the more
chat tools as opposed to like codex
09:09.797 --> 09:13.420
or claw code, they'll spit out some code
and I go,
09:13.420 --> 09:15.111
What am I doing with this tool?
09:15.111 --> 09:18.263
I should be, you know, in a terminal right
now.
09:18.263 --> 09:19.364
Like if you're gonna give me code,
09:18.901 --> 09:20.582
Right. Right. Right.
09:19.364 --> 09:20.595
let me go to and then I
09:20.595 --> 09:24.007
I let's say I'm in clo like Claude Chat.
09:24.007 --> 09:27.029
Then I'll go to Claude Code and say,
09:27.029 --> 09:30.792
Your friend Claude Chat told me and then
like paste some stuff.
09:30.419 --> 09:32.904
Yeah, yeah. huh. Yeah.
09:30.792 --> 09:32.413
I don't know why. I feel it's really
important.
09:32.413 --> 09:33.413
Like, I
09:32.993 --> 09:34.865
haven't been cheating on you, Claude Code.
09:34.865 --> 09:35.865
Like I'm
09:35.667 --> 09:38.218
I'm seeing your brother Claude on the side
or whatever.
09:38.218 --> 09:42.391
Like it's a very weird thing. And ~ I
don't know why.
09:42.391 --> 09:47.814
Like I it and I I try so hard in a way not
to personify
09:47.814 --> 09:49.895
or anthropomorphize like these tools,
09:48.159 --> 09:50.886
Yeah, I know. But it's hard. Yeah.
09:49.895 --> 09:53.197
but they try really hard to have me do
that.
09:50.886 --> 09:54.927
That's right. To anthropomorphize
themselves.
09:53.197 --> 09:55.118
And so there's like this weird
interaction.
09:54.927 --> 09:55.927
Yeah.
09:55.118 --> 09:57.329
Anyway, so so I, you
09:57.329 --> 09:59.190
know, codex and claude code I use,
09:59.190 --> 10:00.641
I wouldn't say interchangeably,
10:00.641 --> 10:03.593
but sometimes I feel like I'm going down a
weird
10:04.467 --> 10:07.820
rabbit hole or something or feel like I'm
off track and I I'll find opening
10:07.820 --> 10:11.783
up another one. ~ mostly because it's
going to give me a different model.
10:11.783 --> 10:14.536
Right. And then of course I've dabbled in
pi (agent SDK) and open weight models
10:14.536 --> 10:17.218
and other stuff just to kind of see where
the state of those
10:15.467 --> 10:17.820
Mm-hmm. Okay. Cool.
10:18.048 --> 10:21.702
because that matters to me in terms of
like everybody's talking about token cost
10:21.702 --> 10:24.284
management. Everybody's talking about like
the evolution of these tools
10:22.623 --> 10:25.393
Yeah, yeah. Yep.
10:24.284 --> 10:25.845
and whatever and optionality.
10:25.845 --> 10:32.821
And but I will switch between sometimes
just for like a fresh perspective.
10:33.344 --> 10:36.486
I guess, right? Like, hey, I got the you
know,
10:36.486 --> 10:37.746
this was built by something else.
10:37.746 --> 10:40.317
~ you know, what's your take on it?
10:40.317 --> 10:41.317
Is there another, you know,
10:40.757 --> 10:41.757
Mm-hmm.
10:41.108 --> 10:44.489
w look at or or you know, and sometimes
I'll do this with the same tool.
10:44.489 --> 10:46.630
Like, okay, cool, we built that together.
10:46.630 --> 10:49.467
One, we have some internal review tools
that we've constructed.
10:49.467 --> 10:52.253
I think everybody's built some review
tools of some point,
10:52.253 --> 10:54.824
right? But like now put your adversarial
hat on,
10:54.824 --> 10:56.095
right? Like if you were checking for this
10:55.637 --> 10:56.637
Mm-hmm.
10:56.095 --> 10:57.756
specifically, do you see
10:58.844 --> 11:02.656
You know, do you see patterns that don't
align with our coding standards?
11:02.656 --> 11:04.427
Do you see issues with security?
11:04.427 --> 11:07.749
Like we we have a bunch of tooling to kind
of force that cycle.
11:07.749 --> 11:09.987
~ a lot of this is is, you know,
11:09.987 --> 11:14.533
we're we're homegrowing and solving our
problems in an effort to build things
11:14.533 --> 11:16.274
we think will solve our customers'
problems.
11:16.274 --> 11:17.274
~
11:16.351 --> 11:17.351
Yeah, yeah.
11:17.294 --> 11:19.325
and so we're, you know, we're trying to
get stuff out there,
11:19.325 --> 11:22.067
but then we're also trying to push the
limits internally and see what what works
11:22.067 --> 11:25.859
and what doesn't. ~ but I think that like
both
11:27.051 --> 11:30.173
Fresh perspective, some and also
sometimes,
11:30.173 --> 11:34.495
you know, like the model providers are
having so many issues with capacity that
11:34.495 --> 11:37.797
sometimes it's just like who who's gonna
answer my query right
11:37.797 --> 11:44.551
now is another driver. So like we we as an
organization earlier this
11:44.551 --> 11:47.942
year, so funny, it feels like it was maybe
10 years ago at the rate that things
11:47.942 --> 11:49.543
are changing, right? But earlier this
year,
11:49.543 --> 11:56.117
we we had been in a place where we were
encouraging folks to go explore any tools.
11:57.045 --> 11:59.316
Pick any of the tool any of the AI tools
off the shelf,
11:59.316 --> 12:01.536
you know, we and we had subscriptions to a
bunch of different ones.
12:01.536 --> 12:03.117
Like we were doing the evaluation,
12:03.117 --> 12:04.627
are we comfortable with their terms of
service,
12:04.627 --> 12:06.177
that kind of stuff. But then saying within
this,
12:06.177 --> 12:09.558
within this pool, because we're a
developer tool provider,
12:09.558 --> 12:12.959
we want people to have experience with all
these tools.
12:12.959 --> 12:18.241
And we found two things. One, a number of
our folks might be trying these things,
12:18.241 --> 12:19.761
but not really giving feedback.
12:19.761 --> 12:21.782
So it wasn't actually valuable to us.
12:20.325 --> 12:22.366
Mm. Mm.
12:21.782 --> 12:22.782
So it
12:22.432 --> 12:25.603
was better to constrain down to a
particular tool set.
12:25.603 --> 12:26.603
And two,
12:27.263 --> 12:30.565
While we like to believe that every
engineer wants to futz about with
12:30.565 --> 12:33.366
all these different tools and is gonna be
really excited about them and whatever,
12:33.366 --> 12:35.288
a lot of people were just like,
12:34.271 --> 12:36.401
Yeah, right. I wanna get my work done.
12:35.288 --> 12:36.869
I have work to get done.
12:36.401 --> 12:37.429
Yeah. Mm-hmm.
12:37.809 --> 12:40.250
Either tell me how to use it and I'll use
it,
12:40.250 --> 12:44.543
or don't tell me and I won't try any of
this stuff because you told me I need
12:44.543 --> 12:46.274
to get this thing done by next week or
whatever,
12:46.274 --> 12:47.945
which is like whatever. It's a such.
12:46.678 --> 12:47.969
I love this. You're you're you're playing
12:47.969 --> 12:49.640
into my into my next set of questions,
12:49.640 --> 12:53.103
right? We we normally sort of open with
like what tools you like versus like what
12:53.103 --> 12:54.944
are you seeing in the time teams you
observe
12:54.739 --> 12:55.739
Yeah.
12:54.944 --> 12:55.944
and and
12:55.925 --> 12:57.006
and it's and it's interesting,
12:57.006 --> 12:59.758
Rob. I mean, you obviously as a tech
leader,
12:59.758 --> 13:01.129
you've got your own team, right,
13:01.129 --> 13:03.841
that you serve and you've got your own
observations there.
13:03.841 --> 13:06.804
But I wonder, like, given your role at
Circle CI,
13:06.804 --> 13:08.705
like you can kind of see what we're all
doing.
13:08.705 --> 13:10.026
what insight does that bring you?
13:10.026 --> 13:15.921
Yeah. And we do. ~ we we have this report
that we put out.
13:15.921 --> 13:17.251
We used to put out every year.
13:17.251 --> 13:18.251
I think
13:17.603 --> 13:18.603
~
13:17.932 --> 13:19.613
we still do like the annual version of it,
13:19.613 --> 13:21.334
right? It's called the state of software
delivery report,
13:21.334 --> 13:23.966
where we started from, you know,
13:23.966 --> 13:27.865
pre-AI era. I think we're in year seven or
eight or something.
13:27.865 --> 13:31.362
~ pre-AI era, you know, Dora metrics,
13:31.362 --> 13:34.134
what can we see about how you know PR
throughput and what
13:32.195 --> 13:33.996
Yeah. Yeah.
13:34.134 --> 13:36.005
like what's actually flowing through the
system?
13:36.389 --> 13:37.389
Mm-hmm.
13:36.389 --> 13:37.389
So a lot,
13:36.749 --> 13:39.842
a lot of what's out there is
survey-driven,
13:39.842 --> 13:42.074
self-reported, which is valuable for sure.
13:42.074 --> 13:46.948
And developers' perception of what's
happening is often more valuable than
what's
13:46.948 --> 13:48.219
actually happening, right? In some cases,
13:48.219 --> 13:52.392
as you're trying to diagnose, but people
don't have the best memories,
13:52.392 --> 13:53.903
right? I can't remember what I had for
lunch.
13:53.123 --> 13:55.029
Right. Right.
13:53.903 --> 13:55.164
So, like, can I remember how many
13:55.164 --> 13:56.906
PRs I got through last week? Like,
13:56.906 --> 13:57.906
not necessarily,
13:57.039 --> 13:58.039
Mm-hmm.
13:57.847 --> 13:59.348
but the systems will tell us, right?
13:59.348 --> 14:02.891
So we do this large analysis across our
entire customer base.
14:02.891 --> 14:05.623
And ~ and we've actually gone to
14:05.829 --> 14:09.911
putting out quarterly updates now because
the state of software delivery
14:09.911 --> 14:12.242
is changing so quickly that
14:11.909 --> 14:12.909
Yeah.
14:12.242 --> 14:14.394
comparing year over year numbers is just
like,
14:14.394 --> 14:15.954
wait, I what was happening last year?
14:15.954 --> 14:17.575
I don't even remember, right? So,
14:17.575 --> 14:20.626
one of the really interesting things that
we saw earlier this year
14:20.626 --> 14:22.198
and we've been seeing continue.
14:22.198 --> 14:25.119
I mean, unsurprisingly we see significant
spreads,
14:25.119 --> 14:30.502
right? Like the difference between median
and P95 is shocking.
14:30.502 --> 14:33.914
And ~ in particular something that shows
up in
14:34.233 --> 14:37.075
In CI numbers, that's been quite telling
is
14:37.636 --> 14:44.400
the difference between increased rate of
branch builds,
14:44.400 --> 14:46.762
which I'll explain why this matters in a
second,
14:46.762 --> 14:49.764
and the rate of main builds, right?
14:49.752 --> 14:50.752
Okay. Yep.
14:49.764 --> 14:50.764
So
14:51.386 --> 14:55.368
if I'm I'm we're seeing significant
increase in CI consumption,
14:55.368 --> 14:58.471
leading companies. So we actually came up
with a new metric which
14:58.471 --> 14:59.992
we call merge efficiency.
15:01.028 --> 15:04.710
How many builds on a branch does it take
before you can merge something to main?
15:04.710 --> 15:10.914
And the really successful companies are
seeing the same sort
15:10.914 --> 15:13.935
of merge rate or higher as they get
better,
15:13.935 --> 15:17.887
right? And everything is increasing
because they're applying AI,
15:17.887 --> 15:20.249
they're creating more PRs, they're
creating them faster,
15:20.249 --> 15:22.240
they're creating even bigger ones,
15:22.240 --> 15:25.482
yet still faster and still merging faster.
15:25.482 --> 15:29.644
The sort of median company is creating a
lot.
15:29.796 --> 15:34.141
more builds on a branch because their work
is broken and it fails and
15:34.141 --> 15:38.185
it fails and it fails and it fails and
then they finally get it merged into main
15:38.465 --> 15:41.058
or whatever your default branch is that we
assume goes to production.
15:40.496 --> 15:41.496
Sure.
15:41.423 --> 15:42.423
Yep.
15:43.126 --> 15:46.697
And your final productivity, you're doing
a lot more work.
15:46.697 --> 15:49.618
You're using more tokens and more energy
and whatever.
15:49.618 --> 15:52.819
But then the final output is kind of the
same.
15:52.819 --> 15:53.819
And so that's
15:53.296 --> 15:54.296
Mm.
15:53.779 --> 15:56.950
a really interesting gap. We've been
trying to help people solve the for that
gap,
15:56.950 --> 15:59.991
right? Like, how do you get more
confidence earlier?
15:59.991 --> 16:03.332
How do you steer your agent? Assuming
you're using an agent to build
16:03.332 --> 16:05.883
or s some version of that, right?
16:05.883 --> 16:10.744
How do you steer your tools so that you're
getting good outcomes earlier?
16:11.248 --> 16:12.248
Interesting.
16:11.290 --> 16:12.290
With less
16:11.800 --> 16:13.461
effort, less token spend, et cetera,
16:13.461 --> 16:17.022
so that your your merge ratio goes back
up,
16:17.022 --> 16:21.663
right? It takes me one or two builds to
know the thing is good to then merge it,
16:21.663 --> 16:24.444
right? Versus like I'm creating all of
this,
16:24.444 --> 16:27.094
you know, it's like all heat, but it's all
friction.
16:27.094 --> 16:29.975
So I, you know, all I've or like I've
dissipated all of that energy.
16:29.975 --> 16:31.516
This is where my physics background,
16:30.404 --> 16:32.470
Yeah. Yeah.
16:31.516 --> 16:32.866
like I always like to use physics
analogies.
16:32.886 --> 16:35.567
But like we had all this energy,
16:35.567 --> 16:36.917
and yes, energy was conserved,
16:36.917 --> 16:39.488
but it's been dissipated into the
environment as heat instead
16:38.830 --> 16:39.830
Yeah.
16:39.488 --> 16:40.488
of.
16:40.091 --> 16:41.824
Light. Yeah.
16:40.347 --> 16:41.878
Turning into actual, like, yeah,
16:41.878 --> 16:43.599
light and output output to our customers.
16:43.599 --> 16:48.222
Right. So ~ it's super interesting to see.
16:48.222 --> 16:52.865
And I would say the spreads are getting
bigger,
16:52.865 --> 16:57.658
not smaller. Right. And so if you if you
imagine,
16:57.658 --> 17:02.041
and lots of people have used exp
expressions like AI as an amplifier,
17:02.041 --> 17:03.902
right? Like if you're if you
17:02.810 --> 17:03.832
Absolutely. Yeah.
17:03.902 --> 17:05.443
have brokenness in your organization,
17:05.443 --> 17:09.224
you just see them brokenness in more stark
contrast than you
17:08.900 --> 17:09.900
Yep.
17:09.224 --> 17:10.224
did before.
17:09.887 --> 17:12.421
If you made messes before, you're gonna
make them faster now.
17:10.604 --> 17:13.345
And right.
17:12.421 --> 17:13.421
Yep.
17:13.405 --> 17:15.675
Really big ones. And you're still not
going to get stuff out.
17:15.675 --> 17:17.646
So, like, if you were really good,
17:17.646 --> 17:21.067
right? If we're all moving along the track
and like some of us are slightly ahead
17:21.067 --> 17:25.468
or whatever, the really good folks are
going way faster.
17:25.468 --> 17:29.809
And the people who were struggling are
still struggling.
17:29.809 --> 17:35.301
So their their velocity hasn't really
changed if you're talking about like
actual
17:35.301 --> 17:38.442
impact to customers, right? Like we could
debate story points all day long.
17:38.442 --> 17:39.442
But like
17:39.386 --> 17:42.307
the actual impact to customers that you're
delivering isn't really shifting
17:42.307 --> 17:46.410
and the folks that really have it dialed
or realized how they needed
17:46.410 --> 17:48.751
to get things dialed really quickly in
order to be able to take advantage
17:48.751 --> 17:52.273
of the tools, those folks are breaking
away from the pack.
17:52.273 --> 17:55.034
So the spread is is even bigger.
17:55.480 --> 17:58.791
That's really interesting. I I wondered I
if you would say something like,
17:58.791 --> 18:01.172
people are like starting stuff and
starting stuff and starting stuff
18:01.172 --> 18:03.093
and it's never finishing. That that was my
own,
18:03.093 --> 18:04.284
especially early on,
18:03.864 --> 18:04.864
Mm.
18:04.284 --> 18:05.824
you know, experience was like,
18:05.824 --> 18:09.076
you're so good at greenfield, I'm just
gonna greenfield all the greenfield.
18:09.076 --> 18:10.076
And and it's like, wait, wait,
18:10.053 --> 18:11.126
wait, I'm not shipping anything.
18:11.126 --> 18:12.847
But but I
18:12.249 --> 18:13.249
Yeah.
18:12.887 --> 18:13.887
I love what you're talking about.
18:13.784 --> 18:19.990
So so to your point, the the better
organizations ~ sort of have less time
18:19.990 --> 18:23.051
on branch then and and fewer builds before
it actually gets merged
18:23.051 --> 18:24.652
and and actually gets shipped and
deployed.
18:25.486 --> 18:26.486
Yeah.
18:25.498 --> 18:26.498
and
18:25.769 --> 18:29.263
one assumes released versus ~ there's
there's this other sort
18:29.263 --> 18:32.137
of set of organizations which you know
that they're spending more time
18:32.137 --> 18:36.312
on that feature branch. ~ and so were they
doing that before
18:36.312 --> 18:39.656
and now they're spending even longer or
did the other ones get faster
18:39.656 --> 18:41.238
or both or help me with that part.
18:42.085 --> 18:46.387
Yeah, I think we've seen an increase in
the kind of
18:47.067 --> 18:50.128
not great branch build.
18:50.128 --> 18:53.647
Although there's a second like thinking of
my own experience,
18:53.647 --> 18:56.691
there's a secondary factor there that I
don't know if we've been able to isolate.
18:56.691 --> 19:00.132
So, but we're seeing more total builds,
19:00.132 --> 19:05.574
right? Take that more total builds in a in
a leading organization,
19:05.574 --> 19:08.526
and more of those will be on main trunk,
19:08.526 --> 19:09.646
whatever you happen to call that.
19:11.150 --> 19:14.641
take a struggling organization and it'll
be more just piling up on branches.
19:14.641 --> 19:16.612
Right. One of the interesting,
19:15.322 --> 19:16.322
Yeah, yeah.
19:16.612 --> 19:20.934
like, because I personally ~ am working
had been working on something,
19:20.934 --> 19:23.825
but I was like, you know, I'm I I'm in a
bit of a tough spot.
19:23.825 --> 19:26.687
I spend a lot of time in meetings and then
a bit like a big week might come
19:26.687 --> 19:27.787
up where I don't have a lot of time.
19:27.787 --> 19:29.838
And so maybe I'm not a great example,
19:29.838 --> 19:33.990
but I know this is real for folks where I
I get something to complete.
19:33.990 --> 19:36.081
And then the next day I'm like,
19:36.081 --> 19:39.312
yeah, maybe I'll maybe I'll actually get
it put up in a PR today or whatever.
19:39.684 --> 19:43.125
But let me just rebase because so much
activity has happened.
19:42.874 --> 19:43.874
Mm-hmm. Mm-hmm.
19:43.125 --> 19:44.125
And it's not just
19:43.936 --> 19:47.007
a git rebase anymore because you can do so
much work.
19:47.007 --> 19:48.007
It's like,
19:47.396 --> 19:48.396
Yeah.
19:47.888 --> 19:53.700
cool, take this intent and reapply it to
main so that it's not a massive diversion.
19:53.700 --> 19:56.841
I'm looking for, you know, I'm looking for
merge conflicts that aren't just this
19:53.924 --> 19:57.452
Right. Right. Yep.
19:56.841 --> 19:58.232
line was changed in two places,
19:57.452 --> 19:59.116
Right.
19:58.232 --> 19:59.252
but this
19:59.252 --> 20:00.453
no longer makes sense together,
20:00.453 --> 20:03.734
or another capability was implemented that
we were leveraging.
20:03.734 --> 20:05.355
So we have to adjust our implementation,
20:05.355 --> 20:07.406
right? And so sometimes my merge
20:07.576 --> 20:12.858
Or my my branch build count is high
because every day I just rebased
20:13.018 --> 20:15.950
in hopes that maybe that day I would get
the thing done and actually be able
20:15.950 --> 20:17.160
to put it up. Or maybe I put it up
20:16.836 --> 20:17.836
Right.
20:17.160 --> 20:19.821
in a PR and it sat there and a different
thing got merged.
20:19.821 --> 20:21.942
And so like that, that kind
20:21.594 --> 20:22.594
Yeah.
20:21.942 --> 20:25.664
of ~ like merge conflict.
20:24.080 --> 20:26.101
But when you rebase, you found more
changes
20:26.101 --> 20:28.432
than you would have a year ago.
20:28.432 --> 20:32.194
~ a year ago there would have been like a
couple of new files,
20:32.194 --> 20:34.315
but you know, there wouldn't have been a
lot of wholesale change,
20:34.315 --> 20:38.117
but now it's like nope, it moves I'm this
is the assumption I'm making based
20:37.903 --> 20:39.754
Yeah, yeah, exactly. Like
20:38.117 --> 20:39.608
on what you're saying. Yeah.
20:39.948 --> 20:40.948
Yeah.
20:40.995 --> 20:42.855
and this is on I won't call it greenfield,
20:42.855 --> 20:44.636
but something we're building that's pretty
new.
20:44.636 --> 20:45.777
And so
20:45.136 --> 20:46.136
Yeah.
20:46.077 --> 20:50.219
we might one morning, like every every
stand up every morning is like
20:50.219 --> 20:53.391
a pretty deep discussion about where's the
market and what do we want
20:53.391 --> 20:55.131
to do today and what did we learn from
yesterday.
20:55.131 --> 20:57.323
Not like still working on this ticket.
20:57.323 --> 20:59.354
Like no one's still working on that
ticket.
20:57.838 --> 21:00.438
Yeah. Right. Yeah.
20:59.354 --> 21:00.354
You know what I mean? It's like
21:01.208 --> 21:02.208
I mean, other than me,
21:01.898 --> 21:03.199
because I got distracted by meetings or
whatever.
21:03.199 --> 21:04.380
But like it's like you you know,
21:04.380 --> 21:05.619
those were stand-ups back in the day.
21:05.619 --> 21:07.141
It's like, well, it's day four.
21:06.842 --> 21:07.842
Yeah. Yeah.
21:07.141 --> 21:08.141
I'm still working on
21:07.782 --> 21:09.181
this ticket. Maybe I don't know what I'm
doing.
21:09.181 --> 21:10.473
Could someone help me? Right. Like,
21:10.262 --> 21:11.262
yeah yeah yeah.
21:10.873 --> 21:12.764
and now it's like, well, I did those seven
tickets,
21:12.764 --> 21:15.736
right? And like, you know, Tuesday we have
a chat,
21:15.736 --> 21:17.276
we agree on something, and by Wednesday,
21:17.276 --> 21:20.078
we fundamentally re-architected some part
of the system.
21:19.876 --> 21:20.876
Yeah, right.
21:20.078 --> 21:21.078
And
21:20.739 --> 21:22.360
then that thing that we were stringing
along,
21:22.360 --> 21:23.360
we're like, you know, you're like,
21:23.210 --> 21:26.362
which of these two major re-architectures
can we land first?
21:26.362 --> 21:28.983
And then we'll rebuild the other one on
top of that,
21:28.983 --> 21:29.983
right? And
21:30.436 --> 21:33.818
That would be so frustrating five years
ago.
21:34.053 --> 21:35.053
Yeah.
21:34.439 --> 21:36.788
And now I'm like, hey, buddy, because I,
21:36.788 --> 21:40.083
you know, I totally have turned my agents
into friends now.
21:40.083 --> 21:42.665
Hey, buddy, hate to tell you, but you
know,
21:42.665 --> 21:44.946
Maine has totally changed. Can you just
21:44.324 --> 21:45.324
Ha ha
21:44.946 --> 21:48.789
reapply this? And Buddy, who might be
codex or Claude or whatever that day,
21:48.789 --> 21:52.211
is like, no problem. That's you you pay me
to do that.
21:50.446 --> 21:53.708
Yeah, yeah. Ego less, yeah.
21:52.211 --> 21:53.552
So I'm happy to do it, right? I'm
21:53.793 --> 21:57.265
I'm serving the goal of whichever
foundation shop is getting all
21:57.265 --> 21:58.746
the token money right now. So
21:58.384 --> 21:59.384
Yeah, yeah, yeah.
21:59.322 --> 22:00.663
Yeah, I'll have at it. And so,
22:00.663 --> 22:05.115
like that you yes, exactly. You're making
like
22:05.355 --> 22:09.818
big thematic changes and then reapplying a
different goal on top
22:09.818 --> 22:14.160
of those thematic changes as opposed to
like the little angle brackets
22:14.160 --> 22:15.541
to the right and left that say,
22:15.541 --> 22:17.562
someone changed this function call,
22:17.562 --> 22:19.537
they added this parameter, and you added
this parameter.
22:18.970 --> 22:21.750
Right. Right. Yeah.
22:19.537 --> 22:21.344
You're like, cool, I'll put both
parameters
22:22.025 --> 22:25.056
and find the two call sites, and now I can
merge it,
22:23.886 --> 22:25.568
Right. Right.
22:25.056 --> 22:26.056
right? Because
22:25.597 --> 22:27.828
that's the amount that someone else could
have done.
22:28.143 --> 22:29.143
In that amount of time.
22:28.812 --> 22:29.812
in
22:28.972 --> 22:30.302
the period where I was, you know,
22:30.302 --> 22:31.604
where we were sort of conflicting.
22:31.604 --> 22:32.604
And so I I think
22:31.844 --> 22:32.844
That's interesting.
22:32.495 --> 22:38.580
that that that whole area of of merging
intent
22:37.969 --> 22:38.969
There's like a hygiene
22:38.829 --> 22:43.752
now around ~ you need to get it into Maine
faster almost,
22:43.752 --> 22:46.813
~ somehow. You so working even smaller,
22:46.813 --> 22:49.955
even more atomically i is the solution
that pops into my head.
22:49.955 --> 22:50.955
I wonder what you think though.
22:50.935 --> 22:51.935
Yeah.
22:51.854 --> 22:52.854
Yeah,
22:52.224 --> 22:55.886
I think I think the reason we're driven is
22:54.756 --> 22:55.798
Well, yeah.
22:55.886 --> 22:59.048
that we can drive bigger changes in a
small amount of time,
22:57.902 --> 22:59.715
Yeah, yeah, yeah, yeah, yeah, right,
22:59.048 --> 23:04.651
right? Like if I can write a thousand
useful lines of code,
22:59.715 --> 23:02.080
right. So that's not gonna happen.
23:02.080 --> 23:05.367
~ Right.
23:04.651 --> 23:07.773
like good, productive, useful lines of
code in an hour or two,
23:05.367 --> 23:08.002
Right. Right.
23:08.293 --> 23:10.594
do you want me to ship you 10 of those at
a time?
23:10.594 --> 23:13.006
Right? Like, hey Kyle, here's another PR.
23:13.006 --> 23:15.147
Like, what you know, here's and here's
one,
23:15.147 --> 23:17.488
and here's one, and here's what like that
wouldn't actually make sense,
23:17.488 --> 23:20.402
right? Like, yeah.
23:18.160 --> 23:20.461
Well, okay, I've had two guests say that
23:20.461 --> 23:25.783
they do that on purpose. One said that
he's basically always telling Claude for
him,
23:25.783 --> 23:31.265
~ that like we what we do in this is going
to be ten lines of code or fewer.
23:31.265 --> 23:33.780
That'll be like a constraint that he'll
give it.
23:33.780 --> 23:34.780
and
23:34.117 --> 23:35.117
Mm-hmm.
23:34.687 --> 23:38.198
and another guest ~ said for s for similar
reasons,
23:38.198 --> 23:39.509
but it's a different approach,
23:39.509 --> 23:44.091
was like, Okay, tell me what pull requests
we're going to do.
23:44.091 --> 23:46.752
So like really thinking it ahead of time
in terms of
23:46.414 --> 23:47.414
Yeah.
23:47.172 --> 23:50.503
We're not just gonna this isn't just gonna
be this giant ball that we're we're gonna
23:50.503 --> 23:53.246
we're gonna plan this out. ~ but I'm with
you.
23:53.246 --> 23:57.869
I think y one is sort of like trying to
avoid and pretend it's not happening
23:57.869 --> 23:59.880
and and the other is sort of like trying
to well,
23:59.880 --> 24:01.381
it is happening, but let's deal with it,
24:01.381 --> 24:02.762
you know, let's let's plan for this.
24:02.244 --> 24:03.244
Yeah.
24:03.242 --> 24:07.425
~ but you're right though. Like the the
idea that we're all gonna be as atomic
24:07.425 --> 24:10.247
as we used to be, yeah, that was a naive
point on my on my point,
24:10.247 --> 24:11.478
I think. Yeah. Sorry, go ahead.
24:12.612 --> 24:14.774
I mean, we're all figuring it out.
24:14.774 --> 24:15.774
I think a little bit
24:15.172 --> 24:16.172
Yeah.
24:15.534 --> 24:18.597
depends on your on your code base,
24:18.597 --> 24:20.008
the maturity of that code base,
24:20.008 --> 24:21.008
the maturity
24:20.302 --> 24:21.302
Yeah.
24:20.619 --> 24:23.151
of your business. Although I would argue
that like it's
24:23.151 --> 24:26.037
not just software development that's been
threatened by AI or changed by AI.
24:25.998 --> 24:28.021
Yeah. Yep.
24:26.037 --> 24:27.504
Like everyone's business model
24:28.565 --> 24:30.407
is immature all of a sudden, right?
24:30.407 --> 24:32.509
Because it's like, what is my business
model in this world?
24:32.509 --> 24:35.611
And so like rate of change,
24:35.611 --> 24:39.014
ad adaptability, all of those things are
more important.
24:39.014 --> 24:41.096
and
24:42.725 --> 24:46.017
That means, I mean,
24:46.017 --> 24:50.230
one of the things that that means to me is
we have to not just be throwing stuff
24:50.230 --> 24:52.952
out like here's an enormous PR and here's
an enormous PR and it seems to work.
24:52.952 --> 24:58.175
So let's just keep going. Because we do
have to be capable of changing and
adapting,
24:58.175 --> 25:02.778
right? Which is one of the reasons we put
a lot of energy into our design,
25:02.778 --> 25:04.089
into small incremental change,
25:04.089 --> 25:06.240
into making sure that other people
understood it.
25:06.240 --> 25:08.501
So then if you go kind of first
principles,
25:08.501 --> 25:10.853
like those were our goals, right?
25:10.853 --> 25:12.314
But a lot of those were aimed at.
25:12.644 --> 25:17.176
The human developer, right? Like a book
that I read three years ago,
25:17.176 --> 25:18.546
four years ago, I can't remember,
25:18.546 --> 25:20.277
called The Art of Readable Code.
25:20.277 --> 25:21.277
I think that's the name of it.
25:21.124 --> 25:22.124
Mm, mm.
25:22.048 --> 25:27.050
Right. Super interesting. I I was super
passionate about readable code
25:27.034 --> 25:28.897
Sure. Yeah.
25:27.270 --> 25:28.931
for all the reasons that I
25:28.931 --> 25:32.232
I need humans to be able to make safe
future changes.
25:32.232 --> 25:35.714
Right. But I think, I'm not sure yet,
25:35.714 --> 25:36.754
we're trying to figure it out.
25:36.754 --> 25:40.866
Readable to a human and readable to an
agent are different.
25:40.856 --> 25:41.856
gonna
25:40.997 --> 25:42.234
be different thing. Yep.
25:42.508 --> 25:45.730
And so the goal was safe,
25:45.730 --> 25:48.182
continued, safe future changes,
25:48.182 --> 25:50.013
therefore adaptability and velocity,
25:50.013 --> 25:55.137
right? And if I assume that my tool is me
prompting an agent,
25:55.137 --> 25:59.280
then the things that I looked at
originally when I saw agents write code
25:59.280 --> 26:02.262
or LLMs or whatever, we weren't really
like agentic or whatever yet,
26:02.262 --> 26:05.785
I was like, that's terrible. And I would
delete all their comments,
26:05.785 --> 26:08.086
you know, this doesn't look like the way
that I want it,
26:08.086 --> 26:09.287
whatever. And I'm like, but if that
26:08.216 --> 26:09.278
Yeah, yeah.
26:09.287 --> 26:10.287
makes the agent
26:10.482 --> 26:11.482
Right. Right.
26:10.948 --> 26:11.948
Work better
26:11.539 --> 26:15.893
on the next pass and is gonna in like lead
to higher chances of success.
26:15.893 --> 26:20.807
Why do I care? I'm not I'm not the primary
sort of like reader and writer
26:20.807 --> 26:22.279
of the of the code at this point.
26:22.279 --> 26:25.241
And I think we're gonna continue to to
shift and sort of like
26:25.762 --> 26:29.265
an emergent property that we will discover
of software
26:29.906 --> 26:36.412
is what is software that is highly
maintainable.
26:37.473 --> 26:39.054
If maintenance is the right tool,
26:39.054 --> 26:41.605
like with you know, can we componentize
and then just throw things
26:41.605 --> 26:42.626
out and replace them at you know,
26:42.626 --> 26:45.997
whatever that is? What does it look like
for long-term,
26:44.059 --> 26:47.328
Phoenix architecture right. Right.
26:45.997 --> 26:47.488
ongoing, sustainable,
26:47.488 --> 26:49.379
high-quality development driven by agents?
26:49.379 --> 26:50.379
And it'll
26:49.679 --> 26:50.679
Yeah.
26:50.050 --> 26:52.061
be everything from, yeah, like design,
26:52.061 --> 26:54.982
language selection, tooling choices,
26:54.982 --> 26:57.744
documentation, repo structure,
26:57.744 --> 26:59.985
like I'm sure we're gonna have mono repo,
26:58.350 --> 27:01.159
Yeah. Yeah, yeah, yeah, yeah, yeah,
26:59.985 --> 27:01.776
multi-repo debates again. You know,
27:01.159 --> 27:02.242
yeah. Yeah.
27:01.776 --> 27:02.776
like
27:02.886 --> 27:05.938
how do we enable these tools? And I think
we're we're still discovering that
27:05.938 --> 27:06.938
and we'll
27:06.500 --> 27:09.971
We'll see based on who's most successful,
27:09.971 --> 27:11.256
you know.
27:11.100 --> 27:12.100
Okay,
27:11.490 --> 27:14.542
so Roll, I I don't know that you'll have
an answer to this,
27:14.542 --> 27:15.923
but it occurs to me that you might.
27:15.923 --> 27:20.925
~ can you do you have any sort of like
economic indicators around like growth
27:20.925 --> 27:22.606
of hiring or shrinking of hiring?
27:22.606 --> 27:25.478
My my own anecdot, which, you know,
27:25.478 --> 27:26.478
is is
27:25.488 --> 27:26.488
Mm-hmm.
27:26.128 --> 27:28.980
never much to go on, is that most of my
guests have reported,
27:28.980 --> 27:31.991
like, I we actually haven't hired anybody
since AI,
27:31.991 --> 27:35.183
but nobody has talked too much about
having laid anybody off.
27:35.183 --> 27:37.034
Now, of course, there are high profile,
27:37.034 --> 27:38.615
you know, CEOs of Bangor Corps,
27:38.615 --> 27:40.316
and you can kind of assume
27:40.526 --> 27:41.526
l
27:40.806 --> 27:42.418
that they would have lots of reasons to do
that.
27:42.418 --> 27:46.962
~ but you have non anek data. You you have
actual data,
27:46.962 --> 27:49.114
maybe? Like can you see much there?
27:48.130 --> 27:51.322
Yeah. ~
27:51.743 --> 27:57.446
I would say what we look at is I well,
27:57.446 --> 28:02.330
I mean, we we have we have both and we
don't always know like when we look
28:02.330 --> 28:03.741
at our data, we have to do some like,
28:03.741 --> 28:06.313
okay, but who's that company and what do
we know about them sort of thing,
28:06.313 --> 28:11.566
right? Like so you have organizations that
are raising enormous amounts
28:11.566 --> 28:16.400
of capital and you know, growing on like
very
28:16.963 --> 28:19.314
investment driven growth and they are
hiring.
28:19.314 --> 28:24.198
And then we see a lot of our base staying
the course,
28:24.198 --> 28:26.879
but growing their delivery velocity,
28:26.879 --> 28:30.317
right? Like they're again without judging
whether they're getting customer outcomes,
28:30.317 --> 28:31.753
we're seeing more builds, more throughput,
28:31.753 --> 28:32.753
more creation.
28:32.208 --> 28:33.208
Yep. Yep.
28:33.183 --> 28:39.608
And and so a lot of what we think about is
in the way that you calculate
28:39.608 --> 28:41.699
net dollar retention, I think is mostly an
engineering
28:41.977 --> 28:43.897
like audience, but so maybe I'll explain
it for a second.
28:43.897 --> 28:47.341
But like you take a cohort and say from
that cohort in the future,
28:47.341 --> 28:51.865
what is our revenue based on the s the
people that were here at this time
28:51.865 --> 28:53.686
a year ago, right? As like net dollar
retention.
28:52.922 --> 28:53.922
Uh-huh.
28:53.686 --> 28:59.551
~ and so you can think of like net
pipeline retention as a number almost like
28:59.551 --> 29:04.905
are we growing from the same number of
like projects and the same number
29:04.656 --> 29:07.097
Mm. Mm.
29:04.905 --> 29:06.996
of users sort of thing. Like so we can see
29:07.317 --> 29:09.518
some health in our business independent of
the other things we're doing.
29:09.518 --> 29:11.257
Like we're signing new customers and all
that kind of stuff,
29:11.257 --> 29:12.257
right?
29:12.098 --> 29:15.112
And we definitely see growth within a
cohort,
29:15.112 --> 29:17.975
despite not a ton of growth. I mean,
29:17.975 --> 29:19.247
again, it's different company by company,
29:19.247 --> 29:22.060
but like a lot of organizations are pretty
stable.
29:20.644 --> 29:22.677
Right. Right.
29:22.060 --> 29:23.060
Although
29:23.002 --> 29:25.064
we we had a really interesting as you
know,
29:25.064 --> 29:26.656
you can only trust your data so much.
29:26.656 --> 29:28.428
What we see is who's committing,
29:28.428 --> 29:31.108
right? So we see more.
29:33.444 --> 29:35.164
The GitHub IDs, let's call it,
29:35.164 --> 29:37.545
associated with a project over a period.
29:35.182 --> 29:38.300
Right, right. Mm-hmm.
29:37.545 --> 29:38.906
Right. And so we're like, cool,
29:38.300 --> 29:40.898
More humans, yeah.
29:38.906 --> 29:40.686
they hired engineers to do this.
29:41.146 --> 29:44.387
And I was in a conversation recently where
we said,
29:44.387 --> 29:46.308
~ you know, we were talking to a customer.
29:46.308 --> 29:47.888
We we see you've done a lot of hiring.
29:47.888 --> 29:49.149
Like, are you are you doing a lot of
hiring?
29:49.149 --> 29:50.829
And they were like, No, we haven't.
29:50.829 --> 29:53.830
And we're like, that's odd. Like,
29:52.058 --> 29:55.318
Hmm. Your count of GitHubes went up?
29:53.830 --> 29:55.651
and it what what we learned
29:56.431 --> 30:00.182
was everyone in the organization is now
committing.
30:01.284 --> 30:03.655
So they're designers and their PMs and
everybody.
30:03.655 --> 30:05.645
So they hadn't hired net new people,
30:05.645 --> 30:10.817
but they had fundamentally changed the way
they work such that to us,
30:10.817 --> 30:12.598
we're like, there's net new contributors,
30:12.598 --> 30:16.949
and our simple brains are like there's
more engineers in the organization.
30:16.949 --> 30:18.220
But what they explained
30:17.487 --> 30:18.487
No, it's not
30:18.220 --> 30:20.210
to us was actually happening, they had
hired some people,
30:20.210 --> 30:23.752
and then they had more people contributing
to the project,
30:23.376 --> 30:24.708
That's amazing. And you hear about
30:23.752 --> 30:24.752
right?
30:24.708 --> 30:25.960
that, you know, and you wonder,
30:25.960 --> 30:29.596
well, you know, the anthropic product
manager goes on a podcast
30:29.596 --> 30:30.908
and says that's happening and you're like,
30:30.908 --> 30:33.442
cool, but but but you're to your point,
30:33.442 --> 30:35.655
you're seeing actual proof of that beyond
just,
30:35.257 --> 30:36.257
Yeah.
30:35.896 --> 30:36.896
yeah.
30:36.398 --> 30:41.269
Right. And it's not like necessarily the
designer showed up and launched like
30:41.269 --> 30:44.541
50 PRs that week right next to the
engineer who put up 50 PRs
30:44.541 --> 30:45.961
or whatever the numbers are like these
days.
30:45.961 --> 30:49.710
But you know, we say, we we see people and
of course,
30:49.710 --> 30:52.024
you know, if you model it, you'd this is
30:51.512 --> 30:52.512
Interesting.
30:52.024 --> 30:53.694
a new engineer, therefore they've only,
30:53.694 --> 30:55.865
you know, they made a small amount of
contribution or whatever.
30:55.865 --> 30:59.557
But it's like at some point, we probably
could have done this a while ago.
30:59.557 --> 31:02.708
The designer is like, I mean, I don't know
if you remember the days,
31:02.708 --> 31:03.708
probably.
31:03.830 --> 31:07.221
they would like make a new Photoshop
image,
31:07.221 --> 31:09.181
right? With where the color and they want
you to
31:08.826 --> 31:09.826
Mm-hmm.
31:09.181 --> 31:11.942
like get the eyedropper and sample the
color and then change the color
31:11.492 --> 31:15.594
Yeah, yeah, yeah, yeah. Yeah. Yeah.
31:11.942 --> 31:15.023
on the button. I'm like, could you please
just edit the CSS?
31:15.023 --> 31:16.023
Like why why are
31:15.703 --> 31:18.424
we doing this? Right? Like this whole
process is ridiculous.
31:18.192 --> 31:22.505
Stop making me slice up PSDs. It is not
ninety-eight anymore.
31:18.424 --> 31:21.485
Right. my God. Yeah, exactly.
31:21.485 --> 31:24.726
Exactly. Right. I mean like slice
31:24.726 --> 31:27.206
it up into little images and try to place
them on the screen and whatever.
31:26.969 --> 31:27.969
Yeah.
31:27.206 --> 31:28.206
And
31:27.446 --> 31:31.228
and so, you know, we we absolutely have
that internally as well.
31:31.228 --> 31:32.888
Just like, just do it.
31:32.452 --> 31:33.452
Yeah.
31:33.230 --> 31:34.600
Right. And and
31:33.321 --> 31:34.551
Cool. Cool.
31:35.791 --> 31:36.841
yeah, I mean, we have the tools,
31:36.841 --> 31:38.552
we have the capacity. So we're we're
31:38.096 --> 31:39.096
Amazing.
31:38.552 --> 31:39.942
I mean, that's a little off of the hiring.
31:39.942 --> 31:42.743
We are seeing what you see on the news,
31:42.743 --> 31:46.105
you know, huge organizations thinning
31:43.236 --> 31:45.749
Yeah, yeah. Yeah.
31:46.105 --> 31:49.606
out, sometimes because they truly believe
AI is shifting things,
31:49.606 --> 31:52.517
sometimes because they're trying to come
up with a reason to justify what they're
31:52.517 --> 31:54.808
doing. And then we're seeing, you know,
31:53.304 --> 31:55.637
Zero interest rates over sure.
31:54.808 --> 31:55.808
yeah.
31:55.637 --> 31:57.310
I mean, yeah, all sorts of reasons.
31:57.310 --> 31:58.552
Yep. Interesting.
31:57.946 --> 31:58.946
company's taking
31:58.701 --> 32:00.011
off and and still hiring you.
32:00.688 --> 32:04.139
Cool. So okay, we ~ that that was really
fun.
32:04.139 --> 32:09.110
~ normally what we shift to ~ right after
like what tools you reach
32:09.110 --> 32:11.241
for is what you're seeing in teams that
you observe.
32:11.241 --> 32:13.612
And we've kinda had that nicely.
32:11.908 --> 32:12.908
Mm-hmm.
32:13.612 --> 32:14.612
You you've got all
32:14.282 --> 32:16.032
sorts of stories about sort of like all of
our teams.
32:16.032 --> 32:19.893
~ but let's let's zoom in a little bit and
to stu and into your
32:19.893 --> 32:21.284
own observations a little bit more,
32:21.284 --> 32:23.914
direct, ~ and less less just out of the
data.
32:23.914 --> 32:28.306
Wha how are you what are you seeing in
terms of how this well actually honestly
32:28.306 --> 32:29.626
you can answer from either position.
32:29.957 --> 32:30.957
But
32:30.167 --> 32:32.249
we'll we'll go through sort of S DLC left
or right.
32:32.249 --> 32:35.742
So how how what do you think you're seeing
change in terms of how
32:35.742 --> 32:37.783
we think about requirements? You know,
32:37.783 --> 32:40.671
in in the old days it might have been
epics and stories.
32:40.671 --> 32:43.108
~ for some organizations it might have
been B R D's,
32:43.108 --> 32:47.962
PRDs, whatever. are we just having the
agents write them now and
32:47.962 --> 32:51.345
the agents read them? Like what what what
are you seeing change in how
32:51.345 --> 32:52.866
all of that works here? Yeah.
32:53.540 --> 32:55.087
Yeah. So
32:57.464 --> 33:02.086
See the whole spectrum, ~ e even
internally,
33:02.086 --> 33:04.428
right? Even internally at Circle CI,
33:04.428 --> 33:06.809
I think the jury is very much still out.
33:06.809 --> 33:10.051
And part of that might be, you know,
33:10.051 --> 33:12.412
your background and the metaphor that you
cling to,
33:12.412 --> 33:15.374
and part of it might be like your
particular form of success that you've
33:15.374 --> 33:16.774
had with your tools that you're trying,
33:16.774 --> 33:22.277
right? And and we see,
33:22.277 --> 33:25.459
you know, like spec driven development,
33:25.459 --> 33:26.780
that whole sort of
33:27.674 --> 33:30.526
~ umbrella term, right, which means
different things to different people.
33:30.526 --> 33:33.568
And to your point, like sitting down and
hammering something out,
33:33.568 --> 33:36.991
maybe with with Claude or whatever you
happen to use,
33:36.991 --> 33:40.553
right? ~ can be really effective as a
thought partner.
33:40.553 --> 33:41.553
Like I think just saying, hey,
33:41.444 --> 33:43.255
I want to build a thing, can you write me
a spec?
33:43.255 --> 33:45.136
I might as well just say, hey,
33:45.136 --> 33:46.757
build me the thing. Right.
33:46.084 --> 33:47.084
Yeah, yep.
33:47.018 --> 33:48.759
But if I say, hey, okay, cool,
33:48.759 --> 33:49.880
let's go look at this company,
33:49.880 --> 33:51.031
I think they're very successful.
33:51.031 --> 33:52.031
They're one of our competitors.
33:52.021 --> 33:54.183
Like what's different about what they're
doing from what we're doing,
33:54.183 --> 33:55.874
right? And what
33:56.036 --> 33:57.557
What is no one considered, right?
33:57.557 --> 34:02.179
Like I I find these tools are really
helpful for giving me like my blind spots
34:02.179 --> 34:03.779
or like things I haven't thought of or
whatever.
34:03.779 --> 34:08.281
But I don't tend to find that sitting down
and like writing a long spec
34:08.281 --> 34:10.839
and trying to line it up and then we're
gonna do this and then we're gonna
34:10.839 --> 34:11.839
do this and then we're gonna do this.
34:11.822 --> 34:14.244
I mean, I'm like trying to waterfall my
agent at that point,
34:14.244 --> 34:15.244
right? That
34:14.543 --> 34:15.543
Right, right.
34:15.114 --> 34:17.804
doesn't that doesn't really work well for
me.
34:17.804 --> 34:23.547
I find that I can open up Claude Code or
Codex or like a coding agent,
34:23.547 --> 34:24.547
which
34:25.339 --> 34:27.909
Pretty much the chat bot plus writing code
better.
34:27.909 --> 34:31.132
And by the time I've figured out what I
want to build,
34:31.132 --> 34:33.413
I've built it. Right. I was like,
34:33.413 --> 34:36.295
if we should do that, then why don't you
go ahead and do that and I'll fire
34:36.295 --> 34:38.505
up another subagent or whatever and we'll
have a chat about
34:36.954 --> 34:38.357
Yeah. Mm-hmm.
34:38.505 --> 34:39.617
something else. It's like, cool,
34:39.617 --> 34:40.888
why don't okay, why don't you go do that?
34:40.888 --> 34:43.239
let's fix it like this. And so for me,
34:43.239 --> 34:46.501
like I'm often reasoning about the thing.
34:46.501 --> 34:48.141
If it's new, like if it's pretty novel,
34:48.141 --> 34:50.342
I'm reasoning
34:48.570 --> 34:49.748
Yep. Right.
34:50.342 --> 34:52.364
about it as I'm building it.
34:52.792 --> 34:55.263
Instead of reasoning about the document
that I'm going to write about
34:55.263 --> 34:57.623
how to build it, which then I'm going to
feed to some other agents,
34:57.623 --> 34:58.775
and then I'm going to come back and say,
34:58.775 --> 35:00.335
~ you all did it wrong, whatever.
35:00.335 --> 35:04.267
And so I I see people having success on
both ends.
35:04.267 --> 35:09.819
I lean towards the like, I think my whole
career I've thought about software
35:09.819 --> 35:13.031
by opening Vim. I know everyone's going to
yell at me and say that
35:13.031 --> 35:16.352
I should use Emacs or like get into the
2000s and get an IDE or whatever,
35:16.352 --> 35:19.684
but like opening and reading the code and
being like,
35:19.684 --> 35:20.684
okay, how does this work?
35:21.614 --> 35:24.165
What if I did it like this? Like make
little experiments and stuff.
35:24.165 --> 35:26.246
It's just the way that my my brain works.
35:26.246 --> 35:31.508
And so for me, I find there's less
ceremony up front,
35:31.508 --> 35:35.910
more like, and we ~ we're a a linear shop.
35:35.910 --> 35:38.151
So we, you know, we use linear tickets and
they're like
35:36.858 --> 35:38.295
Mm-hmm. Mm-hmm.
35:39.001 --> 35:41.552
half a sentence. We got distracted while
35:40.590 --> 35:41.590
Right. Right.
35:41.552 --> 35:42.783
we were writing the ticket, but someone's
like,
35:42.783 --> 35:44.013
Yeah, I think I was working on something
like that,
35:44.013 --> 35:47.645
you know? And and so because our space is
moving also,
35:47.645 --> 35:50.646
like in the and the part that I'm
particularly working on with the team that
35:50.646 --> 35:51.646
I'm working with.
35:51.644 --> 35:57.426
~ it's moving so quickly that like let's
lay out this plan for the next month
35:57.426 --> 35:59.367
or whatever. Like it just doesn't make a
lot of sense.
35:59.367 --> 36:02.729
It's very, you know, we're thinking it
through and we're trying things and
whatever.
36:02.729 --> 36:08.471
~ and so I I find that the spec is the
functioning code,
36:08.471 --> 36:10.072
you know, like and and
36:09.829 --> 36:10.829
Yeah, yeah.
36:10.642 --> 36:12.743
I find myself quoting the Agile Manifesto
a lot.
36:12.743 --> 36:16.064
Like back in 2000 or 2001, I think it was
2001,
36:16.064 --> 36:18.035
right? When that got written, you know,
36:18.035 --> 36:19.035
there's a statement,
36:20.613 --> 36:23.255
I I think it's functional software over
documents or something like that,
36:22.916 --> 36:26.059
Yeah. Mm-hmm. Mm-hmm. Working software
over documentation.
36:23.255 --> 36:26.667
right? And right,
36:26.059 --> 36:27.059
Yeah.
36:26.667 --> 36:28.848
right. Read it more recently than I have,
36:28.848 --> 36:33.001
or just have a better memory. And so that
like we can do that now.
36:33.001 --> 36:35.122
Right? We could do that in minutes.
36:34.458 --> 36:37.804
Yeah. Yeah, yeah. Yeah. Yep.
36:35.122 --> 36:36.323
Like, you want it to work like this here?
36:36.323 --> 36:37.384
Let me just try that real quick.
36:37.384 --> 36:38.384
And okay,
36:37.984 --> 36:40.376
let's try it. that's weird. It does have
this weird behavior.
36:40.376 --> 36:41.546
Well, we hadn't considered that.
36:41.546 --> 36:44.388
And to me, like that evolutionary,
36:44.388 --> 36:46.400
like that process of evolving software is
yeah,
36:44.941 --> 36:47.362
Immediacy is really addictive.
36:46.400 --> 36:49.952
is really right. But then go
36:47.362 --> 36:49.823
Yeah. Yeah. I I I work like you,
36:49.952 --> 36:50.952
ahead.
36:50.083 --> 36:52.844
for sure. I can relate to to what you're
talking about.
36:52.844 --> 36:56.616
Yeah. All the all the sort of spectrum
adherents are trying to convince
36:56.616 --> 36:58.147
me to to give up on my immediacy,
36:58.147 --> 37:00.047
but ~ but but that's a whole other show.
36:59.376 --> 37:01.758
Yeah, I do want
37:00.047 --> 37:01.328
Yeah. Okay.
37:02.018 --> 37:04.059
some things that will take a long time.
37:04.059 --> 37:05.059
Cause
37:04.549 --> 37:05.549
Yeah.
37:04.930 --> 37:07.462
I have this weird, like someone showed me
a chart of this that
37:07.462 --> 37:09.163
was that really resonated. But basically,
37:09.163 --> 37:13.686
like the feedback isn't immediate because
the agent goes and does something.
37:13.686 --> 37:15.247
But it's
37:14.062 --> 37:15.107
Yeah. Yeah.
37:15.247 --> 37:19.029
not far enough off that I can really get
engaged with something else,
37:19.029 --> 37:20.480
right? So I'm trying to find,
37:19.246 --> 37:20.331
Right, right, right, right.
37:20.480 --> 37:23.973
okay, can I give you a little and I have
built harnesses that
37:22.456 --> 37:23.819
Yeah. Right.
37:23.973 --> 37:27.775
are like goal seeking, like you'll know
you're done when you achieve this.
37:27.266 --> 37:28.397
Okay, yep. Mm-hmm.
37:27.775 --> 37:28.775
That
37:28.526 --> 37:32.978
That to me is I mean, it's different from
the like I write a like a markdown spec,
37:32.978 --> 37:33.978
but
37:33.306 --> 37:34.306
Mm-hmm.
37:33.669 --> 37:35.800
I will have an agent build up a harness,
37:35.800 --> 37:39.482
right? Which might just be my test suite
in a for loop,
37:39.482 --> 37:41.813
right? But like it's a little test driven
development.
37:41.813 --> 37:44.096
I think we started calling it harness
engineering because we didn't want
37:44.096 --> 37:45.645
to have any more debates about test driven
development.
37:45.645 --> 37:48.787
But like saying, Okay, like this is what
complete would look like,
37:48.787 --> 37:53.259
right? When you meet these w when this all
passes or this all works properly
37:53.259 --> 37:55.220
or whatever. And certainly I'll say things
like,
37:55.220 --> 37:56.501
Okay, what else haven't we thought of?
37:56.501 --> 37:57.892
What would break? And then
37:58.128 --> 37:59.628
Claude will this would break and this
would break.
37:59.628 --> 38:01.631
I'm like, Could you write tests that would
prove that that doesn't work?
38:01.631 --> 38:04.334
Absolutely. Great. Now can you make them
work?
38:03.246 --> 38:07.212
Yeah. Yeah. Yeah.
38:04.334 --> 38:06.586
Like that that cycle gets me longer
windows.
38:06.586 --> 38:07.586
And when I have longer
38:07.346 --> 38:10.009
windows, I open up another terminal and
start another agent kind of thing,
38:09.496 --> 38:11.207
And start another one. Yep. 100%.
38:10.009 --> 38:11.500
right? And it's like, Yeah. Yeah.
38:11.207 --> 38:12.207
Yep. Yep.
38:13.018 --> 38:18.440
Cool. Cool. Thank you. ~ the actual so
you've talked in there about
38:18.440 --> 38:20.732
we specification and thinking about
planning.
38:20.732 --> 38:23.433
And we talked about a little bit into the
actual development.
38:23.433 --> 38:24.804
You even touched on testing as well.
38:24.804 --> 38:28.245
~ any other changes in either I mean,
38:28.245 --> 38:31.447
obviously the the development itself is
the thing which has changed the most.
38:31.447 --> 38:32.447
So anything you
38:32.015 --> 38:33.015
Yeah.
38:32.017 --> 38:33.017
wanna say
38:32.337 --> 38:33.648
about that that isn't obvious.
38:34.229 --> 38:35.954
~ or or anything on the testing side.
38:35.954 --> 38:38.593
Like what what are you seeing there that
that as far
38:38.255 --> 38:39.255
Yeah.
38:38.593 --> 38:40.098
as change to our industry? Yeah.
38:40.815 --> 38:42.966
Yeah, so so probably two things.
38:42.966 --> 38:44.427
I would say on the development,
38:44.427 --> 38:49.460
like I just described designing and
interacting with the customer problem
38:49.460 --> 38:51.681
as the process of building. Like why I
might as well be built,
38:51.681 --> 38:53.632
right? By the time I figured it out.
38:53.632 --> 38:55.614
What's interesting to me about that,
38:55.614 --> 38:57.114
and I said this a little bit earlier,
38:57.114 --> 38:59.776
is our stand-ups with my team,
38:59.776 --> 39:03.118
which ranges from some other very senior
engineers down to like new grads.
39:03.118 --> 39:06.300
We talk every day about
39:06.862 --> 39:08.934
The business, the state of the business,
39:08.934 --> 39:11.797
the state of the market, what matters to
our customer.
39:11.797 --> 39:15.380
So that when those folks leave and spend a
day working on something,
39:15.380 --> 39:17.202
they can make those decisions as well.
39:17.202 --> 39:22.346
Right. So we're sort of like, and it's
probably a little weird because you're
39:22.346 --> 39:24.749
sitting with the CTO who like knows a lot
about the business or whatever,
39:24.749 --> 39:28.352
but but we're distributing the product
thinking.
39:28.352 --> 39:29.523
And we have a product
39:28.686 --> 39:29.686
Yeah, yeah.
39:29.523 --> 39:31.114
manager who's helping us as well,
39:31.114 --> 39:32.696
does a lot of the like,
39:33.454 --> 39:35.475
I went and looked at the metrics and it
looks like this is happening
39:35.475 --> 39:37.187
and whatever and brings that to our
discussions.
39:37.187 --> 39:41.161
but it's not like a product manager
writing tickets and handing them out to
people.
39:41.161 --> 39:42.492
Like if you move to this model,
39:42.492 --> 39:45.074
which is like we just we build and we
discover as we go,
39:45.074 --> 39:49.837
the engineers need to really understand
the problem we're trying to solve.
39:49.837 --> 39:51.309
We happen to build developer tools,
39:51.309 --> 39:53.821
which probably makes it a little easier to
have engineers that understand that.
39:53.584 --> 39:54.584
Right. Right.
39:53.821 --> 39:54.842
But junior
39:54.842 --> 39:59.066
engineers don't necessarily understand the
problems of like large engineering
39:57.914 --> 40:00.299
Yeah. That's right.
39:59.066 --> 40:00.487
systems, right? So we really have
40:00.487 --> 40:01.728
to bring those folks along.
40:02.364 --> 40:04.315
~ and I think it's I think it's a
worthwhile investment.
40:04.315 --> 40:06.786
I think that's how you build leaders in
your organization anyway.
40:06.786 --> 40:12.470
But I think we've had a split over the
last I always quote 25 years because it's
40:12.470 --> 40:13.987
kind of like I hell, I guess I've doing it
longer than that.
40:13.987 --> 40:19.163
my god. Anyway, ~ over that time into like
really product thinking engineers
40:19.163 --> 40:21.895
and then very sort of like ticket taking
engineers,
40:21.895 --> 40:22.895
right? And I think the
40:22.201 --> 40:23.201
Is it?
40:22.595 --> 40:25.497
ticket taking is is gonna be hard to
figure out in this world.
40:25.497 --> 40:27.978
So I think how we think about what we're
gonna build,
40:27.978 --> 40:30.810
it's not just that we don't necessarily
write out a long spec.
40:31.353 --> 40:35.614
It's that that requires an investment in
the people participating,
40:35.614 --> 40:37.955
that they really do have a good
40:37.092 --> 40:38.092
Yeah. Yep.
40:37.955 --> 40:40.566
understanding and are comfortable just
saying,
40:40.566 --> 40:42.436
like, hey, yo, I actually don't
understand.
40:42.436 --> 40:45.117
Like I did this and it doesn't seem like
it's working,
40:45.117 --> 40:47.367
or I went and talked to this other
engineer and they don't understand what
40:47.367 --> 40:48.828
I'm trying to tell them. I must be missing
something,
40:48.828 --> 40:51.418
like just creating that sort of that that
free flow.
40:51.418 --> 40:55.440
So then on the on the testing side,
40:55.440 --> 40:56.820
like, and we talked about this.
40:57.592 --> 41:00.723
you know, merge efficiency, like are my am
I getting all these failed builds
41:00.723 --> 41:04.425
on my branch, et cetera. ~ it is actually
part of what I've been working
41:04.425 --> 41:07.496
on with this team is like trying to shift
that earlier,
41:07.496 --> 41:14.239
meaning putting sort of what I would call
CI level validation into
41:14.239 --> 41:17.190
the hands of the agent. So as the agent is
working,
41:17.190 --> 41:18.951
we're saying, your test suite would pass,
41:18.951 --> 41:21.292
your test suite would pass. Actually your
test suite failed right now.
41:21.292 --> 41:22.822
Like here's some feedback about that.
41:22.822 --> 41:27.094
And then trying to build some tools around
that that increase the or val like
41:27.468 --> 41:30.550
check and then can help improve the
quality of the testing.
41:30.550 --> 41:32.912
Because I think one of the fears that we
have
41:31.693 --> 41:32.971
Mm-hmm. Yep.
41:33.372 --> 41:36.124
is if my agent writes the code and my
agent writes the test,
41:36.124 --> 41:37.986
like we've all been in this scenario where
the agent's like,
41:37.986 --> 41:40.157
well, this test passed, so I or this test
failed,
41:40.157 --> 41:41.458
so I deleted it. And you're like,
41:41.458 --> 41:43.600
that does get us to a green build,
41:43.600 --> 41:45.631
but it's not necessarily the green build
we wanted,
41:45.631 --> 41:48.163
right? Like well, we we needed to know
that that was failing.
41:48.163 --> 41:50.325
Let's fix the thing. Also,
41:49.828 --> 41:50.828
Yeah.
41:50.668 --> 41:54.458
you know, I for there's this like model
from the
41:54.896 --> 41:56.387
I think it was originally created in the
70s,
41:56.387 --> 41:58.919
like all cool things in software called
mutation testing,
41:58.919 --> 42:04.624
where you basically manipulate the
implementation and then check to
42:01.552 --> 42:06.819
Amen. Yep. Mm-hmm. I'll make sure it
actually fails.
42:04.624 --> 42:06.016
see whether your test would catch it,
42:06.016 --> 42:07.777
right? And it's
42:06.819 --> 42:07.819
Yeah. Yep.
42:07.777 --> 42:10.049
right. And it's like super expensive.
42:10.049 --> 42:12.822
And probably like these things were
invented in the 70s and nobody
42:12.822 --> 42:14.833
did them because it's like so much compute
power.
42:14.833 --> 42:19.087
But like we all have access to ridiculous
amounts of compute right now,
42:17.902 --> 42:20.740
Well, in the seventies we were probably
doing it by hand.
42:19.087 --> 42:22.276
right? So yeah, exactly.
42:20.740 --> 42:22.004
We're probably going in a l
42:22.285 --> 42:24.631
chain the
42:24.185 --> 42:25.185
What if I
42:24.665 --> 42:27.986
put some paper over this hole in the punch
card and I like punch
42:27.322 --> 42:28.322
Yeah, yeah.
42:27.986 --> 42:31.107
out this hole instead? So, so ~ yeah,
42:31.107 --> 42:33.107
we're, you know, we built some tooling
around that.
42:33.107 --> 42:38.168
and and trying to push all that to the
left and think about what else we could do
42:36.366 --> 42:38.165
Yeah. Yep.
42:38.168 --> 42:42.449
to improve the quality so that we have
really high confidence and we
42:42.449 --> 42:44.590
can get that confidence as early as
possible.
42:44.590 --> 42:46.850
So, so really like on the testing front,
42:46.850 --> 42:51.631
trying to create, again, improve the
quality of the testing and ensure that
42:51.631 --> 42:53.685
by the time we get to
42:54.190 --> 42:56.852
that place we're gonna run into the merge
conflicts and wait for the build time
42:56.852 --> 43:00.044
and all that stuff, that it's in really
good shape and we have really high
43:00.044 --> 43:03.336
confidence. And now we're just worried
about the merge as opposed
43:03.336 --> 43:06.118
to worried about like, does my thing even
pass the local test suite?
43:06.118 --> 43:09.911
Right. And I think for people with large
monoliths,
43:09.911 --> 43:11.312
which I think is a lot of people,
43:11.312 --> 43:13.493
right? Even people who have microservices
architectures,
43:13.493 --> 43:16.155
90% of their code is still in a giant
monolith somewhere.
43:16.155 --> 43:18.516
Those things are really hard to test,
43:18.516 --> 43:20.758
right? And they're really hard to test
locally usually.
43:20.758 --> 43:23.810
and so how do you kind of like
43:24.256 --> 43:26.617
We've built a bunch of different tools to
optimize testing.
43:26.617 --> 43:29.528
So, like the I guess the other thing when
we talk about testing that we're seeing
43:29.528 --> 43:31.539
we're feeling it, our customers are
feeling it,
43:31.539 --> 43:35.241
is like some fundamental queuing theory,
43:35.241 --> 43:41.124
right? Like your arrival rate of change is
escalating to the point where
43:41.504 --> 43:44.996
the the time spent executing or validating
that change,
43:44.996 --> 43:46.927
particularly if you have to serialize
that,
43:46.927 --> 43:48.647
right, as you get something out to deploy.
43:50.436 --> 43:52.237
You're gonna reach a point where the
arrival,
43:52.237 --> 43:58.161
like it's impossible to service the change
faster than the arrival rate.
43:58.161 --> 44:02.024
And as a result, you're going to
infinitely cue.
44:02.024 --> 44:07.008
Right. So then you have a real like you
have a real fundamental problem
44:07.008 --> 44:10.370
of the design of your system. And so then
it's like,
44:10.370 --> 44:11.911
okay, how fast can I get that?
44:11.911 --> 44:14.953
Like, how tight can I make that little
serialization point?
44:14.953 --> 44:18.906
Right. And how can I minimize the risk?
44:19.459 --> 44:21.400
Of us of something breaking at that point,
44:21.400 --> 44:23.722
right? Like, and there's there's
techniques that we have already
44:23.722 --> 44:27.534
and then other techniques that we're
introducing to try to to try to make that
work.
44:27.534 --> 44:30.036
And then that's where when I'm when we
talked about architecture,
44:30.036 --> 44:33.318
like yes, architecture for agents for them
to be able to reason about a system
44:33.318 --> 44:37.861
is is important. But then architecture of
the system in terms of how do
44:37.861 --> 44:42.204
I get that throughput, right, at some cost
optimized approach,
44:42.204 --> 44:44.325
in a way that I'm not just gonna
44:44.801 --> 44:46.475
know, queue all night and be waiting.
44:46.475 --> 44:48.068
Mm you know, I come in Tuesday morning,
44:48.068 --> 44:50.283
I'm still waiting for Monday night's
builds to go out so that we
44:50.283 --> 44:51.606
can start working again, sort of thing.
44:51.536 --> 44:52.536
I
44:51.626 --> 44:55.048
see. I see. Got it, gotta gotta 'cause I
was I was about to to ask you,
44:55.048 --> 44:57.829
like, okay, we we talked earlier about how
changes tend to be bigger now,
44:57.829 --> 45:00.440
but you're you're talking about optimizing
that loop and making sure that
45:00.440 --> 45:03.282
it doesn't but but to your point that's
sort of in the overnights then where it's
45:03.502 --> 45:06.483
going to the mutation extras and and all
these other things.
45:06.483 --> 45:07.483
Yeah.
45:06.628 --> 45:07.628
Well, I think I
45:07.148 --> 45:12.172
think it's all I I've I've met companies
where they're they're building
45:12.172 --> 45:14.634
up so much backlog over the course of the
day.
45:14.634 --> 45:17.172
I'm gonna say they're primarily not Circle
CI customers.
45:17.172 --> 45:21.140
They're building up so much backlog over
the course of the day that they're hoping
45:21.140 --> 45:22.781
that when they come back in the morning,
45:22.781 --> 45:24.592
yesterday's stuff will be done.
45:24.592 --> 45:27.525
And I'm just talking about like the builds
that were piling up waiting
45:27.456 --> 45:30.465
Mm. Yeah. ~ wow. Wow.
45:27.525 --> 45:30.187
to go through, right? And so, I mean,
45:30.187 --> 45:31.228
the the
45:30.465 --> 45:31.465
~
45:31.228 --> 45:33.300
goal of C D is the immediacy, right?
45:33.300 --> 45:34.510
Like, we just broke production.
45:34.510 --> 45:36.312
Awesome. That's 'cause I hit merge four
minutes ago.
45:36.802 --> 45:37.802
Right. Not
45:36.805 --> 45:37.805
Right.
45:37.974 --> 45:40.267
three o'clock in the morning, someone's
getting page 'cause their three
45:40.267 --> 45:42.600
PM thing finally made it to production.
45:42.600 --> 45:46.394
Right. So I there is some like just how do
we again,
45:46.394 --> 45:49.038
how do we optimize this and minimize it to
Yeah.
45:46.434 --> 45:49.265
Okay, no, I I get it. You're you're
45:49.265 --> 45:53.899
you're helping me flash back to client
sites I got to play with and sort
45:53.899 --> 45:56.101
of twenty seventeen, eighteen where yeah,
45:56.101 --> 45:59.013
t like QA wouldn't get something for,
45:59.013 --> 46:01.214
you know, like a month because they were
so queued up.
46:01.214 --> 46:04.266
and to your point, like all the jobs were
waiting on the other jobs in front
46:04.266 --> 46:05.737
of them in the c in the CI. Okay,
46:05.242 --> 46:06.242
Yeah.
46:05.737 --> 46:06.918
fine. Yeah, yeah, yeah, yeah, yeah.
46:06.918 --> 46:09.320
Right. Yes, fair points. Okay,
46:09.320 --> 46:10.320
cool.
46:09.946 --> 46:12.814
So yeah, sp speed matters a lot now.
46:12.984 --> 46:15.146
Yeah. Yeah. Yeah. For sure. Yeah.
46:15.146 --> 46:17.658
Otherwise it that that Q just grows even
faster.
46:17.658 --> 46:23.593
Hmm. W what do you think change in terms
of like infrastructure and and
46:23.593 --> 46:27.716
how we think about like how we ship ~ and
~ those kinds of things.
46:29.155 --> 46:33.976
Yeah. ~ AI begets AI.
46:33.976 --> 46:40.528
That's one thing I would say. Like we're
as we move faster with less less,
46:40.528 --> 46:44.059
I'm gonna say less comprehensive
understanding of every detail,
46:44.059 --> 46:48.210
right? Like we've got the same number of
humans with the same brains processing
46:47.150 --> 46:49.655
Yeah. Yep. Mm-hmm.
46:48.210 --> 46:49.760
way more change, right? So
46:49.760 --> 46:53.001
the likelihood is higher that we put
something into a production environment
that
46:53.001 --> 46:58.262
we don't fully understand. ~ and we are
building all the tools that I mean we are
46:58.436 --> 47:00.226
Building it for ourselves, but like as a
software industry,
47:00.226 --> 47:03.197
there's a ton of investment going into the
tools to make that as safe as possible.
47:03.197 --> 47:06.468
Right. And in many cases, it's gonna be
safer than what we did before.
47:06.468 --> 47:10.149
Like, you know, we could spend a sidebar
on humans reviewing PRs,
47:10.149 --> 47:12.350
like, were you even paying attention while
you're reviewing that?
47:12.350 --> 47:14.211
Like, so so there's that,
47:12.424 --> 47:13.985
I know. huh.
47:14.211 --> 47:16.231
but then but then you so you get stuff
into production.
47:16.231 --> 47:18.752
I mean, ~ like telemetry, observability,
47:18.752 --> 47:21.823
like pick your favorite word is gonna be
critical,
47:21.823 --> 47:26.074
right? Because and what I will say.
47:26.368 --> 47:31.379
Think about debugging production systems
is I know people,
47:31.379 --> 47:36.711
I know engineers who are not great at
debugging the thing they just wrote.
47:37.136 --> 47:38.136
Mm-hmm.
47:37.691 --> 47:41.932
And I know engineers who are great at
debugging a system they've never seen
before.
47:41.932 --> 47:42.932
So
47:42.192 --> 47:43.192
Mm-hmm.
47:43.913 --> 47:47.414
we're gonna be probably more in that
latter category,
47:47.414 --> 47:50.245
right? Like this notion that we can't put
something into production that
47:50.245 --> 47:54.096
we don't understand. Like I I've watched
people parachute into someone else's
47:54.096 --> 47:55.606
environment and be like, it's right there.
47:56.026 --> 47:58.049
Yeah. Yeah.
47:56.590 --> 47:58.251
That's the problem. And that
47:58.251 --> 48:02.215
takes good tooling, right? Like either
they work really hard and it's slow
48:02.215 --> 48:06.909
to get to that understanding, or they have
really good tooling.
48:06.909 --> 48:11.062
~ and specifically, I'm thinking about
like traces and understanding like
48:11.062 --> 48:12.473
how is stuff flowing through the system.
48:12.473 --> 48:15.496
And here's the full stack trace when
something goes wrong.
48:15.496 --> 48:20.060
and here's the source that I can reference
back to so I can go look at it.
48:20.060 --> 48:22.042
And and now when I said AI begets AI,
48:22.042 --> 48:24.824
like I'm going to use, first of all,
48:25.723 --> 48:27.154
To find that in the first place,
48:27.154 --> 48:29.615
right? Debugging production systems,
48:29.615 --> 48:31.006
particularly distributed systems,
48:31.006 --> 48:36.218
is an exercise in pattern matching and
correlation across enormous data sets.
48:36.442 --> 48:37.442
Yeah. Mm-hmm.
48:37.339 --> 48:38.760
You know what's really good at that?
48:38.760 --> 48:41.501
Is AI. Like way better than humans,
48:39.448 --> 48:41.997
AI. Mm-hmm. Mm-hmm.
48:41.501 --> 48:42.501
right?
48:42.151 --> 48:43.442
Like you may not have written it,
48:43.442 --> 48:46.714
but just saying this all appears to be
this is related to that,
48:46.714 --> 48:49.425
and this is related to that. Like that was
my sound of a computer anyway,
48:49.425 --> 48:50.786
is like my modem from when I was kid.
48:50.786 --> 48:54.908
Anyway, like that connection exists.
48:55.320 --> 48:57.581
And this is likely this. And I went
through this code,
48:57.581 --> 48:59.922
right? Like I have access to all the repos
and I have access to all this data.
48:59.922 --> 49:02.503
Like that will be a remarkable
improvement.
49:02.503 --> 49:05.145
I think people have nibbled at the edges
of that so far,
49:05.145 --> 49:06.965
but I think it's we're gonna see more of
that.
49:06.965 --> 49:11.708
And there's going to be a greater need,
49:11.708 --> 49:17.350
I think, for the humans to really be able
to do that large scale thinking.
49:17.350 --> 49:19.011
It'll be it'll be tool assisted,
49:19.011 --> 49:21.092
right? But
49:21.358 --> 49:22.649
You know, the systems will be complex,
49:22.649 --> 49:25.610
they'll be fresh. You won't you won't
actually know what the structure of
49:25.610 --> 49:28.810
all of it is. But I mean, I've been doing
this a long time.
49:28.810 --> 49:31.634
I've been to a lot of incidents where I
had nothing to do with the code that
49:31.634 --> 49:35.596
was acting badly, and I still was able to
figure out what what was happening.
49:35.596 --> 49:36.916
And I know peop lots of people like that,
49:35.952 --> 49:37.515
Yeah. Yeah.
49:36.916 --> 49:37.916
right? So
49:37.548 --> 49:42.379
I I think I mean, well, we responded to
49:42.379 --> 49:47.562
a Linux CVE recently that we were trying
to fit how does this impact us?
49:47.562 --> 49:50.894
And we got four people on a call reading
the Linux kernel.
49:51.652 --> 49:55.175
very claw-assisted, right? But and then
building,
49:55.175 --> 49:57.697
you know, tests to validate, you know,
49:57.697 --> 49:59.019
whether or not it impacts us kind of
thing.
49:59.019 --> 50:01.081
And like that's that's going to be the
world that we're in,
49:59.707 --> 50:01.911
wow. Yeah. Yeah.
50:01.081 --> 50:02.081
right? And that's like that was
50:02.021 --> 50:03.803
not the Linux kernel wasn't written by AI,
50:03.803 --> 50:05.264
but it also wasn't written by us.
50:04.430 --> 50:06.966
Right. Right. Right.
50:05.264 --> 50:06.996
Right. And so
50:06.996 --> 50:09.098
like a very let's it's not a distributed
system,
50:09.098 --> 50:12.851
but a very large scale application that's
extremely complex,
50:12.851 --> 50:16.725
where I can be guided to a point and then
make rational decisions about
50:16.725 --> 50:18.136
how that impacts my system.
50:18.905 --> 50:21.587
Right, I think is is going to be a real
part of the job.
50:21.587 --> 50:25.851
And so I think being really comfortable
leveraging the tools that we have,
50:25.851 --> 50:31.636
but also like I don't I'm sad to say I
don't know how to get people
50:31.636 --> 50:34.168
to be great debuggers. It's not something
I've spent a lot of time on,
50:34.168 --> 50:35.780
but I think about it. ~
50:35.016 --> 50:36.016
yeah, yeah.
50:35.780 --> 50:39.314
as I s, you know, I've spent time in
incidents like anybody else that runs
50:39.314 --> 50:45.548
a distributed system. And I I was I was
working with one group at some point
50:45.548 --> 50:46.548
and I was like,
50:47.108 --> 50:49.313
I want you all to spend the next week
playing Wordle.
50:51.524 --> 50:53.115
And then I want you to tell me at the end,
50:53.115 --> 50:57.907
like how you thought about reducing the
problem space.
50:57.907 --> 51:01.800
Right? Like, how do you take because I've
got the alphabet and I've
51:01.800 --> 51:02.920
got every word that's ever ex well,
51:02.920 --> 51:04.181
every five-letter word that exists,
51:04.181 --> 51:05.181
right? I think it's five letters.
51:05.111 --> 51:06.111
I haven't played Wordle in a while.
51:05.986 --> 51:06.986
It is. It is fine.
51:08.113 --> 51:10.354
And I've got a fixed number of guesses,
51:10.354 --> 51:14.527
right? I mean, anyone that's listening to
this doesn't know how to play Wordle,
51:14.527 --> 51:15.837
just look it up. Like, I know why I'm
explaining.
51:15.837 --> 51:20.660
But like the the I have this broad problem
space and I use techniques.
51:21.306 --> 51:25.318
To analyze, to probe the space to try to
determine likelihoods,
51:25.318 --> 51:28.969
right? Of where I'm going. And of course
the game is kind of designed
51:28.969 --> 51:30.080
to drive you through that, right?
51:30.080 --> 51:31.236
But that was like part of the.
51:31.236 --> 51:34.522
And so I think like that kind of
reasoning.
51:34.522 --> 51:37.283
There's also this other thing that I'm
totally obsessed with.
51:37.283 --> 51:38.823
I'm so sorry. This is totally off the
rails.
51:38.823 --> 51:40.964
But it's my God, I'm gonna forget the name
of it.
51:40.964 --> 51:43.946
I think it's called the Matson Consensus
Protocol.
51:43.946 --> 51:47.447
And it's used in wilderness search and
rescue,
51:47.447 --> 51:49.328
where
51:50.018 --> 51:52.809
so someone is missing or an aircraft is I
think it came from aircraft originally.
51:52.809 --> 51:56.140
And someone is missing or an airplane is
missing or whatever,
51:56.140 --> 51:57.911
and you get together a bunch of experts,
51:57.911 --> 52:04.513
and they all get points to allocate to
search areas based
52:04.513 --> 52:07.794
on their expertise and understanding of
likelihood.
52:07.794 --> 52:12.546
And that's how you decide with a limited
search party where you should go look.
52:13.326 --> 52:15.628
Right. And one of the things I love about
software is we always have
52:15.628 --> 52:17.029
to invent everything for ourselves.
52:17.029 --> 52:20.361
Like there are so many industries and
disciplines where people have solved these
52:20.361 --> 52:24.594
things, right? And so this notion that
like we have a bunch of experts on
52:24.594 --> 52:27.706
a call or whatever, like in like talking
about production debugging.
52:27.706 --> 52:31.299
Sorry, this is all built from the
assumption that AI is gonna make
52:31.299 --> 52:32.553
a mistake and then we're gonna put
something in production.
52:32.553 --> 52:36.312
We're gonna try like having tool and you
could automate that too,
52:36.312 --> 52:41.276
right? Like having tools and techniques to
say to drive towards
52:41.637 --> 52:44.539
the likely cause of the problem instead
of,
52:44.539 --> 52:46.861
I don't know, AI built all of this.
52:46.861 --> 52:50.863
Let's start at line one and read through
to line fifty million
52:50.863 --> 52:52.905
and maybe somewhere in there we'll see
something that's wrong.
52:51.236 --> 52:53.579
Yeah. Yeah, yeah.
52:52.905 --> 52:53.905
Right. And so I think
52:53.785 --> 52:56.687
I think we'll be forced to build,
52:56.687 --> 52:59.549
you know, better tooling, better
approaches,
52:59.549 --> 53:01.951
better practices. Again, I assume this
will
53:01.110 --> 53:02.110
Mm, interesting.
53:01.951 --> 53:03.652
all show up in software because that's how
we solve things.
53:04.526 --> 53:07.428
I I like I like how you're talking about
we we talked earlier about
53:07.428 --> 53:11.340
how you were saying that like we all need
to have more of a product lean these days.
53:11.340 --> 53:13.461
and lots of guests have talked about like,
53:13.461 --> 53:15.922
yeah, we we're all closer to the product
now.
53:15.922 --> 53:18.233
~ that that shows up in my life,
53:18.233 --> 53:23.827
you know. ~ I I advise people talk to the
AI about the domain.
53:23.827 --> 53:26.098
If you're talking to it about the curly
braces and the semicolons,
53:26.098 --> 53:28.199
you're missing the point. Tell it about
53:27.600 --> 53:28.600
Mm-hmm.
53:28.199 --> 53:29.860
the business problem you're trying to
solve,
53:29.860 --> 53:32.831
right? And and it will be better at
helping you solve that business problem.
53:32.831 --> 53:33.831
So I think
53:34.074 --> 53:35.074
There's
53:34.334 --> 53:37.136
a there was a whole category of software
developers who really wanted
53:37.136 --> 53:41.169
to make sure that, you know, each line was
perfectly idiomatic and you know that we,
53:41.169 --> 53:45.713
you know, and I'm not sure that skill set
is as important as it used to be.
53:45.713 --> 53:51.838
But to your point, this I can debug
something ~ skill set
53:51.838 --> 53:55.721
is still quite valuable, especially in
these complex distributed systems that
we've
53:55.721 --> 53:57.863
created. So I think for a lot of us,
53:57.863 --> 53:59.183
like we're just gonna ask Claude to do it,
53:59.183 --> 54:00.183
you know.
53:59.970 --> 54:03.801
~ it's funny when when I'm doing that with
my own sort of smaller systems,
54:03.801 --> 54:07.172
~ if this is part of my solo work,
54:07.172 --> 54:09.423
like I will it's not that I'm racing it,
54:09.423 --> 54:11.603
but I will I will get it kicked off and
say,
54:11.603 --> 54:12.734
start looking at this production issue,
54:12.734 --> 54:14.964
please, right away. And then I'll go do
the same thing.
54:14.964 --> 54:17.845
And it's it's not so much that I think I'm
going to win,
54:17.845 --> 54:19.876
but that I might spot something that I can
share with it,
54:19.876 --> 54:21.706
you know, to to to help it along.
54:21.706 --> 54:23.226
And every now and again I win,
54:23.226 --> 54:25.387
but it's pretty rare. Usually Claude gets
there first.
54:25.387 --> 54:28.778
But I I I think the point you're making is
a different one though.
54:28.869 --> 54:33.291
Which is that like the if you've tapped
out the agent's ability to figure
54:33.291 --> 54:36.903
out what's going on, then that other skill
set of being the human who
54:36.903 --> 54:41.566
can jump on that, you know, severity one
call and like spot something that might
54:41.566 --> 54:45.318
help d also continues to be an important
skill set,
54:45.318 --> 54:47.189
you know. That maybe the idiomatic
54:46.478 --> 54:47.478
Yeah.
54:47.189 --> 54:51.491
code guy or or or or developer is less
useful than they used to be,
54:51.491 --> 54:55.133
but the being able to spot the issue in a
large distributed system needle
54:55.133 --> 54:56.724
in a haystack is still quite valuable.
54:56.724 --> 54:57.724
Yeah.
54:57.828 --> 55:00.289
Yeah, I mean, I would describe it as
systems thinking,
55:00.289 --> 55:01.410
right? Like
55:00.622 --> 55:01.622
Yeah, yeah.
55:01.710 --> 55:04.792
understanding this thing over here impacts
this thing over here,
55:04.762 --> 55:05.762
Mm-hmm.
55:05.112 --> 55:08.674
right? And you could allow your agent to
build up that context.
55:08.674 --> 55:13.957
And like we'll probably build more
permanent memory and context around these
systems
55:13.957 --> 55:17.209
so that it isn't, hey, could you read all
this code right now like
55:17.209 --> 55:19.431
you know nothing about our code base and
start from scratch and here's
55:19.431 --> 55:22.382
the error kind of thing? Like I think
that's a thing we'll we'll advance.
55:22.382 --> 55:24.954
But I also think the the
55:26.148 --> 55:30.960
The fun and terrifying thing about
prompting is we pass on our human biases,
55:30.960 --> 55:32.591
right? Like we we we had
55:31.760 --> 55:32.760
Course. Yeah.
55:32.591 --> 55:35.092
an incident recently where a couple of us
were trying to
55:35.092 --> 55:37.853
use some production debugging agent tools.
55:37.853 --> 55:41.275
And after the incident,
55:41.275 --> 55:44.216
in our review of like what what could we
have done better,
55:44.216 --> 55:46.738
we both kind of independently went and
retried,
55:46.738 --> 55:49.529
knowing what had happened, right?
55:49.529 --> 55:51.550
No now we know, so it's a little bit of a
cheat,
55:51.550 --> 55:52.550
but like.
55:52.324 --> 55:54.916
What could I have asked that would have
got me there sooner?
55:54.916 --> 55:59.889
And you could see the questions we
initially asked were so burdened with bias
that
55:59.889 --> 56:03.392
we sent our agents off on wild goose
chases instead
56:01.220 --> 56:03.189
Down the wrong direction. Yeah.
56:03.392 --> 56:05.093
of giving them the freedom to be like,
56:05.093 --> 56:07.675
I don't know, I think it might be over
here.
56:07.675 --> 56:08.749
You know what I mean? Cause it was like,
56:08.749 --> 56:10.427
what you know, what is causing X to
happen?
56:10.427 --> 56:12.138
But X was a symptom, not a cause,
56:12.138 --> 56:13.138
sort of thing, right?
56:12.196 --> 56:13.196
Right, right.
56:13.259 --> 56:17.341
And ~ and so I think it takes both,
56:17.341 --> 56:21.024
right? I I one, I need to guide the agent
in the right direction.
56:21.601 --> 56:25.562
Or spin up a bunch. I mean, in that that
whole sort of like search protocol idea,
56:24.590 --> 56:26.447
Yeah. It does. It does.
56:25.562 --> 56:26.573
the thing is I kind
56:26.573 --> 56:28.964
of have unbounded agents if I want them,
56:28.964 --> 56:31.865
assuming you know, if soon of my
foundation model is actually
56:31.865 --> 56:33.125
up and servicing requests right now.
56:33.125 --> 56:34.125
And
56:33.252 --> 56:34.252
Yeah.
56:33.635 --> 56:37.377
so if it's their incident, they're they
probably I wonder if they
56:37.377 --> 56:41.398
use other people's models just let's not
talk about that.
56:41.398 --> 56:44.590
Okay, so ~ yeah,
56:42.528 --> 56:43.630
It turtles all the way down.
56:44.590 --> 56:49.641
yeah, exactly. So ~ they pr they have like
some someone has a like open source model
56:49.641 --> 56:51.112
under their desk that they just point all
that at.
56:51.364 --> 56:56.206
So you, you know, you have kind of this
unlimited capacity and you
56:56.206 --> 57:01.307
can but but knowing how to seed not random
57:01.307 --> 57:04.869
but multi-path investigation, like that's
what a good IC does,
57:03.834 --> 57:05.738
Yeah. Right.
57:04.869 --> 57:05.869
right? You go
57:05.869 --> 57:07.610
look at this, you go look at that.
57:07.610 --> 57:10.311
In in a less well managed incident,
57:10.311 --> 57:12.671
12 people are looking at the same thing
when the
57:12.324 --> 57:13.324
Yeah.
57:12.671 --> 57:13.822
real problem is somewhere else,
57:13.822 --> 57:14.822
right?
57:14.125 --> 57:15.366
It's funny when when you say that,
57:15.366 --> 57:16.745
what what occurs to me is like,
57:16.745 --> 57:20.599
yeah, I feel like it goes better if I
describe to Claude,
57:20.599 --> 57:23.210
inevitably, ~ in my world anyway,
57:23.210 --> 57:25.971
what I'm observing and not try to,
57:25.390 --> 57:26.390
Mm.
57:26.112 --> 57:28.193
you know, just just for observations.
57:28.193 --> 57:29.483
Like even
57:28.996 --> 57:29.996
Yeah.
57:29.483 --> 57:32.205
something as dumb as is down can be really
helpful.
57:32.205 --> 57:35.817
I mean, i because it'll just go go start
doing getting its own observations,
57:35.817 --> 57:36.817
you know, and it might be
57:36.421 --> 57:37.421
Yeah.
57:36.777 --> 57:41.220
faster at it than I am. So I will have had
an observation that pulled me to it.
57:41.970 --> 57:42.970
so maybe
57:42.029 --> 57:43.029
Uh-huh.
57:42.430 --> 57:44.761
it's like an AWS Cloud Watch alert or
something.
57:44.761 --> 57:48.153
I might like copy the cut text of that and
just paste it in and say go.
57:48.153 --> 57:52.325
~ and I I don't bother to do much
theorizing at that at that stage.
57:52.229 --> 57:53.229
Yeah.
57:52.765 --> 57:55.486
But but I also wanna there there's
something you said about how like,
57:55.486 --> 57:56.727
Kyle, that's just systems thinking.
57:56.727 --> 58:00.259
And and I think you're right, but I have
known people that I thought
58:00.259 --> 58:03.770
of as good systems thinkers forward in
terms of like designing a system,
58:03.770 --> 58:05.191
but weren't necessarily
58:04.496 --> 58:05.496
Mm.
58:05.191 --> 58:07.432
the person you would call when it all
breaks.
58:08.142 --> 58:12.877
Right. I think there there were other
people that were really good at like
gisting
58:12.877 --> 58:18.302
a situation very quickly. ~ that were and
and they might also have been good forward
58:18.302 --> 58:20.975
systems thinkers, who knows? But sometimes
people who I think of
58:20.975 --> 58:24.288
as very good forward systems thinkers were
not good at the quick gist.
58:24.288 --> 58:27.111
~ so yeah, I mean, you're right.
58:27.111 --> 58:28.392
It is a kind of systems thinking,
58:28.392 --> 58:30.715
but I still think that there's like
specializations on it,
58:30.715 --> 58:31.876
I guess, if that makes any sense.
58:32.996 --> 58:35.703
Yeah yeah. I think
58:38.191 --> 58:44.213
Di yeah, diagnosing is different than ~
than building a system.
58:44.213 --> 58:47.039
and I th I think
58:49.200 --> 58:51.391
I don't know. Yeah, the other the other
big thing that I think about there
58:51.391 --> 58:53.031
in there is is this pattern matching.
58:53.031 --> 58:55.182
And maybe that's not quite systems thing.
58:55.182 --> 58:56.182
And it's like
58:56.250 --> 58:57.250
Yeah.
58:56.723 --> 58:57.833
it's like thinking fast and slow.
58:57.833 --> 58:59.684
We run into the like, well, we've seen
this before,
58:58.638 --> 59:01.356
Right, right. Right.
58:59.684 --> 59:01.404
therefore it must be this. That's the
worst kind
59:01.404 --> 59:07.046
of pattern matching, right? As opposed to
that thing there feels like
59:07.046 --> 59:08.697
it might be related to this thing over
here,
59:08.697 --> 59:09.837
even though no one's seeing that.
59:09.837 --> 59:11.568
And it's it's ~
59:10.372 --> 59:11.843
Right. That
59:11.843 --> 59:14.514
that story about the firefighter who runs
into the building and just says
59:14.514 --> 59:17.505
get out and does can't really explain why
or how he knows,
59:17.505 --> 59:19.135
but like just get out. Yeah,
59:17.912 --> 59:18.999
Yeah. Yeah.
59:19.176 --> 59:22.587
and that is useful, right? And and
sometimes thinking fast is not what you
need.
59:22.587 --> 59:23.947
It's thinking slow that you need.
59:23.947 --> 59:26.819
But yeah. Wow. We are really
59:25.805 --> 59:26.805
Right.
59:26.819 --> 59:27.919
solving the world's problems now,
59:27.919 --> 59:28.919
Rob. Sorry,
59:28.802 --> 59:30.583
Well, I mean, I I do
59:28.879 --> 59:29.879
keep going.
59:30.583 --> 59:32.714
think someone is gonna build a really good
version of this,
59:32.714 --> 59:34.755
right? Which is like, you know,
59:33.801 --> 59:37.091
Mm. Mm. Yeah.
59:34.755 --> 59:36.156
I said, Well, that's a good IC.
59:36.156 --> 59:37.257
But do I
59:37.537 --> 59:42.259
like being a good IC, like there there are
these things I I always joke about this,
59:42.259 --> 59:43.540
or at least joke about it recently,
59:43.540 --> 59:46.492
like in the late nineties, I think it was
the late nineties,
59:46.492 --> 59:48.873
we were really into this concept of expert
systems,
59:48.873 --> 59:49.873
which is like we're
59:49.072 --> 59:50.072
Mm-hmm.
59:49.472 --> 59:51.894
gonna take all the world's knowledge and
build it into computers and then
59:51.894 --> 59:53.525
we can just ask it and it'll solve
anything.
59:53.525 --> 59:56.587
And we obviously gave up like so many
other things,
59:56.587 --> 59:58.388
that in the unified email inbox.
59:58.722 --> 01:00:02.216
And like in 2026,
01:00:02.216 --> 01:00:04.508
we're accidentally building those things,
01:00:04.508 --> 01:00:06.150
right? We're taking all of our knowledge.
01:00:05.550 --> 01:00:08.569
Mm, yeah. Hmm.
01:00:06.150 --> 01:00:07.251
And every time we have a conversation,
01:00:07.251 --> 01:00:08.652
we're like, cool, write a markdown file,
01:00:08.652 --> 01:00:11.425
write a markdown file. And we just have
these enormous markdown files.
01:00:11.425 --> 01:00:16.170
But we can give like useful direct
guidance and direction to an LLM
01:00:16.430 --> 01:00:21.395
and have it execute on something based on
our learned experience,
01:00:21.395 --> 01:00:22.395
right? Like
01:00:21.668 --> 01:00:22.668
Yeah,
01:00:22.041 --> 01:00:23.041
yeah.
01:00:22.287 --> 01:00:23.287
Here's
01:00:22.537 --> 01:00:24.569
how you review code. We talked about
everyone building their
01:00:24.569 --> 01:00:27.931
own code reviewers because we've done the
same review over and over.
01:00:27.931 --> 01:00:32.034
Like the way that we built ours was we
just pulled from all of our existing
repos.
01:00:32.034 --> 01:00:36.227
We said, go read all the reviews and find
the authors that comment the most
01:00:36.227 --> 01:00:38.297
and that people trust, and then figure
01:00:37.818 --> 01:00:38.818
Hmm.
01:00:38.297 --> 01:00:42.511
out what they say over and over and over
and just say it yourself.
01:00:42.511 --> 01:00:45.053
Right? Because every every senior engineer
is like,
01:00:45.053 --> 01:00:47.534
why am I telling you the same thing again
today?
01:00:47.534 --> 01:00:49.916
We talked about it yesterday. So let's not
bother.
01:00:50.170 --> 01:00:51.170
Nice.
01:00:50.810 --> 01:00:53.261
Giving that broken thing to the senior
engineer to say,
01:00:53.261 --> 01:00:54.541
again, like
01:00:53.520 --> 01:00:54.520
Yeah, yeah.
01:00:54.721 --> 01:00:56.302
we talked about this, do it this way,
01:00:56.302 --> 01:00:57.942
right? Like we we can encode all that
stuff.
01:00:57.942 --> 01:01:03.882
And like when it became really easy and
and the the
01:01:03.882 --> 01:01:08.275
f the ris return on investment for that
engineer became really clear,
01:01:08.275 --> 01:01:10.786
like, wow, I can stop having this
conversation because like,
01:01:10.786 --> 01:01:13.526
you know, cute little Claude told you not
to do that.
01:01:13.526 --> 01:01:14.667
And like you listen to Claude,
01:01:14.667 --> 01:01:16.927
great, like I can move on and do higher
value things.
01:01:16.927 --> 01:01:19.068
And I think, you know.
01:01:19.492 --> 01:01:22.333
These things like being a great incident
commander,
01:01:22.333 --> 01:01:24.603
I keep saying I see, but incident
commander is what I mean,
01:01:24.603 --> 01:01:26.355
where you
01:01:26.071 --> 01:01:27.071
Yep.
01:01:26.615 --> 01:01:31.337
where it's encoded knowledge that you've
built through reps,
01:01:31.337 --> 01:01:34.679
and then you try to train up a new person,
01:01:34.679 --> 01:01:38.881
right? And like I think, I think we could
build the tooling with
01:01:38.881 --> 01:01:42.742
all of the knowledge that we have to guide
you through that process.
01:01:42.742 --> 01:01:44.603
And it would be some combination of like,
01:01:44.603 --> 01:01:46.234
I fed it, you know.
01:01:47.246 --> 01:01:50.999
Pager duty or incident IOs, like how to be
an incident commander training guide.
01:01:50.999 --> 01:01:54.641
And then I fed it all of our historical
incident reports.
01:01:54.641 --> 01:01:58.894
And then I fed it access to honeycomb,
01:01:58.894 --> 01:02:02.547
data dog, grafana, like whatever your tool
is.
01:02:02.574 --> 01:02:03.574
All the things.
01:02:04.248 --> 01:02:08.261
And now it's like paging the right person
at the right.
01:02:08.261 --> 01:02:10.022
~ this looks like it's a problem over
here.
01:02:10.022 --> 01:02:11.103
Like, you know who's the expert on that?
01:02:11.103 --> 01:02:12.984
Because I read all of the Git history,
01:02:12.984 --> 01:02:15.086
right? And I know that this is the person
who knows that.
01:02:13.860 --> 01:02:15.346
Yeah. Yeah, yeah, yeah.
01:02:15.487 --> 01:02:16.487
Yeah.
01:02:15.960 --> 01:02:19.211
Let's call them, right? And I I don't
know.
01:02:19.211 --> 01:02:21.322
I mean, it's some someone's probably
someone's building that already.
01:02:20.388 --> 01:02:22.503
Hundred percent. It probably happening,
01:02:21.322 --> 01:02:24.724
But like I I think that it and right.
01:02:22.503 --> 01:02:25.250
yeah. Somebody's vibe coding it right now,
01:02:24.724 --> 01:02:27.105
Yeah, that might
01:02:25.250 --> 01:02:26.250
in fact.
01:02:27.105 --> 01:02:28.245
not be the version I think. Yeah,
01:02:28.245 --> 01:02:30.586
well, whatever. Vib coding doesn't I don't
even know what that means anymore.
01:02:30.586 --> 01:02:34.778
But like huge amounts of data to process,
01:02:34.778 --> 01:02:38.569
right? Like this is it's such a fantastic
application of of this tooling.
01:02:37.636 --> 01:02:39.094
Love it. Love it.
01:02:38.569 --> 01:02:39.569
And I think
01:02:39.194 --> 01:02:42.871
it it will it will force us to have to
build it and it will make
01:02:42.871 --> 01:02:45.242
it possible to build it. And I think
that's a that's like a weird.
01:02:45.787 --> 01:02:46.787
Yeah.
01:02:45.923 --> 01:02:48.740
Interplay of a lot of scenarios with AI.
01:02:46.785 --> 01:02:47.790
Yeah, yeah, yeah, yeah.
01:02:48.879 --> 01:02:49.879
Yep.
01:02:49.222 --> 01:02:51.427
Right now it's important, but now it's
possible.
01:02:51.427 --> 01:02:54.646
Right? Yeah. Yeah.
01:02:51.458 --> 01:02:53.810
And we can afford it. Like you were
talking about mutation testing earlier.
01:02:53.810 --> 01:02:54.810
It's like we couldn't afford
01:02:54.680 --> 01:02:56.341
it before and we maybe didn't need it.
01:02:56.341 --> 01:02:58.113
We do now. Yeah. Yep.
01:02:57.220 --> 01:02:58.220
Right, right.
01:02:58.193 --> 01:02:59.203
And suddenly we can afford it,
01:02:59.203 --> 01:03:03.106
which is great. Okay. What one more
question on the what you're observing
01:03:03.106 --> 01:03:04.837
and then we need to sort of start,
01:03:04.837 --> 01:03:05.908
you know, windy down the hill.
01:03:05.908 --> 01:03:07.428
You've been very generous with your time
so far,
01:03:07.428 --> 01:03:10.201
Rob. Thank you. ~ but but one more
question.
01:03:10.201 --> 01:03:13.083
We we've talked a little bit about
industry trends and what you're seeing
01:03:13.083 --> 01:03:16.385
and what you can grok about what's
happening with hiring and economics
01:03:16.385 --> 01:03:18.737
and those kinds of things. And you talked
a little bit about how,
01:03:18.737 --> 01:03:20.268
you know, everybody's committing now and
01:03:20.612 --> 01:03:22.073
Really, really good insightful stuff.
01:03:22.073 --> 01:03:24.315
Anything else that you're seeing around,
01:03:24.315 --> 01:03:26.016
you know, how teams are built,
01:03:26.016 --> 01:03:30.100
right? I think especially in the
consultancy world that I've that I've
lived
01:03:30.100 --> 01:03:33.443
in for a lot of my career, we got really
used to like,
01:03:33.443 --> 01:03:34.894
okay, you're gonna need, you know,
01:03:34.894 --> 01:03:38.917
sort of call it six software developers
and a project manager
01:03:38.917 --> 01:03:40.238
and a business analyst, you know,
01:03:40.238 --> 01:03:42.570
especially sort of pre-Eric Reese,
01:03:42.570 --> 01:03:45.493
you know, ~ lean thinking, like maybe
after that it would have been
01:03:45.493 --> 01:03:47.754
a product manager and not so much a
business analyst,
01:03:47.754 --> 01:03:49.596
but you got used to certain shapes.
01:03:49.912 --> 01:03:50.913
teams, right? And
01:03:50.469 --> 01:03:51.469
Yeah.
01:03:51.093 --> 01:03:53.544
and of those six developers, like you
needed a couple of front end and
01:03:53.544 --> 01:03:55.816
a couple of back end, they all really
needed to be full stack,
01:03:55.816 --> 01:03:57.337
but you still needed some specialists,
01:03:57.337 --> 01:03:58.584
you know, and and
01:03:58.042 --> 01:03:59.042
Yeah.
01:03:58.584 --> 01:03:59.738
maybe depending on the domain,
01:03:59.738 --> 01:04:01.339
you needed something else and something
else.
01:04:01.339 --> 01:04:05.041
But you know, there was sort of a a shape
that you could almost just stamp out,
01:04:05.041 --> 01:04:08.604
you know. Are you seeing that change any
like
01:04:10.748 --> 01:04:16.190
Yeah. I yes. But I don't know what the
destination is.
01:04:16.190 --> 01:04:19.350
I just know it doesn't feel right anymore.
01:04:19.350 --> 01:04:20.679
Right. So there's like you know,
01:04:20.679 --> 01:04:26.671
we already talked about kind of giving
more product thinking to the engineers.
01:04:26.671 --> 01:04:28.383
And maybe that's what that was a path
01:04:27.108 --> 01:04:28.257
Yeah. Mm-hmm.
01:04:28.383 --> 01:04:30.703
we were on anyway, but it was just more
clear,
01:04:30.703 --> 01:04:34.454
right? I think the if you think about
01:04:35.155 --> 01:04:38.386
the cycle, right?
01:04:38.386 --> 01:04:39.908
The life cycle of
01:04:40.357 --> 01:04:42.708
Let's call it a feature, doesn't really
matter.
01:04:42.708 --> 01:04:44.199
Pick big project, small project.
01:04:44.199 --> 01:04:48.351
And the time horizons to execute on a
thing,
01:04:48.351 --> 01:04:50.342
call that unit of work, you know,
01:04:50.342 --> 01:04:56.525
X, X amount of work. The handoffs between
multiple parties within that
01:04:56.584 --> 01:05:00.247
X amount of work might account for 10%,
01:05:00.247 --> 01:05:01.247
right?
01:05:00.687 --> 01:05:01.687
Uh-huh.
01:05:01.188 --> 01:05:02.882
Maybe 20. I mean, I'm being really
generous.
01:05:02.882 --> 01:05:08.881
It's probably like 80, but and then take X
and make it achievable in
01:05:08.881 --> 01:05:09.952
one tenth of the time.
01:05:10.640 --> 01:05:11.640
Mm-hmm.
01:05:10.720 --> 01:05:11.720
Or like
01:05:11.501 --> 01:05:12.801
one one hundredth of the time.
01:05:12.801 --> 01:05:13.922
If
01:05:13.327 --> 01:05:14.327
Mm-hmm.
01:05:13.922 --> 01:05:18.845
that what was twenty percent time spent on
handoffs would now become
01:05:19.426 --> 01:05:23.929
two hundred percent, right? Like we could
probably use real numbers
01:05:23.929 --> 01:05:25.430
and model this out, but it's not that
important.
01:05:25.430 --> 01:05:29.393
200% of your time spent handing things off
is completely unacceptable,
01:05:29.393 --> 01:05:33.255
right? So you want in my mind,
01:05:33.255 --> 01:05:38.159
you want to take more of the roles and
capabilities of the team
01:05:38.539 --> 01:05:40.040
and consolidate them.
01:05:40.746 --> 01:05:45.998
Into fewer actors. Now that doesn't mean
like all PMs don't have jobs now,
01:05:45.998 --> 01:05:52.660
right? Like people bring strengths to
teams in ways that others
01:05:52.660 --> 01:05:53.980
are not capable of picking up,
01:05:53.980 --> 01:05:55.961
right? And sort of saying, well,
01:05:55.961 --> 01:05:58.502
well, everyone's just an engineer and
they'll just do all the other stuff,
01:05:58.502 --> 01:06:02.813
right? Like I've I've designed apps before
and been like laughed out of the room.
01:06:02.813 --> 01:06:04.974
So like someone needs to do some design
thinking,
01:06:04.974 --> 01:06:06.184
right? And so
01:06:07.022 --> 01:06:10.815
I but how we structure that, like what is
what is the ratio,
01:06:10.815 --> 01:06:14.425
right? Like it's like we kind of need less
because the engineers are doing
01:06:14.425 --> 01:06:16.730
a lot themselves, but we need more because
they can get so much done.
01:06:16.730 --> 01:06:21.374
Like, I think that's the part that we we
can't quite figure out just yet.
01:06:19.748 --> 01:06:22.153
Yeah. Where it's gonna go. Yeah.
01:06:21.374 --> 01:06:24.506
It's gonna be it's gonna be something that
will evolve and we'll see emerge,
01:06:22.153 --> 01:06:25.410
Yeah. Yeah.
01:06:24.506 --> 01:06:25.537
right? And different
01:06:25.537 --> 01:06:29.450
teams will be different. If you are a
startup before people like that
01:06:29.450 --> 01:06:32.673
was always just the engineers doing it and
you accepted that their design was crap,
01:06:32.673 --> 01:06:34.454
or they just lifted it off some site.
01:06:34.454 --> 01:06:36.596
I mean now they're using claw design or
whatever to make it.
01:06:36.869 --> 01:06:42.273
It can happen. And so, like, everyone has
to do more yet less.
01:06:42.273 --> 01:06:46.827
We can all get more done. Everyone's tools
are assisted,
01:06:46.827 --> 01:06:48.438
right? Like the PM can generate,
01:06:48.438 --> 01:06:50.974
you know, perspective that can analyze the
data.
01:06:50.974 --> 01:06:53.302
The designer can create designs that can
do all these things,
01:06:53.302 --> 01:06:59.007
also augmented and faster. And so I think
then
01:06:59.069 --> 01:07:04.571
I do think overall team size is gonna
trend smaller.
01:07:04.571 --> 01:07:05.571
That's my gut.
01:07:06.448 --> 01:07:08.589
purely based on just communication
overhead,
01:07:08.589 --> 01:07:10.392
right? Like ~
01:07:08.940 --> 01:07:10.721
Mm-hmm. Yeah, right.
01:07:11.413 --> 01:07:12.865
was every, you know, whatever,
01:07:12.865 --> 01:07:16.439
every node I add is I I forget if it's
like factorial or whatever.
01:07:14.800 --> 01:07:16.874
Two to the end. Yeah. Mm-hmm.
01:07:16.439 --> 01:07:17.439
Yeah, exactly.
01:07:17.180 --> 01:07:18.180
So
01:07:19.758 --> 01:07:22.969
We want to reduce that, which then means,
01:07:22.969 --> 01:07:28.610
you know, principles like team topologies
and how do I create independence between
01:07:28.610 --> 01:07:30.491
teams so they can execute, you know,
01:07:30.491 --> 01:07:32.912
on their own value stream or whatever
terminology you want to use.
01:07:32.784 --> 01:07:33.784
Mm-hmm.
01:07:32.912 --> 01:07:33.912
I
01:07:33.272 --> 01:07:36.653
think that becomes important. But then
it's like,
01:07:36.653 --> 01:07:39.163
I could literally touch every system in
five minutes.
01:07:39.163 --> 01:07:41.404
I like I I spent four weeks working
01:07:39.780 --> 01:07:41.259
Yeah. Yeah.
01:07:41.404 --> 01:07:43.794
on something in a programming language
that I still don't understand.
01:07:43.794 --> 01:07:47.556
Because I know enough about programming
that I can tell roughly if it's right,
01:07:43.918 --> 01:07:45.873
Right, right. Yeah. Yeah.
01:07:47.556 --> 01:07:48.556
but
01:07:48.528 --> 01:07:49.528
I
01:07:49.424 --> 01:07:52.104
Couldn't sit down and handwrite that
because I've never taken the time
01:07:52.104 --> 01:07:53.148
to learn that particular language,
01:07:53.148 --> 01:07:55.586
but it doesn't like like the ability to
kinda
01:07:54.222 --> 01:07:55.679
And the last thirty percent doesn't matter
as
01:07:55.679 --> 01:07:58.110
much as it used to because machines are
maintaining it now,
01:07:58.110 --> 01:07:59.436
not us. So yeah, yeah.
01:07:58.488 --> 01:07:59.508
Yeah. I mean, I
01:07:59.508 --> 01:08:03.369
know if it's well tested. I know if it
does what I wanted it to do.
01:08:03.369 --> 01:08:06.880
I have security scanner. Like have all
this other automation to make sure that
01:08:06.880 --> 01:08:08.041
I'm not making massive blunders.
01:08:08.041 --> 01:08:10.101
I have other people. We have all these
review tools we build.
01:08:10.101 --> 01:08:14.972
Like we have the machinery to make that
happen in a way that then allows me to be,
01:08:14.972 --> 01:08:18.754
you know, to move independently across a
system as opposed to like I'm tucked
01:08:18.754 --> 01:08:20.304
in my little corner and I just do this
thing.
01:08:20.304 --> 01:08:21.634
So ~ it feels like
01:08:20.622 --> 01:08:21.622
Yeah, yeah, for sure.
01:08:21.634 --> 01:08:24.635
the greatest non-answer. Like I it's
absolutely changing,
01:08:24.635 --> 01:08:27.656
but like you can see factors that are
pushing.
01:08:28.185 --> 01:08:33.837
In both directions, and maybe we end up
after all of that push and pull with
01:08:33.837 --> 01:08:36.827
the exact same team that just functions
really differently.
01:08:36.827 --> 01:08:39.388
I don't know. I I really think though,
01:08:39.388 --> 01:08:43.549
the ~ you know, I am the one person who
does this part.
01:08:43.549 --> 01:08:48.029
I think that is gonna ~ be less.
01:08:48.029 --> 01:08:49.371
I think we're gonna see less of that,
01:08:48.899 --> 01:08:52.528
Yeah. Yep. A hundred percent. Hm.
01:08:49.371 --> 01:08:53.721
right? ~ and if you're really focused.
01:08:52.528 --> 01:08:53.831
Are we all generalizing more
01:08:53.831 --> 01:08:57.198
again? Interesting. It's a
regeneralization almost.
01:08:57.198 --> 01:08:58.198
Yeah. Go ahead.
01:08:58.551 --> 01:09:01.263
Yeah, yeah. I mean, well, one,
01:09:01.263 --> 01:09:04.774
we we already talked about the fact that
like knowing the intricacies
01:09:04.774 --> 01:09:10.477
of the curly braces, right? Like I
remember my my one college course
01:09:10.477 --> 01:09:13.398
and programming, like I was why why?
01:09:13.398 --> 01:09:15.629
Why is there a semicolon on this line and
not this line?
01:09:15.629 --> 01:09:18.229
Like someone make that make sense to me,
01:09:18.229 --> 01:09:21.401
right? Like, no, no kid is worried about
that anymore.
01:09:21.401 --> 01:09:22.962
It's just not a thing, right? Like,
01:09:22.810 --> 01:09:23.810
Yeah, yeah.
01:09:22.962 --> 01:09:23.962
like it just
01:09:23.652 --> 01:09:27.524
it just doesn't happen. And so like that.
01:09:28.416 --> 01:09:33.296
Specific component is dialed back in terms
of the role of an engineer,
01:09:33.296 --> 01:09:35.298
right? So the engineer has a lot more
capacity.
01:09:34.200 --> 01:09:36.224
For sure. For sure.
01:09:35.298 --> 01:09:36.328
It might just be I'm
01:09:36.328 --> 01:09:38.698
going to build more, but it might be I'm
going to think more strategically
01:09:38.698 --> 01:09:40.457
and I'm going to have like these questions
about the customer.
01:09:40.457 --> 01:09:42.220
I'm going to spend that time thinking
about product,
01:09:42.220 --> 01:09:46.531
which maybe up-levels the PM or the PM can
support more teams,
01:09:42.234 --> 01:09:46.069
Something else. Yep. Yep. Mm-hmm.
01:09:46.069 --> 01:09:47.069
Yep.
01:09:46.531 --> 01:09:47.531
or the PM
01:09:47.060 --> 01:09:52.893
writes some code. Like, and a lot of that
is going to be individual aptitudes
01:09:52.893 --> 01:09:55.203
and and skill sets, right? Like,
01:09:53.678 --> 01:09:56.822
Yeah. Interests and rewards and yeah.
01:09:55.203 --> 01:09:56.356
yeah,
01:09:57.064 --> 01:09:58.064
I think.
01:09:57.884 --> 01:10:00.605
You it's interesting because you talked
about like stamping out the teams.
01:10:00.605 --> 01:10:03.686
Right? Well, we know it's exactly looks
like this.
01:10:03.686 --> 01:10:07.847
But I I've always been I don't know about
skeptical,
01:10:07.847 --> 01:10:12.548
but curious about that. Because on any
given team like,
01:10:12.548 --> 01:10:15.439
sure, we could have two front end
developers on this team and two on this
team
01:10:15.439 --> 01:10:18.310
and two on this team. But this one over
here is basically a designer.
01:10:18.310 --> 01:10:22.131
This one over here is great at getting the
pixels that the designer told
01:10:19.866 --> 01:10:23.082
Yeah. Yeah. Yeah, yeah.
01:10:22.131 --> 01:10:24.057
him to get into place, right? So like they
kind
01:10:23.082 --> 01:10:24.082
Yep.
01:10:24.057 --> 01:10:26.832
of look the same, but they don't.
01:10:26.832 --> 01:10:27.832
And like I
01:10:27.749 --> 01:10:30.560
You know, pick your f I watch a lot of
soccer in my family.
01:10:30.560 --> 01:10:32.920
It's World Cup summer, so maybe that's
topical.
01:10:32.920 --> 01:10:36.651
~ NFL football starting, or maybe CFL's
already underway.
01:10:36.651 --> 01:10:37.751
I don't know if you're a CFL, whatever.
01:10:37.562 --> 01:10:38.562
I don't know.
01:10:37.751 --> 01:10:39.492
Pick pick your favorite sport.
01:10:39.492 --> 01:10:40.502
Okay, you a basketball behind
01:10:40.502 --> 01:10:43.113
so maybe but like, right? Well,
01:10:43.113 --> 01:10:44.353
okay, so I live in the Bay Area.
01:10:44.353 --> 01:10:46.263
I guess we kind of invented small ball,
01:10:46.263 --> 01:10:48.514
right? Like everybody had a giant center,
01:10:48.514 --> 01:10:50.764
and then one day we're just like,
01:10:50.764 --> 01:10:51.835
you know what? What if we didn't?
01:10:51.835 --> 01:10:55.075
Right? Like what if what if the structure
of our team,
01:10:55.075 --> 01:10:56.326
right, was
01:10:55.866 --> 01:10:56.866
Yeah.
01:10:56.326 --> 01:10:57.326
such that
01:10:57.246 --> 01:10:58.246
Yeah.
01:10:58.180 --> 01:11:00.322
That isn't actually the best way to play,
01:11:00.322 --> 01:11:05.368
right? Because we we these are individual
humans with individual interactions,
01:11:04.570 --> 01:11:05.793
Yeah. Mm-hmm.
01:11:05.368 --> 01:11:06.368
the way they interact
01:11:06.078 --> 01:11:08.971
with other humans, the skills that they
bring to the table,
01:11:08.971 --> 01:11:10.313
you know, whatever that might be.
01:11:10.313 --> 01:11:13.396
And so, like that, like,
01:11:13.396 --> 01:11:15.258
how do we take full advantage of that?
01:11:15.258 --> 01:11:16.339
Right. And then that becomes a thing,
01:11:16.339 --> 01:11:18.301
and suddenly everyone in the NBA is
throwing threes.
01:11:18.301 --> 01:11:19.301
But like
01:11:18.832 --> 01:11:19.832
Mm-hmm.
01:11:20.223 --> 01:11:21.223
that.
01:11:22.198 --> 01:11:25.849
Is someone right in their particular place
stopping to say,
01:11:25.849 --> 01:11:28.810
like, what are the strengths that I have
at the table right here?
01:11:28.810 --> 01:11:29.810
And so
01:11:29.188 --> 01:11:30.188
Yeah, yeah.
01:11:29.711 --> 01:11:32.852
I think I particularly as we go through
this evolution,
01:11:32.852 --> 01:11:33.852
and I'm just like, I don't know,
01:11:33.622 --> 01:11:37.624
it could be anything. Like we have
multiple different teams at Circle
01:11:37.624 --> 01:11:39.625
CI and they're all shaped differently,
01:11:39.625 --> 01:11:43.396
right? Because that yeah.
01:11:39.642 --> 01:11:40.963
Differently. For sure. No, yeah.
01:11:40.963 --> 01:11:42.756
I I don't mean to overstate that.
01:11:42.756 --> 01:11:43.756
But but
01:11:43.456 --> 01:11:46.220
it but it does feel like that the like you
said,
01:11:46.220 --> 01:11:49.363
~ the whatever we were doing before is not
quite right.
01:11:49.363 --> 01:11:54.529
~ so so maybe maybe my preamble there was
a bit of a you know oversimplification.
01:11:54.529 --> 01:11:57.332
But but but the basis of the question I
think is still interesting,
01:11:57.332 --> 01:11:59.835
you know, which like whatever we were
doing it this feels,
01:11:57.672 --> 01:12:00.444
Yeah, no well, it's super interesting
because I
01:11:59.835 --> 01:12:00.835
yeah.
01:12:00.444 --> 01:12:05.576
think what we love in engineering is
cookie cutter patterns,
01:12:05.576 --> 01:12:06.576
right? We're
01:12:06.000 --> 01:12:07.000
Yeah.
01:12:06.427 --> 01:12:08.858
like, let's, you know, this is the job of
a PM.
01:12:08.858 --> 01:12:12.000
And you're like, okay, but that PM's not
very good at that part of the job.
01:12:11.852 --> 01:12:14.023
No, yeah. Everybody does it their own way.
01:12:12.000 --> 01:12:16.843
But this this other thing, this other this
team member who's like a junior
01:12:14.023 --> 01:12:16.556
Yeah. Yeah.
01:12:16.843 --> 01:12:18.704
engineer is actually really great at that
thing.
01:12:18.704 --> 01:12:21.048
Could they do that instead of the PM doing
it right like?
01:12:21.048 --> 01:12:24.647
And I think it it's it takes a it's taken
me a long time,
01:12:24.647 --> 01:12:26.318
I would say in my in my career.
01:12:26.779 --> 01:12:28.150
To get to the point where I'm like,
01:12:28.150 --> 01:12:34.614
yeah, I'm like, I'm a CTO and other people
are CTOs and they do the job away
01:12:34.634 --> 01:12:38.797
and I do the job differently. And that's
not because one of us is a bad CTO,
01:12:35.992 --> 01:12:37.094
A different way. That's right.
01:12:37.094 --> 01:12:39.466
Yep. No. Yeah.
01:12:38.797 --> 01:12:39.818
but like
01:12:40.158 --> 01:12:43.220
at some point you you realize you're a
member of a team.
01:12:43.220 --> 01:12:45.091
And ultimately
01:12:44.292 --> 01:12:45.292
I love that.
01:12:45.091 --> 01:12:46.562
the goal is the success of the team,
01:12:46.562 --> 01:12:48.584
not your success, right? And
01:12:46.842 --> 01:12:48.575
Yep. Yep.
01:12:48.584 --> 01:12:54.067
so other people on my executive team bring
strengths to the table that I don't have,
01:12:54.067 --> 01:12:55.067
and vice versa.
01:12:55.308 --> 01:12:59.050
And my role might look a little bit
different than the way someone else does
it,
01:12:59.050 --> 01:13:04.253
right? As a CTO, like a very classic
distinction would be someone who's can't
shut
01:13:04.253 --> 01:13:08.796
up, right? Versus someone who like never
wants to talk in front of other humans.
01:13:08.796 --> 01:13:10.687
That's those are two
01:13:09.370 --> 01:13:10.411
Yeah, right?
01:13:10.687 --> 01:13:13.808
totally valid and useful and contributing
CTOs,
01:13:13.808 --> 01:13:14.808
right? And then you just ask
01:13:14.224 --> 01:13:15.224
That's right.
01:13:14.609 --> 01:13:16.300
the question, like, okay, in our
organization,
01:13:16.300 --> 01:13:18.091
who fills in this role and who fills in
this role?
01:13:18.091 --> 01:13:21.673
So I think you know, if if anyone were to
take anything about out
01:13:21.673 --> 01:13:23.704
of this ramble about teams, like,
01:13:24.120 --> 01:13:29.183
Now is a great time to really assess who
on your team brings what to the table.
01:13:29.183 --> 01:13:30.344
And that may be
01:13:29.562 --> 01:13:30.562
Yeah. Yep.
01:13:30.344 --> 01:13:32.125
across multiple teams in your
organization.
01:13:32.125 --> 01:13:37.168
And how can you build a team that's gonna
be successful given
01:13:37.168 --> 01:13:38.749
the constraints of this environment?
01:13:37.456 --> 01:13:39.796
I love it. I love it.
01:13:38.749 --> 01:13:39.960
Right. don't take my
01:13:39.960 --> 01:13:42.071
like it's gonna be two of these and one of
these and one of these.
01:13:42.071 --> 01:13:43.212
Like, don't know.
01:13:43.844 --> 01:13:46.729
Yeah. So we actually have three questions
left,
01:13:46.729 --> 01:13:48.732
~ believe it or not, and
01:13:48.516 --> 01:13:49.516
Look at us.
01:13:49.113 --> 01:13:51.126
and we've come to the end of our time
also.
01:13:51.126 --> 01:13:55.243
~ so I'm gonna I'll call an audible and
and just ask one of them.
01:13:55.243 --> 01:13:58.038
what's something that's keeping you joy
right now?
01:13:59.502 --> 01:14:03.224
Giving me joy. Okay, this is I know you're
from Toronto,
01:14:03.224 --> 01:14:04.494
so hopefully you'll appreciate this.
01:14:04.494 --> 01:14:07.475
This is totally random. ~ but Rush,
01:14:07.475 --> 01:14:11.267
the band from Toronto, right? ~
01:14:10.298 --> 01:14:11.298
Mm-hmm.
01:14:11.267 --> 01:14:14.268
is on tour. I have to confess I have not
gone to see them,
01:14:14.268 --> 01:14:15.839
but I've seen like every YouTube video.
01:14:15.839 --> 01:14:18.770
And ~ for anyone who doesn't know the
band,
01:14:18.770 --> 01:14:20.701
they have this very famous drummer,
01:14:20.701 --> 01:14:23.042
Neil Peart, who like changed drumming,
01:14:23.042 --> 01:14:27.423
right, for the better. And he passed away
in I think twenty sixteen.
01:14:27.423 --> 01:14:28.684
Twenty sixteen, twenty seventeen,
01:14:28.684 --> 01:14:29.684
somewhere there.
01:14:30.052 --> 01:14:31.672
So they're back on tour. Actually,
01:14:31.672 --> 01:14:33.633
it was later, my bad. But whatever.
01:14:33.633 --> 01:14:36.884
You can look it up. They hadn't toured
since 2016.
01:14:36.884 --> 01:14:38.544
And then they found a new drummer,
01:14:38.544 --> 01:14:41.395
Anika Neele, I think is how you pronounce
her name.
01:14:41.395 --> 01:14:44.916
And went out on tour. And the fan,
01:14:44.916 --> 01:14:50.618
it's like the hardest job in the world is
to fill in the seat of this man,
01:14:50.618 --> 01:14:52.218
Neil. And she is
01:14:51.736 --> 01:14:52.736
Mm.
01:14:52.298 --> 01:14:56.849
absolutely crushing it. And the Rush fan
base absolutely loves her.
01:14:56.849 --> 01:14:58.500
And
01:14:57.454 --> 01:14:58.454
nice.
01:14:58.500 --> 01:14:59.500
it is like
01:14:59.726 --> 01:15:01.467
When I first started seeing the videos
rolling,
01:15:01.467 --> 01:15:03.138
because it shows up in my YouTube feed or
whatever,
01:15:03.138 --> 01:15:04.138
I was
01:15:03.631 --> 01:15:04.631
Yeah.
01:15:04.018 --> 01:15:08.070
like, finally something just wholesome and
happy in the world right now.
01:15:07.866 --> 01:15:08.866
Yeah, yeah.
01:15:08.070 --> 01:15:09.070
You know what I mean? Like there's
01:15:08.858 --> 01:15:10.322
there's so much angst out there.
01:15:10.322 --> 01:15:14.064
And like there's like on the internet in
like YouTube comments,
01:15:14.064 --> 01:15:16.505
like it's so easy to be a bad person.
01:15:14.446 --> 01:15:17.751
Yeah. It'd be dark. Yeah.
01:15:16.505 --> 01:15:17.916
And the amount
01:15:17.916 --> 01:15:21.898
of joy generated by something that could
have easily been a very,
01:15:21.898 --> 01:15:26.951
very bad outcome, right? There's all kinds
of sort of skepticism and whatever.
01:15:24.142 --> 01:15:27.630
I love that. Yeah. Sure.
01:15:26.951 --> 01:15:27.951
~
01:15:27.751 --> 01:15:29.432
and it is like
01:15:29.636 --> 01:15:31.517
They're obviously having a huge amount of
fun.
01:15:31.517 --> 01:15:33.128
The fan base is having a lot of fun.
01:15:33.128 --> 01:15:36.329
And you know, when you ask the way that
you asked the question just landed,
01:15:36.329 --> 01:15:38.901
like it's lately I've been saying there's
01:15:38.160 --> 01:15:39.160
Perfect.
01:15:38.901 --> 01:15:42.662
like the one thing that's giving me great
joy right now is just that people
01:15:42.662 --> 01:15:45.064
are so happy about this about this
situation.
01:15:45.064 --> 01:15:46.064
So yeah, it's awesome.
01:15:45.252 --> 01:15:46.252
I love it.
01:15:46.015 --> 01:15:47.469
I love it. Okay. One more thing.
01:15:47.469 --> 01:15:49.666
~ you mentioned earlier the name of the
newsletter.
01:15:49.666 --> 01:15:52.355
so let's hear it one more time and I'll
make sure it's in the show notes.
01:15:52.758 --> 01:15:56.932
~ our the state of software delivery
report ~ is a thing that we publish
01:15:56.932 --> 01:15:59.435
and then we have a newsletter actually
called the confident commit
01:15:59.435 --> 01:16:02.328
and a podcast called the confident commit
is like a umbrella for some
01:16:02.328 --> 01:16:03.499
of the other material that we put out.
01:16:03.499 --> 01:16:04.499
So you can look for
01:16:03.952 --> 01:16:04.952
Cool.
01:16:04.310 --> 01:16:05.310
too.
01:16:04.856 --> 01:16:07.010
Cool. Rob, thank you so much for coming on
the show today.
01:16:07.010 --> 01:16:08.010
Really appreciate you.
01:16:08.483 --> 01:16:09.916
Yeah, you bet, Kyle. It's really fun.