สำเนาบทสนทนา 🔗
WEBVTT
00:00:00.000 --> 00:00:03.120
If more than one person is responsible for feeding
00:00:03.120 --> 00:00:06.900
the cat, the cat starves. My guest today used
00:00:06.900 --> 00:00:10.640
that line for a question a lot of teams are dodging
00:00:10.640 --> 00:00:14.960
right now. Who owns AI-generated code once it's
00:00:14.960 --> 00:00:18.660
running in production? His company surveyed engineering
00:00:18.660 --> 00:00:22.460
teams. Almost half already run AI-generated
00:00:22.460 --> 00:00:26.280
code in production. And about a third write it
00:00:26.280 --> 00:00:30.210
but won't ship it. He says, Code review is only
00:00:30.210 --> 00:00:34.689
act two. The bigger bill comes later, when somebody
00:00:34.689 --> 00:00:38.130
has to understand code that nobody really wrote.
00:00:38.450 --> 00:00:41.929
And the leaders who come out ahead won't be the
00:00:41.929 --> 00:00:45.969
no squad. They'll be the yes squad, with guardrails.
00:00:46.149 --> 00:00:48.770
I'm Brian Teller from Teller's Tech, and this
00:00:48.770 --> 00:00:50.431
is Ship It Weekly.
00:01:06.930 --> 00:01:08.569
Welcome back to Ship It Weekly,
00:01:08.769 --> 00:01:12.010
where I filter the noise and focus on what matters
00:01:12.390 --> 00:01:14.790
when you're the one running infrastructure and
00:01:14.790 --> 00:01:18.189
owning reliability. Most weeks, it's a quick
00:01:18.189 --> 00:01:22.829
DevOps, SRE, platform, cloud, and security news
00:01:22.829 --> 00:01:26.209
recap. In between those, I do conversation episodes
00:01:26.209 --> 00:01:29.150
with people building and operating these systems.
00:01:29.750 --> 00:01:33.230
Ship It Weekly turns one this Friday. To celebrate
00:01:33.230 --> 00:01:35.969
through mid-December, you are getting two conversations
00:01:35.969 --> 00:01:39.290
a week, Mondays and Wednesdays, with the news
00:01:39.290 --> 00:01:42.359
every Friday. Then... we're back to our usual
00:01:42.359 --> 00:01:46.459
rhythm. Today, I'm joined by Ted Julian, founder
00:01:46.459 --> 00:01:49.980
and CEO of Flux, an engineering intelligence
00:01:49.980 --> 00:01:53.799
platform. Ted has been building security and
00:01:53.799 --> 00:01:56.939
infrastructure companies for more than 20 years,
00:01:57.099 --> 00:02:00.500
including Resilient Systems, which IBM acquired,
00:02:00.900 --> 00:02:05.000
Arbor Networks, and @stake. Earlier this year,
00:02:05.180 --> 00:02:09.159
Flux put out a report on the gap between AI-generated
00:02:09.159 --> 00:02:12.500
code getting written and teams actually trusting
00:02:12.500 --> 00:02:16.560
what it changes. We talk about why so much AI
00:02:16.560 --> 00:02:20.000
-generated code gets written but never shipped,
00:02:20.199 --> 00:02:23.740
why code review is the first thing to break,
00:02:23.919 --> 00:02:27.919
a simple way to measure what AI is actually doing
00:02:27.919 --> 00:02:31.759
for your team, who should own AI-generated code
00:02:31.759 --> 00:02:35.330
in production, and why he thinks this is the
00:02:35.330 --> 00:02:38.969
best career opportunity engineers will ever get.
00:02:39.169 --> 00:02:41.149
Alright, let's jump in.
00:02:45.798 --> 00:02:47.368
Ted, thank you for joining me.
00:02:47.368 --> 00:02:49.541
It's a pleasure. Thanks for the invitation.
00:02:49.541 --> 00:02:53.217
So I've read the report. I'm interested. Give me your thesis.
00:02:53.566 --> 00:02:59.076
AI-generated code is clearly being used now, but what are teams still not ready for?
00:02:59.425 --> 00:03:04.689
I think the surprise to me was to see the the gap between the code
00:03:04.689 --> 00:03:07.050
generation and the code actually in production.
00:03:07.449 --> 00:03:11.110
Okay. And then what the survey users did tell
00:03:11.110 --> 00:03:13.069
us, like, it's like every survey, you kind of
00:03:13.069 --> 00:03:15.210
want to do it again, because you get answers
00:03:15.210 --> 00:03:16.729
to questions, and then you want to know, oh,
00:03:16.750 --> 00:03:19.270
wow, it'd be great to know more about that. But
00:03:19.270 --> 00:03:21.030
anyway, they did give us some examples of like,
00:03:21.069 --> 00:03:23.169
what was holding them back in terms of security
00:03:23.169 --> 00:03:24.870
concerns and other things that I'm sure we'll
00:03:24.870 --> 00:03:28.439
talk about. Okay, so high level, what are some
00:03:28.439 --> 00:03:32.669
concerns that you've seen? For sure, it's There's
00:03:32.669 --> 00:03:37.449
a lack of trust, especially in more enterprise
00:03:37.449 --> 00:03:41.210
class, more regulated environments, about taking
00:03:41.210 --> 00:03:43.729
this generated code that maybe was for a prototype
00:03:43.729 --> 00:03:46.669
and it went super quick and very excited, legitimately
00:03:46.669 --> 00:03:49.370
excited about that. But when you pause to think
00:03:49.370 --> 00:03:50.930
about putting it in production, if it's like
00:03:50.930 --> 00:03:53.969
a HIPAA environment or if it's a highly sensitive,
00:03:54.150 --> 00:03:57.550
like every minute costs us some huge amount of
00:03:57.550 --> 00:04:01.620
dollars environment around the reliability of
00:04:01.620 --> 00:04:04.439
that code broadly defined and then they did enumerate
00:04:04.439 --> 00:04:07.379
some specific concerns whether it was complexity
00:04:07.379 --> 00:04:09.599
that might be spiraling out of control and could
00:04:09.599 --> 00:04:12.759
lead to an outage or the introduction of a new
00:04:12.759 --> 00:04:15.419
security vulnerability based on a library that
00:04:15.419 --> 00:04:18.360
was put in you know a pull request to to sort
00:04:18.360 --> 00:04:20.120
of generate this prototype that we're now going
00:04:20.120 --> 00:04:25.350
to make real um or other concerns around regulated
00:04:25.350 --> 00:04:28.509
data? And has this code really been scrutinized
00:04:28.509 --> 00:04:30.990
sufficiently for how it's handling this data
00:04:30.990 --> 00:04:32.430
that looked great in the
Load full transcript
prototype? But again,
00:04:32.569 --> 00:04:34.610
we're a regulated healthcare company, and now
00:04:34.610 --> 00:04:37.029
this is winding up in production. Yeah, for sure.
00:04:37.230 --> 00:04:41.930
Your report says 44.7% have AI-generated code
00:04:41.930 --> 00:04:45.970
in production, and 35% are writing it but not
00:04:45.970 --> 00:04:48.879
shipping it. What does that tell you? This is
00:04:48.879 --> 00:04:51.620
an example where I'd love to do the survey, to
00:04:51.620 --> 00:04:54.620
like really double click on that, because I just
00:04:54.620 --> 00:04:57.759
talked about some of the hesitations, but I sure
00:04:57.759 --> 00:05:00.540
would love to know more about them. And there's
00:05:00.540 --> 00:05:02.740
some things we didn't ask that I suspect honestly
00:05:02.740 --> 00:05:06.680
could be behind that too. So for example, if
00:05:06.740 --> 00:05:08.399
you think about the time in which the survey
00:05:08.399 --> 00:05:09.740
was done, there were probably a lot of people
00:05:09.740 --> 00:05:12.920
that were still in the early era of experimentation,
00:05:13.360 --> 00:05:15.399
right? So it literally could have been like some
00:05:15.399 --> 00:05:17.100
throwaway something, but you know what? It was
00:05:17.100 --> 00:05:19.850
a chance for us to, Try Claude because we haven't
00:05:19.850 --> 00:05:23.029
done that yet. And, you know, it was just a little
00:05:23.029 --> 00:05:24.629
prototype. Maybe you turn into something, maybe
00:05:24.629 --> 00:05:26.790
not. So honestly, like just things like that,
00:05:26.829 --> 00:05:30.529
just tests, proof of concepts could easily explain
00:05:30.529 --> 00:05:33.949
some percentage of that. But we do know based
00:05:33.949 --> 00:05:37.009
on what they did tell us that it that is was
00:05:37.009 --> 00:05:39.769
not all that was going on here. You know, there
00:05:39.769 --> 00:05:41.870
were those other concerns that we alluded to
00:05:41.870 --> 00:05:44.170
that were the things that were giving them pause.
00:05:45.550 --> 00:05:48.980
And then also just highlighted a general visibility
00:05:48.980 --> 00:05:52.600
need, I think it's very safe to say, around what's
00:05:52.600 --> 00:05:54.800
happening in our teams as they're embracing these
00:05:54.800 --> 00:05:57.579
tools that would be required to sort of close
00:05:57.579 --> 00:06:01.079
that gap and increase that percentage. Yeah,
00:06:01.139 --> 00:06:04.100
that's for sure. So do you think there's like
00:06:04.100 --> 00:06:06.740
a code quality problem? Or do you think it's
00:06:06.740 --> 00:06:10.139
more like operational model problem? There is
00:06:10.139 --> 00:06:12.620
a code quality problem. I mean, we could use
00:06:12.620 --> 00:06:14.939
something like the Amazon outage from it was
00:06:14.939 --> 00:06:17.439
probably six months, eight months ago now. Right.
00:06:17.500 --> 00:06:18.600
And that was kind of a shot heard around the
00:06:18.600 --> 00:06:19.819
world because that was the one where there's
00:06:19.819 --> 00:06:22.740
the memo that leaked that basically said you're
00:06:22.740 --> 00:06:26.449
going to sign off on all PRs, which I don't know
00:06:26.449 --> 00:06:28.069
how that's going to work in practice with the
00:06:28.069 --> 00:06:30.610
team that big, but they're smarter than me. But
00:06:30.610 --> 00:06:33.389
anyway, like we're joking about it, but an outage
00:06:33.389 --> 00:06:35.189
at a place like Amazon is a really big deal.
00:06:35.509 --> 00:06:37.689
And for somebody that is so focused on moving
00:06:37.689 --> 00:06:40.430
fast like Amazon is to put that kind of countermeasure
00:06:40.430 --> 00:06:43.370
in place goes to show you that this is not theoretical,
00:06:43.569 --> 00:06:47.379
that it is happening, but it's early. I think
00:06:47.379 --> 00:06:49.459
in some respects you could say that's one of
00:06:49.459 --> 00:06:51.420
the early canaries in the coal mine, and it's
00:06:51.420 --> 00:06:55.220
probably a function of leaning in hard and leaning
00:06:55.220 --> 00:06:58.279
in early. But it's a cautionary tale for the
00:06:58.279 --> 00:07:00.759
rest of us that the models are the worst they
00:07:00.759 --> 00:07:03.360
will ever be. In other words, they're going to
00:07:03.360 --> 00:07:05.819
get better. And so maybe this problem isn't quite
00:07:05.819 --> 00:07:07.579
as bad for those that are coming along behind
00:07:07.579 --> 00:07:10.740
it. But that the more you lean in and the more
00:07:10.740 --> 00:07:16.379
that you are regulated, care about... five nines
00:07:16.379 --> 00:07:19.420
and so on, then the more that the smart leaders
00:07:19.420 --> 00:07:21.959
are going to be putting guardrails, looking to
00:07:21.959 --> 00:07:24.199
get that visibility to avoid those kinds of things
00:07:24.199 --> 00:07:28.079
happening to them. Yeah. For me, I think of the
00:07:28.079 --> 00:07:32.579
review side of things. So if AI makes code faster
00:07:32.579 --> 00:07:36.000
to produce, like what breaks first in the review
00:07:36.000 --> 00:07:39.259
and release process, right? 100%. We see this
00:07:39.259 --> 00:07:41.300
at all our customers and we're starting to build
00:07:41.300 --> 00:07:43.910
kind of a journey map for organizations as they
00:07:43.910 --> 00:07:46.889
go through their AI transformation. And without
00:07:46.889 --> 00:07:49.949
a doubt, that's kind of act two. And it's like
00:07:49.949 --> 00:07:52.870
any good play. Act one, you're introducing characters
00:07:52.870 --> 00:07:55.209
and there's all sorts of promise and things start
00:07:55.209 --> 00:07:57.110
to look really interesting. But act two, that's
00:07:57.110 --> 00:08:00.050
when some things happen. There's a villain or
00:08:00.050 --> 00:08:01.970
there's a tragedy or something like that because
00:08:01.970 --> 00:08:03.670
it makes it a great story. It'd be boring otherwise.
00:08:04.389 --> 00:08:06.370
But I'd say this is maybe the beginning of act
00:08:06.370 --> 00:08:10.500
two where, yeah, code review becomes... a real
00:08:10.500 --> 00:08:12.399
problem. And what I think is interesting about
00:08:12.399 --> 00:08:15.120
that is it gets into the people dynamics, right?
00:08:15.500 --> 00:08:18.899
Where this isn't just a technical bottleneck.
00:08:18.920 --> 00:08:20.540
And so we're going to throw more computer storage
00:08:20.540 --> 00:08:22.699
or something at it. And I think in this case,
00:08:22.740 --> 00:08:25.319
it's also not as easy as just using AI to do
00:08:25.319 --> 00:08:27.000
code review, which is probably a very, very good
00:08:27.000 --> 00:08:30.139
idea. But if you use Amazon as an example, whether
00:08:30.139 --> 00:08:32.019
or not that's the right conclusion, I think there
00:08:32.019 --> 00:08:34.179
still has to be humans in the loop when we're
00:08:34.179 --> 00:08:35.759
talking about really sensitive environments.
00:08:35.899 --> 00:08:40.049
So it isn't a simple, it isn't just More tech,
00:08:40.210 --> 00:08:44.210
different tech. And so now there's a different
00:08:44.210 --> 00:08:47.590
way of... It's more important to look at the
00:08:47.590 --> 00:08:49.669
team dynamics and measure those team dynamics
00:08:49.669 --> 00:08:52.590
because progressive improvement is going to require
00:08:52.590 --> 00:08:57.269
solving that problem around a people. And whether
00:08:57.269 --> 00:08:59.289
that's more people or different people or swapping
00:08:59.289 --> 00:09:01.710
them around or whatever, until you solve that
00:09:01.710 --> 00:09:03.669
problem, that continuous improvement will be
00:09:03.669 --> 00:09:08.059
gated on that. How do we measure the... efficiency
00:09:08.059 --> 00:09:11.919
gains of AI? Is it bigger PRs, more frequent
00:09:11.919 --> 00:09:14.860
PRs? I mean, that's what everyone likes to use
00:09:14.860 --> 00:09:18.600
as metrics. There is a once in a career hack
00:09:18.600 --> 00:09:26.320
available to you now, which is to just do, this
00:09:26.320 --> 00:09:29.539
is what we see smart customers do. They sort
00:09:29.539 --> 00:09:33.250
of start to measure, they find a way to measure
00:09:33.250 --> 00:09:35.570
features versus maintenance versus bug fixes,
00:09:35.629 --> 00:09:39.509
this type of thing. Flux does this, but there's
00:09:39.509 --> 00:09:40.950
lots of ways you could do this. You could tag
00:09:40.950 --> 00:09:44.450
it in Jira. You could tag things in GitHub. You
00:09:44.450 --> 00:09:47.129
could use some other tool, some way or another.
00:09:47.549 --> 00:09:50.149
Try to find a more programmatic way of getting
00:09:50.149 --> 00:09:52.340
this measurement than... What maybe people have
00:09:52.340 --> 00:09:54.440
done in the past, which often is for R&D tax
00:09:54.440 --> 00:09:57.500
credits, and it's sort of at the end of the quarter,
00:09:57.600 --> 00:09:59.620
and you look at the team, and the leader will
00:09:59.620 --> 00:10:01.940
be like, yeah, looking at GitHub, I'd say 30
00:10:01.940 --> 00:10:03.820
% were features, and so that's what we'll put
00:10:03.820 --> 00:10:06.059
in the spreadsheet, and that determines R&D
00:10:06.059 --> 00:10:08.159
tax credits. Something that is a little bit more
00:10:08.159 --> 00:10:11.919
real-time to try to get that measurement. Anyway,
00:10:12.080 --> 00:10:14.620
so you backfill, and you define it as before,
00:10:14.879 --> 00:10:17.700
right? So as of, let's just say, the beginning
00:10:17.700 --> 00:10:21.070
of this calendar year. We've backfilled our PRs
00:10:21.070 --> 00:10:23.389
and we know that we believe that our features
00:10:23.389 --> 00:10:27.259
as a percentage of all engineering was 10%. Now,
00:10:27.379 --> 00:10:30.700
if you're even remotely like leaning into code gen
00:10:30.700 --> 00:10:32.799
and maybe you're migrating from GitHub Copilot
00:10:32.799 --> 00:10:34.720
to Claude or you're rolling it out across more
00:10:34.720 --> 00:10:38.320
teams, I mean, that should grow quite a bit.
00:10:38.779 --> 00:10:41.700
Because if you simply say that the number of
00:10:41.700 --> 00:10:43.659
the amount of code that we generate, let's just
00:10:43.659 --> 00:10:46.620
keep the number simple, goes up by 10x. Well,
00:10:46.659 --> 00:10:48.740
then probably the features go up by 10x or something
00:10:48.740 --> 00:10:51.460
like this. Right. And so now you've got this
00:10:51.460 --> 00:10:53.960
before and you got this after, which not only
00:10:53.960 --> 00:10:56.759
shows velocity improving, but it shows features
00:10:56.759 --> 00:10:59.399
in particular growing by, let's say, 5x. Keep
00:10:59.399 --> 00:11:03.120
it that simple. I can tell you based on our customers,
00:11:03.240 --> 00:11:06.799
like that one customer that became a board chart,
00:11:07.250 --> 00:11:10.710
like their Q1 board meeting. That was a chart
00:11:10.710 --> 00:11:12.570
in the board meeting. And I know somebody in
00:11:12.570 --> 00:11:14.490
the meeting and the board was thrilled. They're
00:11:14.490 --> 00:11:17.340
like. We were pushing you guys so hard last year
00:11:17.340 --> 00:11:20.120
to lean in the AI. We thought the team was kind
00:11:20.120 --> 00:11:24.039
of stagnant. They went from 9% to 43% by the
00:11:24.039 --> 00:11:25.940
end of last year. So they were like, this is
00:11:25.940 --> 00:11:28.980
tremendous. So anyway, I don't care how you do
00:11:28.980 --> 00:11:32.559
it. This is kind of a once-in-a-lifetime career
00:11:32.559 --> 00:11:34.159
hack. We're not going to have the curve like
00:11:34.159 --> 00:11:37.309
it is now, right? it's only going to flatten
00:11:37.309 --> 00:11:39.889
a little over time. So you're kind of crazy not
00:11:39.889 --> 00:11:42.129
to avail yourself if you are leaning in the code
00:11:42.129 --> 00:11:45.590
gen to document this spike in feature delivery
00:11:45.590 --> 00:11:48.450
to both give the team the credit that they're
00:11:48.450 -->
00:11:50.629
due for driving this transformation because let's
00:11:50.629 --> 00:11:52.629
face it, like it's a little scary and it's a
00:11:52.629 --> 00:11:55.620
new tech. And so people have to feel comfortable
00:11:55.620 --> 00:11:57.539
about that, what that means for their careers,
00:11:57.620 --> 00:11:59.299
their jobs, that kind of thing. So you absolutely
00:11:59.299 --> 00:12:02.600
want to ground this in data and to give them
00:12:02.600 --> 00:12:04.779
the well-deserved pat on the back. But at the
00:12:04.779 --> 00:12:06.600
same time, you'd be crazy not to leverage that
00:12:06.600 --> 00:12:08.600
with senior management to show like, hey, we're
00:12:08.600 --> 00:12:10.379
doing it. We're doing exactly what you asked
00:12:10.379 --> 00:12:12.419
us to do. And look at these results. They're
00:12:12.419 --> 00:12:15.639
tremendous. Okay, so shifting gears a little
00:12:15.639 --> 00:12:18.850
bit. Let's say we figured out how our AI efficiency
00:12:18.850 --> 00:12:21.370
gains are. We've nailed all that down. We've
00:12:21.370 --> 00:12:24.289
nailed down the FinOps side of it. How do we
00:12:24.289 --> 00:12:27.730
deal with the security side? Your report calls
00:12:27.730 --> 00:12:30.409
out security dependency changes and performance
00:12:30.409 --> 00:12:34.230
impact as some of the hardest week-to-week
00:12:34.230 --> 00:12:37.620
changes to detect. Those are exactly the things
00:12:37.620 --> 00:12:40.080
that become incidents, at least in what I've
00:12:40.080 --> 00:12:42.799
been through. Why are those so easy to miss?
00:12:43.389 --> 00:12:45.789
I mean, we'll start with the less scary example,
00:12:45.950 --> 00:12:49.429
which is simply you got teams leaning really
00:12:49.429 --> 00:12:51.830
hard into generated code because there's an enormous
00:12:51.830 --> 00:12:53.429
amount of pressure and they want to do the right
00:12:53.429 --> 00:12:55.830
thing. And so they're prototyping stuff and it's
00:12:55.830 --> 00:13:00.809
going great. And it is so easy to just keep,
00:13:00.929 --> 00:13:04.269
you know, tab, tab, tab, you know, or accept,
00:13:04.409 --> 00:13:07.149
accept, accept. And if you're not careful, like
00:13:07.149 --> 00:13:09.750
there's a new library and that new library has
00:13:09.750 --> 00:13:12.750
a vulnerability. Thing works. It doesn't create
00:13:12.750 --> 00:13:16.210
a functional issue. And that winds up into production.
00:13:16.470 --> 00:13:19.210
I mean, this is absolutely a very real concern
00:13:19.210 --> 00:13:22.409
without some guardrails. It's just human nature.
00:13:22.789 --> 00:13:25.610
So I don't think it's sufficient to say, oh,
00:13:25.669 --> 00:13:28.809
we're just going to have really experienced developers
00:13:28.809 --> 00:13:30.110
and they're going to know better. I mean, of
00:13:30.110 --> 00:13:31.470
course, you want to have that as part of the
00:13:31.470 --> 00:13:34.049
best practice. But it's belt and suspenders time.
00:13:34.169 --> 00:13:35.690
I mean, there have to be some other guardrails
00:13:35.690 --> 00:13:37.490
on this process or else you're just asking for
00:13:37.490 --> 00:13:41.080
trouble. Now that all assumes the traditional
00:13:41.080 --> 00:13:44.759
dev team, you know, but operating in a new way.
00:13:44.840 --> 00:13:47.740
But you and I know, like, that's not the new
00:13:47.740 --> 00:13:49.899
reality. The new reality is everybody's coding,
00:13:50.059 --> 00:13:54.220
right? Marketing coding, sales is coding, supports
00:13:54.220 --> 00:13:58.059
coding. And it's actually great, you know, managed
00:13:58.059 --> 00:14:00.220
properly. So then the last thing you as an engineering
00:14:00.220 --> 00:14:02.840
team was be the no squad and like shutting all
00:14:02.840 --> 00:14:04.639
this down. I mean, you want to be the yes squad,
00:14:04.840 --> 00:14:07.720
but... You know, if you think it's easy for a
00:14:07.720 --> 00:14:09.720
developer to miss that, like a marketing person
00:14:09.720 --> 00:14:11.519
with all the best intentions in the world trying
00:14:11.519 --> 00:14:15.100
to do the right thing, like forget it. So it's
00:14:15.100 --> 00:14:17.139
you can't have one without the other, you know,
00:14:17.139 --> 00:14:19.080
like you can't say we're going to lean really
00:14:19.080 --> 00:14:21.139
hard into this organization wide. It's not just
00:14:21.139 --> 00:14:23.039
the engineering team. We're encouraging everybody
00:14:23.039 --> 00:14:26.639
to lean into this opportunity to maximize productivity
00:14:26.639 --> 00:14:29.500
gain and to innovate and to have fun and all
00:14:29.500 --> 00:14:32.039
those fabulous things while not acknowledging
00:14:32.039 --> 00:14:34.100
that these are absolutely side effects that come
00:14:34.100 --> 00:14:36.059
with that. You can't have the one without the
00:14:36.059 --> 00:14:38.379
other. And so the smart leaders are the ones
00:14:38.379 --> 00:14:41.820
that are going to be the, not the no squad, they're
00:14:41.820 --> 00:14:43.720
going to be the yes squad, but they're going
00:14:43.720 --> 00:14:46.740
to find ways to put those guardrails in to architect
00:14:46.740 --> 00:14:50.649
processes and so forth that help protect the
00:14:50.649 --> 00:14:53.950
organization. Yeah. Where do you think those
00:14:53.950 --> 00:14:56.830
performance regressions hide? when changes are
00:14:56.830 --> 00:15:01.750
coming faster? We talked about code review, where
00:15:01.750 --> 00:15:05.450
I think we're early is in, it's basically dealing
00:15:05.450 --> 00:15:08.149
with the tech debt and, and dealing with the
00:15:08.149 --> 00:15:11.009
complexity. And I think it's only the early adopters
00:15:11.009 --> 00:15:13.210
that are living this on a day-to-day basis.
00:15:13.210 --> 00:15:15.710
I think this is a bill that's going to come due
00:15:15.710 --> 00:15:18.929
for maybe, like, the rest of us over the next year,
00:15:18.929 -->
00:15:21.590
18 months, depending on where you are in your
00:15:21.590 --> 00:15:25.590
journey. Um, but this is inevitable, and I do think
00:15:25.590 --> 00:15:29.309
it's, it's going to be, uh, very much an eye-opener,
00:15:29.309 --> 00:15:32.070
like the code review problem or that bottleneck
00:15:32.070 --> 00:15:34.389
will look like a small problem than when this
00:15:35.110 --> 00:15:38.289
Code complexity, tech debt issue starts to rear
00:15:38.289 --> 00:15:41.389
its head because that will be the first time
00:15:41.389 --> 00:15:44.090
that we really have to figure the code out, if
00:15:44.090 --> 00:15:47.210
I'm honest, right? And there's going to be no
00:15:47.210 --> 00:15:49.190
choice. It's not going to be just click accept
00:15:49.190 --> 00:15:52.909
that looks reasonably okay. Somebody's going
00:15:52.909 --> 00:15:55.929
to have to understand that code in a detailed
00:15:55.929 --> 00:15:58.590
fashion and nobody really wrote it. It was generated.
00:15:59.129 --> 00:16:02.190
And God knows, like, it was outsourced and we
00:16:02.190 --> 00:16:06.509
dropped them. We're on another thing. Yeah, Sally
00:16:06.509 --> 00:16:08.289
was amazing, but she found a great opportunity.
00:16:08.409 --> 00:16:13.460
She's not here anymore. So, Joe, good luck. Yeah,
00:16:13.539 --> 00:16:16.220
and it's not documented anywhere. No, no, no.
00:16:16.259 --> 00:16:18.799
And again, the tooling's all going to get better.
00:16:18.980 --> 00:16:22.299
None of this is cast in stone, right? And so,
00:16:22.419 --> 00:16:25.940
yeah, there will be AI-driven improvements in
00:16:25.940 --> 00:16:27.500
these various areas that are probably going to
00:16:27.500 --> 00:16:28.899
make it better. But I don't think at the end
00:16:28.899 --> 00:16:32.779
of the day, it's not a smart career decision
00:16:32.779 --> 00:16:35.759
to say that it'll be fine. I think, again, the
00:16:35.759 --> 00:16:37.200
smart leaders are going to be the ones that are
00:16:37.200 --> 00:16:39.080
putting some guardrails, thinking about process
00:16:39.080 --> 00:16:42.600
that will avert these problems is the one side
00:16:42.600 --> 00:16:44.139
of the coin. And I guess the other side of the
00:16:44.139 --> 00:16:46.940
coin is proactively identifying the bottlenecks
00:16:46.940 --> 00:16:48.700
and getting out ahead of them so that they continue
00:16:48.700 --> 00:16:52.960
to go through the various phases of their journey
00:16:52.960 --> 00:16:55.120
and they're unlocking improvements at every step
00:16:55.120 --> 00:17:06.130
of the way. OK, we need to take this seriously,
00:17:06.130 --> 00:17:07.849
especially as we're moving quicker. Right. It
00:17:07.849 --> 00:17:10.829
matters even more. Right. Please correct me because
00:17:10.829 --> 00:17:13.529
I'm coming. I bring a cyber background to this
00:17:13.529 --> 00:17:15.470
and a cyber point of view. And so what I'm about
00:17:15.470 --> 00:17:19.970
to say may be wrong, may be
offensive. So I'd
00:17:19.970 --> 00:17:21.950
encourage to be challenged. But this to me is
00:17:21.950 --> 00:17:24.609
very much a security mindset. I mean, security
00:17:24.609 --> 00:17:28.069
for effectively forever has had this dilemma
00:17:28.069 --> 00:17:33.289
of we're the no squad. Right. You name it. Commercialization
00:17:33.289 --> 00:17:36.690
of the Internet, mobile, cloud, whatever, every
00:17:36.690 --> 00:17:39.730
step of the way, security has to walk this tightrope
00:17:39.730 --> 00:17:44.369
between, you know, managing risk, but also you
00:17:44.369 --> 00:17:46.930
cannot be the no squad. Like if you just are
00:17:46.930 --> 00:17:48.569
saying no all the time, like they're going to
00:17:48.569 --> 00:17:51.190
find somebody else who has a way of saying, totally.
00:17:51.519 --> 00:17:55.160
That's all that stuff. Yes. Well put. Right.
00:17:55.279 --> 00:17:57.740
And I just, I'm just not sure engineering's quite
00:17:57.740 --> 00:18:01.500
been in this position before to the same level.
00:18:01.640 --> 00:18:05.309
And maybe to me, this is, It's part of this unprecedented
00:18:05.789 --> 00:18:10.230
territory we're in right now. I mean, engineering
00:18:10.230 --> 00:18:12.130
is being upended completely for the first time
00:18:12.130 --> 00:18:15.210
ever. It's never happened. It happened with sales,
00:18:15.349 --> 00:18:17.690
like with Salesforce. It happened with marketing,
00:18:17.789 --> 00:18:20.150
with HubSpot, where those processes digitized.
00:18:20.650 --> 00:18:22.849
And that was really painful and shook things
00:18:22.849 --> 00:18:24.549
up, but it also was an enormous opportunity.
00:18:25.049 --> 00:18:27.130
And I think that's at a very, very high level.
00:18:27.190 --> 00:18:28.390
That's kind of where we are with engineering
00:18:28.390 --> 00:18:30.170
right now. This whole thing is being completely
00:18:30.170 --> 00:18:33.230
upended. And yes, it's absolutely fabulous, but
00:18:33.230 --> 00:18:36.250
it's also scary and risky. And it's like I said,
00:18:36.289 --> 00:18:40.470
you can't have one without the other. Yeah. Okay,
00:18:40.569 --> 00:18:42.990
so let's stay, I guess, on that thread a little
00:18:42.990 --> 00:18:46.470
bit with security and the compliance legal side
00:18:46.470 --> 00:18:50.230
of it or engineering leadership side. One thing
00:18:50.230 --> 00:18:51.950
that stood out to me is that this is not just
00:18:51.950 --> 00:18:54.609
an engineering concern anymore. Security, compliance,
00:18:54.950 --> 00:18:59.309
CTO, CIO and legal all showing concern. What
00:18:59.309 --> 00:19:02.170
changed? I mean, we've seen this movie before.
00:19:02.309 --> 00:19:08.150
This is what always happens. Again, I kind of
00:19:08.150 --> 00:19:10.009
just talked about the big transformations, right?
00:19:10.130 --> 00:19:11.569
Like this happened with the commercialization
00:19:11.569 --> 00:19:14.349
of the Internet, right? Like, just get the website
00:19:14.349 --> 00:19:17.660
up. For God's sakes, you know, X, Y, Z competitors
00:19:17.660 --> 00:19:20.359
are all doing it. We need a website. And everybody
00:19:20.359 --> 00:19:22.259
in IT or in the security department is like,
00:19:22.319 --> 00:19:25.960
oh, hold on a second. Like, you're taking a network
00:19:25.960 --> 00:19:28.500
that has never been connected to anything outside.
00:19:29.180 --> 00:19:31.019
You know, it's just been completely internal.
00:19:31.400 --> 00:19:33.200
And now you're connecting it literally to the
00:19:33.200 --> 00:19:36.700
entire world. Like, there's some things we need
00:19:36.700 --> 00:19:41.279
to do to be smart about how we do this. Ditto
00:19:41.279 --> 00:19:43.839
cloud, ditto mobile. So I think that's the same
00:19:43.839 --> 00:19:46.960
thing is that the organizations have leaned so
00:19:46.960 --> 00:19:49.660
hard into this AI transformation, but eventually
00:19:49.660 --> 00:19:52.480
these other functions realize that it is their
00:19:52.480 --> 00:19:55.759
job to force the organization to be smart about
00:19:55.759 --> 00:19:59.019
these kinds of things. And so that's why you
00:19:59.019 --> 00:20:01.259
kind of go through these waves of like this initial
00:20:01.259 --> 00:20:05.079
elation, but then things cool a little bit as
00:20:05.079 --> 00:20:06.839
all these other functions weigh in and figure
00:20:06.839 --> 00:20:10.119
out how to do this in a responsible way. So that's
00:20:10.119 --> 00:20:13.000
like we've seen this movie before. And this is
00:20:13.000 --> 00:20:15.859
maybe, again, the early part of act two, where
00:20:15.859 --> 00:20:20.400
these teams sort of go from latent concern to
00:20:20.400 --> 00:20:23.940
tooling up and figuring out like, this is how
00:20:23.940 --> 00:20:27.140
we believe the organization needs to tackle this,
00:20:27.279 --> 00:20:29.940
what the procedures are, they discover maybe
00:20:29.940 --> 00:20:33.250
some tooling to help. So who owns the behavior
00:20:33.250 --> 00:20:36.009
of AI generated code and production then? Is
00:20:36.009 --> 00:20:40.069
it the engineers? Is it the... CIO? Is it the
00:20:40.069 --> 00:20:43.730
AI director? What level? I mean, I guess everybody
00:20:43.730 --> 00:20:47.390
does to an extent, but like at a traditional company,
00:20:47.390 --> 00:20:50.529
who owns that? I don't know that that's been discerned.
00:20:50.529 --> 00:20:52.730
I can't say that, like, I don't think we can canonically
00:20:52.730 --> 00:21:00.440
say, like, the universe has voted, 62% say.
00:21:00.630 --> 00:21:05.339
But, like, it's who feeds the cat, right? If multiple
00:21:05.339 --> 00:21:07.359
people feed the cat, the cat starves. So it's
00:21:07.359 --> 00:21:09.359
got to be one person that's responsible for feeding
00:21:09.359 --> 00:21:12.279
the cat. And I think it's got to be the head
00:21:12.279 --> 00:21:15.259
of engineering and ultimately the CIO. I mean,
00:21:15.279 --> 00:21:17.140
it's the code. Let me ask it a different way
00:21:17.140 --> 00:21:19.859
then. What does governance look like without
00:21:19.859 --> 00:21:24.640
crushing velocity? That is the $60 million question.
00:21:25.539 --> 00:21:29.380
No, I mean, it always is. Yeah. And I don't think
00:21:29.380 --> 00:21:31.559
there's any easy answer because I think every
00:21:31.559 --> 00:21:35.380
organization is different. And it's a function
00:21:35.380 --> 00:21:38.500
of how regulated they are and how sensitive their
00:21:38.500 --> 00:21:41.359
industry, their business is, how important reliability
00:21:41.359 --> 00:21:45.339
is, where they are in their own maturity for
00:21:45.339 --> 00:21:47.779
how they handle these things. I guess what I
00:21:47.779 --> 00:21:50.539
would say is this isn't the first rodeo for anybody.
00:21:50.920 --> 00:21:53.660
And so you just got to have to look back to these
00:21:53.660 --> 00:21:56.819
past transformations to figure out what your
00:21:56.819 --> 00:21:59.440
map is for going forward. It's just it's in a
00:21:59.440 --> 00:22:02.339
very different domain. And so the controls are
00:22:02.339 --> 00:22:05.259
different and maybe even not completely understood.
00:22:06.950 --> 00:22:10.589
But that it's not like if you've it's not like
00:22:10.589 --> 00:22:12.650
most modern organizations of any size haven't
00:22:12.650 --> 00:22:14.490
seen this before. They've seen it before. It's
00:22:14.490 --> 00:22:17.589
just figuring out how to make that muscle work
00:22:17.589 --> 00:22:19.589
in a new way, you know, where these new exercises
00:22:19.589 --> 00:22:22.420
or something. I don't know. OK, let's switch
00:22:22.420 --> 00:22:26.319
gears to safeguards. So the report says teams
00:22:26.319 --> 00:22:30.410
are buying code quality analysts automated code
00:22:30.410 --> 00:22:34.150
review training, which safeguards actually move
00:22:34.150 --> 00:22:37.390
the needle and which ones can become checkbox
00:22:37.390 --> 00:22:40.170
theater? What should block a release versus what
00:22:40.170 --> 00:22:43.230
should create a tracked risk? I really do think,
00:22:43.289 --> 00:22:46.369
no, I mean, I think that's an individual, I mean,
00:22:46.390 --> 00:22:48.869
that determines like how regulated you are or
00:22:48.869 --> 00:22:51.789
how important uptime is or what have you. And
00:22:51.789 --> 00:22:54.430
again, I really think that we're still figuring
00:22:54.430 --> 00:22:58.980
out like gross best practices. Right. Like I
00:22:58.980 --> 00:23:01.559
think if we just use code review as an example,
00:23:01.759 --> 00:23:04.859
what we see all the time was organizations are
00:23:04.859 --> 00:23:07.599
they get really dynamic as they're leading in
00:23:07.599 --> 00:23:10.500
the code gen. Right. People who were just front
00:23:10.500 --> 00:23:12.119
end developers, back end developers, whatever,
00:23:12.200 --> 00:23:13.960
sort of becoming full stack
developers because
00:23:13.960 --> 00:23:18.799
they kind of can. And so and code review needs
00:23:18.799 --> 00:23:24.339
are, you know, skyrocketing and seniors are just
00:23:24.339 --> 00:23:26.420
like jumping into the breach, which in a way
00:23:26.420 --> 00:23:31.180
is amazing, right? That Sally sort of maybe done
00:23:31.180 --> 00:23:34.660
this part time, but now there's just so much
00:23:34.660 --> 00:23:37.220
need for it. And she loves doing it. And she's
00:23:37.220 --> 00:23:39.700
super good at it, that that's where she's spending
00:23:39.700 --> 00:23:42.119
the bulk of her time. And it's a fabulous thing.
00:23:42.240 --> 00:23:45.019
She's doing great. She's by the way, by virtue
00:23:45.019 --> 00:23:46.859
of all these code reviews, she's mentoring like
00:23:46.859 --> 00:23:49.500
all these more juniors and making them better
00:23:49.500 --> 00:23:51.059
at what they're doing. I honestly have goosebumps
00:23:51.059 --> 00:23:53.079
just talking with you about this, Brian. This
00:23:53.079 --> 00:23:56.039
dynamic is incredible to the development of an
00:23:56.039 --> 00:23:59.440
engineering team. And people are certainly adding
00:23:59.440 --> 00:24:02.980
in tech there too, like using CodeRabbit for
00:24:02.980 --> 00:24:05.240
code reviews, things of this nature, any part
00:24:05.240 --> 00:24:08.440
of the CI/CD pipeline. But per the Amazon comment
00:24:08.440 --> 00:24:11.160
earlier, you're never going to take humans out
00:24:11.160 --> 00:24:13.630
of the loop. I don't think it would be smart.
00:24:13.750 --> 00:24:16.250
And I also just don't think it would even fly
00:24:16.250 --> 00:24:18.750
regulatory wise, right? Like you need to have
00:24:18.750 --> 00:24:20.630
a trusted independent party like reviewing this
00:24:20.630 --> 00:24:22.690
stuff and a human should be involved at some
00:24:22.690 --> 00:24:26.390
point in the process. But I guess what I was
00:24:26.390 --> 00:24:28.029
getting at is I think that these people dynamics
00:24:28.029 --> 00:24:30.450
are under discussed and critically important
00:24:30.450 --> 00:24:33.990
to success. And so much of this is shadow work
00:24:33.990 --> 00:24:39.359
that has never been in a ticket, really. I mean,
00:24:39.400 --> 00:24:41.420
it's the kind of thing that it comes up because
00:24:41.420 --> 00:24:43.079
you're doing something else related to a pull
00:24:43.079 --> 00:24:45.400
request and the person just spends some time
00:24:45.400 --> 00:24:47.519
on it and then the pull request gets done. But
00:24:47.519 --> 00:24:51.500
what this transformation is unveiling is there's
00:24:51.500 --> 00:24:53.940
examples of this that are not trivial, like code
00:24:53.940 --> 00:24:56.960
review. And if you aren't looking at like developer
00:24:56.960 --> 00:24:59.680
participation ratios around commits versus code
00:24:59.680 --> 00:25:01.980
reviews or that kind of thing, then you're going
00:25:01.980 --> 00:25:05.380
to be blind to a very important dynamic to the
00:25:05.380 --> 00:25:09.000
success of the engineering team in maximizing
00:25:09.000 --> 00:25:11.380
this transformation. But just as importantly,
00:25:11.640 --> 00:25:14.500
like the team dynamics that were underneath that,
00:25:14.619 --> 00:25:17.220
right? Like this opportunity for Sally to shine
00:25:17.220 --> 00:25:19.440
in a way that like she didn't even see coming
00:25:19.440 --> 00:25:21.740
and she's loving it. And the impact that that's
00:25:21.740 --> 00:25:25.200
having on the rest of the team. I guess the reason
00:25:25.200 --> 00:25:27.019
I get goosebumps talking about it is like, this
00:25:27.019 --> 00:25:29.940
is what any manager just, it's what makes the
00:25:29.940 --> 00:25:33.819
job awesome. Like truly awesome. It makes you
00:25:33.819 --> 00:25:35.839
like never want to stop doing it. Like I'll take
00:25:35.839 --> 00:25:38.319
that dopamine hit and the team dynamics like
00:25:38.319 --> 00:25:41.940
all day, every single day. So anyway, that opportunity
00:25:41.940 --> 00:25:44.259
is there, but you have to be tooled to look for
00:25:44.259 --> 00:25:47.480
it and to find it. Yeah. It just makes me think
00:25:47.480 --> 00:25:49.720
like, where will we be in five years even? You
00:25:49.720 --> 00:25:52.000
know, just five years, which isn't a long time,
00:25:52.000 --> 00:25:55.099
but is in the current climate. It's happening
00:25:55.099 --> 00:25:57.539
everywhere. Like, in our prep, we were talking a
00:25:57.539 --> 00:25:59.619
little bit about, so we can maybe save this for
00:25:59.619 --> 00:26:02.019
later, but it's like, what do teams look like in
00:26:02.019 --> 00:26:04.039
the future? And I was just thinking about how,
00:26:04.039 --> 00:26:07.440
you know, we're going back to, like, MRDs and PRDs
00:26:07.440 --> 00:26:09.920
in a way because that's what we've learned is
00:26:09.920 --> 00:26:14.720
like the next unlock. Um, and here too, what does
00:26:14.720 --> 00:26:17.460
that mean for the people you need to have on
00:26:17.460 --> 00:26:20.140
the team and for what their skill sets are as
00:26:20.140 --> 00:26:23.980
a leader what does it mean for How you're tracking,
00:26:24.160 --> 00:26:29.930
you know, people's contributions and how are
00:26:29.930 --> 00:26:32.970
you sort of measuring them? And based on that,
00:26:33.049 --> 00:26:35.390
how are you fostering continuous improvement
00:26:35.390 --> 00:26:38.990
and development of the team? And here again is
00:26:38.990 --> 00:26:41.549
where it's like unprecedented territory that
00:26:41.549 --> 00:26:43.970
we're in. Like these are not things that were
00:26:43.970 --> 00:26:47.700
part of like just day in, day out blocking and
00:26:47.700 --> 00:26:50.380
tackling engineering team conversations. They're
00:26:50.380 --> 00:26:52.440
things you talk about once a quarter, once a
00:26:52.440 --> 00:26:55.039
year, when you're thinking about your teams and
00:26:55.039 --> 00:26:57.920
how they're developing. But that is not nearly
00:26:57.920 --> 00:27:00.099
fast enough right now if you want to be the best
00:27:00.099 --> 00:27:04.440
of the best. Yeah, I even see companies iterating
00:27:04.440 --> 00:27:08.240
over their OKRs continually now, which is interesting.
00:27:08.759 --> 00:27:11.660
Okay, so I guess let's talk about that. Regarding
00:27:11.660 --> 00:27:14.569
the report, what did you glean as far as the
00:27:14.569 --> 00:27:18.480
future of AI? development? Yeah, I mean, to me,
00:27:18.500 --> 00:27:20.400
it felt like we're in the first inning, you know,
00:27:20.420 --> 00:27:23.099
these are just like, honestly, such basic concerns.
00:27:24.640 --> 00:27:26.519
That if you want to think about that future,
00:27:26.619 --> 00:27:29.839
where we're just verbosely prescriptive, and
00:27:29.839 --> 00:27:32.779
we're in like this continuous loops of development,
00:27:33.000 --> 00:27:35.940
and so on and so forth, like that ain't possible.
00:27:37.440 --> 00:27:40.839
Right? Unless we have done a lot of spade work
00:27:40.839 --> 00:27:44.619
on all of these guardrails and controls, because
00:27:45.059 --> 00:27:46.980
Here again, you can't have one without the other.
00:27:47.059 --> 00:27:50.460
You can't say you love loops and what that does
00:27:50.460 --> 00:27:53.119
for your velocity without understanding that
00:27:53.119 --> 00:27:56.039
there's a dark side. It will accelerate just
00:27:56.039 --> 00:27:58.740
as much the bad as it does the good, so you better
00:27:58.740 --> 00:28:01.740
have some guardrails around that, or else you'll
00:28:01.740 --> 00:28:04.319
find yourself worse off. That, I suppose, is
00:28:04.319 --> 00:28:06.480
maybe the cautionary tale for engineering leaders,
00:28:06.680 --> 00:28:12.539
is that there's an exacerbating effect here that...
00:28:12.809 --> 00:28:15.650
If you lean really hard into this stuff without
00:28:15.650 --> 00:28:18.450
thinking carefully about the guardrails or about
00:28:18.450 --> 00:28:21.089
what the processes and controls are, it can spiral
00:28:21.089 --> 00:28:23.549
out of control really quickly. There's going
00:28:23.549 --> 00:28:26.089
to be some smoking craters here. The Amazon thing,
00:28:26.130 --> 00:28:30.430
I think, will seem small. I've read the Medium
00:28:30.430 --> 00:28:32.970
articles on the Ralph Wiggum loops. I've read
00:28:32.970 --> 00:28:36.730
Claude's program where they had the internal
00:28:36.730 --> 00:28:42.240
vending machine that went rogue. There's just
00:28:42.240 --> 00:28:45.500
countless examples of people just open claw,
00:28:45.680 --> 00:28:47.599
even just what people were doing with open claw
00:28:47.599 --> 00:28:50.779
early on. There's just so many examples of, yes,
00:28:50.799 --> 00:28:54.059
if you let AI just loose, there are going to
00:28:54.059 --> 00:28:56.880
be some interesting consequences that you couldn't
00:28:56.880 --> 00:28:59.420
have even fathomed really before letting that
00:28:59.420 --> 00:29:02.740
go. So guardrails, obviously, just goes back
00:29:02.740 --> 00:29:05.960
to that. So important. It seems like the boring
00:29:05.960 --> 00:29:09.859
side of the conversation, but like... It's paramount
00:29:09.859 --> 00:29:12.640
to everything that comes after that. Yeah. I
00:29:12.640 --> 00:29:17.000
mean, it's an engineering process. So every time
00:29:17.000 --> 00:29:18.599
you'd make something better, that's going to
00:29:18.599 --> 00:29:20.059
create a new bottleneck and you're going to have
00:29:20.059 --> 00:29:21.859
to figure out how to tackle that bottleneck.
00:29:22.059 --> 00:29:24.099
I mean, just thinking back on this question,
00:29:24.200 --> 00:29:26.119
I think it's safe to say that in each of the
00:29:26.119 --> 00:29:30.259
discrete functions, there will emerge progressively
00:29:30.259 --> 00:29:34.769
better. AI driven solutions for that thing, whatever
00:29:34.769 --> 00:29:37.150
it is, code review, generating documentation,
00:29:37.670 --> 00:29:40.769
finding and fixing bugs, you name it. And so
00:29:40.769 --> 00:29:42.049
of course, you're going to want to be tracking
00:29:42.049 --> 00:29:45.170
all of that to try to maximize your productivity.
00:29:46.009 --> 00:29:47.769
But every time you do that, there's going to
00:29:47.769 --> 00:29:49.750
be a change in process and that's going to require
00:29:49.750 --> 00:29:54.410
considering what the controls are, right? Like
00:29:54.410 --> 00:29:57.230
does this, now that we've improved the way that
00:29:57.230 --> 00:30:00.440
we're finding and fixing bugs, does that Does
00:30:00.440 --> 00:30:03.180
the guardrails we had or the human control that
00:30:03.180 --> 00:30:04.559
we had, is that still appropriate? Does that
00:30:04.559 --> 00:30:06.039
need to change? Does that need to be updated?
00:30:06.680 --> 00:30:09.119
I think that's the nature of the work that'll
00:30:09.119 --> 00:30:12.660
need to happen. And I think it does also suggest
00:30:12.660 --> 00:30:16.880
that there's some sort of a control plane that's
00:30:16.880 --> 00:30:19.279
going to have to sit above all of this. Because
00:30:19.279 --> 00:30:22.119
I do think what this will bring to a head is
00:30:22.119 --> 00:30:25.920
the stovepiped nature of the way a lot of these
00:30:25.920 --> 00:30:28.740
things happen today, right? A lot of these things
00:30:28.740 --> 00:30:31.140
like... in your environment you'd know far better
00:30:31.140 --> 00:30:33.400
than me like it's probably people looking at
00:30:33.400 --> 00:30:35.559
like network stuff and application level stuff
00:30:35.559 --> 00:30:37.740
and cloud level stuff and you're trying to get
00:30:37.740 --> 00:30:39.339
some signal from all that noise to like figure
00:30:39.339 --> 00:30:41.140
out what's going on in terms of performance of
00:30:41.140 --> 00:30:44.359
reliability outages all the rest but that's
just
00:30:44.359 --> 00:30:49.690
as true in all of these other domains And at
00:30:49.690 --> 00:30:52.529
the end of the day, the engineering leader is
00:30:52.529 --> 00:30:54.630
going to need to have some visibility across
00:30:54.630 --> 00:30:58.109
all of this in order to be effective. If them
00:30:58.109 --> 00:31:00.490
being effective is going to require them to look
00:31:00.490 --> 00:31:03.150
discreetly at all these different tools or to
00:31:03.150 --> 00:31:08.029
manually... Sorry, but this is really very funny.
00:31:08.329 --> 00:31:10.410
I mean, right now, these are pretty static dashboards
00:31:10.410 --> 00:31:13.450
is our best attempt at doing this, right? So
00:31:13.450 --> 00:31:14.930
how the hell are you going to do that when we
00:31:14.930 --> 00:31:17.569
move into loop world? It's just not going to work.
00:31:17.670 --> 00:31:21.549
So you're going to need to have, you know, some
00:31:21.549 --> 00:31:24.029
triggers or controls and this sort of management
00:31:24.029 --> 00:31:26.410
plane that in real time is kind of looking at
00:31:26.410 --> 00:31:28.670
what's happening that ultimately gives that engineering
00:31:28.670 --> 00:31:31.190
leader like that insight, because it's probably
00:31:31.190 --> 00:31:34.210
often going to be blended, you know, or that's
00:31:34.210 --> 00:31:36.769
that would be the ideal scenario, right, where
00:31:36.769 --> 00:31:40.450
security is a little tricky. So maybe we'll talk
00:31:40.450 --> 00:31:43.140
about there's a there's some churn happening
00:31:43.140 --> 00:31:45.119
in a portion of the code base and that's slowing
00:31:45.119 --> 00:31:47.660
your velocity for that that team and that part
00:31:47.660 --> 00:31:50.480
of the application oh and by the way we think
00:31:50.480 --> 00:31:54.740
that's because there was um A new library with
00:31:54.740 --> 00:31:56.720
a vulnerability and the team is figuring out
00:31:56.720 --> 00:32:00.079
like how to solve for this while preserving the
00:32:00.079 --> 00:32:01.940
functionality of the application or something
00:32:01.940 --> 00:32:04.599
like this. That's my best way off the top of
00:32:04.599 --> 00:32:06.359
my head I can think of to talk about. Like there's
00:32:06.359 --> 00:32:08.700
multiple things that are involved in understanding
00:32:08.700 --> 00:32:11.500
that problem. And if you're in loop world where
00:32:11.500 --> 00:32:14.039
the code is just writing itself and you're trying
00:32:14.039 --> 00:32:16.299
to like... after it's already into production,
00:32:16.460 --> 00:32:18.279
try to figure this out with a weekly snapshot
00:32:18.279 --> 00:32:19.819
or something like that. It's just not going to
00:32:19.819 --> 00:32:22.119
fly. We're going to need agentic systems is what
00:32:22.119 --> 00:32:24.579
I think I'm using a lot of words to say. And
00:32:24.579 --> 00:32:26.220
those agentic systems are going to have to be
00:32:26.220 --> 00:32:29.680
multidisciplinary in order to be effective and
00:32:29.680 --> 00:32:31.759
to support this vision that we're talking about
00:32:31.759 --> 00:32:35.579
of like kind of autonomous, what are they calling
00:32:35.579 --> 00:32:37.480
it? Dark factories, you know, just cranking the
00:32:37.480 --> 00:32:41.099
out. For sure. Okay. So wrapping up, your
00:32:41.099 --> 00:32:43.779
conclusion says the code isn't the hard part
00:32:43.779 --> 00:32:46.980
anymore. Visibility and controls are. What are
00:32:46.980 --> 00:32:51.220
high performing teams doing differently? It's
00:32:51.220 --> 00:32:54.380
really interesting. What we're seeing is a lot
00:32:54.380 --> 00:32:56.740
of people coding their own stuff, actually, to
00:32:56.740 --> 00:33:00.400
try to solve this problem. And to be honest,
00:33:00.539 --> 00:33:02.920
I think that's a function of there's a lot of
00:33:02.920 --> 00:33:05.579
legacy tooling that people have used in the past.
00:33:05.700 --> 00:33:08.740
And it's all been based on like derivatives of
00:33:08.740 --> 00:33:12.200
work, right? You're, you've got to Ticketing
00:33:12.200 --> 00:33:14.259
system, and that's great. I'm not saying get
00:33:14.259 --> 00:33:16.039
rid of that or anything like that, but you have
00:33:16.039 --> 00:33:17.559
this ticketing system and people are using it
00:33:17.559 --> 00:33:19.240
a certain way, and so things are being tagged
00:33:19.240 --> 00:33:21.460
a certain way, and maybe you've got similar things
00:33:21.460 --> 00:33:23.579
happening in GitHub, and then you're building
00:33:23.579 --> 00:33:26.240
a dashboard maybe off of all of that to try to
00:33:26.240 --> 00:33:28.980
get the visibility that you want. And the problem,
00:33:29.019 --> 00:33:30.619
of course, is that there's no way that can keep
00:33:30.619 --> 00:33:33.440
up as we move to loops and as we move to dark
00:33:33.440 --> 00:33:37.170
factories. Or why would you? You're just going
00:33:37.170 --> 00:33:39.069
to have the AI working as hard to keep the tickets
00:33:39.069 --> 00:33:41.549
up to date as it is creating the code. That seems
00:33:41.549 --> 00:33:44.150
silly. So there's got to be a better way. And
00:33:44.150 --> 00:33:46.670
this does get to what we do, but forget whether
00:33:46.670 --> 00:33:49.670
it's Flux or anything else. We have to move to
00:33:49.670 --> 00:33:52.210
a code-based approach. And we can move to a
00:33:52.210 --> 00:33:53.809
code-based approach because we have AI now,
00:33:53.829 --> 00:33:56.589
and AI understands code. But we need a better
00:33:56.589 --> 00:33:58.089
mousetrap, I guess is what I'm trying to say.
00:33:58.130 --> 00:33:59.529
Whether it's Flux or anything else, we need a
00:33:59.529 --> 00:34:01.890
better mousetrap that's better suited to the
00:34:01.890 --> 00:34:04.990
pace of development that we've now... unleashed
00:34:04.990 --> 00:34:09.070
in the organization, to provide that, that, to close
00:34:09.070 -->
00:34:11.690
that gap, to close that visibility gap. And the
00:34:11.690 --> 00:34:13.289
first step that we're seeing people do because
00:34:13.289 --> 00:34:15.869
they can, because it's easy to try, is coding their
00:34:15.869 --> 00:34:19.130
own thing. Um, because they're finding those, like,
00:34:19.130 --> 00:34:21.269
it's like easier to do that than it is to retool
00:34:21.269 --> 00:34:24.070
your legacy solution, to be honest, to get the
00:34:24.070 --> 00:34:26.650
visibility that you desire in this new AI environment.
00:34:26.650 --> 00:34:28.949
So I think we'll see where that goes. Like, is
00:34:28.949 --> 00:34:31.329
that sufficient? Maybe. But this, I think, is the
00:34:31.329 --> 00:34:34.110
the canonical question for, like, coding your own
00:34:34.110 --> 00:34:38.349
anything. Um, it's easy to try it, and I'm not saying
00:34:38.349 --> 00:34:42.340
it can't work, but like, that argument writ large
00:34:42.340 --> 00:34:43.960
means you're writing all your own software for
00:34:43.960 --> 00:34:45.579
everything, and that just on its face doesn't
00:34:45.579 --> 00:34:49.019
make any sense. So people are thinking we'll have
00:34:49.019 --> 00:34:51.260
to choose, like, where that's good enough and where,
00:34:51.260 --> 00:34:54.219
you know what, like, this is important enough, there's
00:34:54.219 --> 00:34:56.139
enough change, like, we don't need to own this,
00:34:56.139 --> 00:34:58.219
like we're better off making this somebody else's
00:34:58.219 --> 00:35:01.760
problem. So how can Flux help? What does Flux offer?
00:35:02.260 --> 00:35:04.619
I mean, the whole reason for starting the company
00:35:04.619 --> 00:35:08.920
was seeing that starting with code generation,
00:35:09.219 --> 00:35:11.960
that there was a complete upending of engineering
00:35:11.960 --> 00:35:15.000
and that to manage that process would require
00:35:15.000 --> 00:35:16.659
a completely different approach than everything
00:35:16.659 --> 00:35:19.639
that we've tried so far. And so that's why we
00:35:19.639 --> 00:35:22.599
built Flux, is to do a code-first approach to
00:35:22.599 --> 00:35:25.579
how you're understanding how your code is changing
00:35:25.579 --> 00:35:27.760
and how your teams are changing as you go through
00:35:27.760 --> 00:35:31.590
your AI transformation. In a nutshell, That's
00:35:31.590 --> 00:35:34.110
what we're here to do is to help make it easy
00:35:34.110 --> 00:35:37.349
for engineering leaders to prove the efficacy
00:35:37.349 --> 00:35:41.090
of their AI transformation and to continuously
00:35:41.090 --> 00:35:43.949
improve it going forward with a code-first approach
00:35:43.949 --> 00:35:46.449
because it's more accurate. It's better scalable.
00:35:46.630 --> 00:35:49.349
Oh, and by the way, it removes all the toil of
00:35:49.349 --> 00:35:51.469
all those legacy systems. So you get the benefits
00:35:51.469 --> 00:35:54.710
of this approach without having to put more burden
00:35:54.710 --> 00:35:57.210
on your teams to change how you're doing ticketing
00:35:57.210 --> 00:35:59.070
or change how things are happening in GitHub
00:35:59.070 --> 00:36:02.170
or any of that. Very cool. Awesome. Any last
00:36:02.170 --> 00:36:03.650
thoughts you'd like to leave our audience with,
00:36:03.710 --> 00:36:06.190
Ted? I guess it's mainly like, what a fun time
00:36:06.190 --> 00:36:10.449
this is. Yeah, no, I agree. Right on, you know,
00:36:10.489 --> 00:36:14.820
I mean, the transformation that's happening in
00:36:14.820 --> 00:36:16.119
engineering has never happened before and it
00:36:16.119 --> 00:36:18.559
never will happen again. And so that's both scary,
00:36:18.559 --> 00:36:20.340
but it's also what an incredible opportunity
00:36:20.340 --> 00:36:22.619
for everybody that's in this field and their
00:36:22.619 --> 00:36:26.619
careers, wherever you sit in the stack. When
00:36:26.619 --> 00:36:29.980
I mentor... people, like in college, this, what
00:36:29.980 --> 00:36:32.920
I have said forever is a tech disruption, that
00:36:32.920 --> 00:36:34.980
is your best opportunity to advance your career,
00:36:34.980 --> 00:36:38.639
right? Is to go deep on that, to become an expert
00:36:38.639 --> 00:36:40.139
in that, because in the early days there are no
00:36:40.139 --> 00:36:43.780
experts. And that is, that has always been the,
00:36:43.780 --> 00:36:46.400
like the silver bullet for really accelerating
00:36:46.400 --> 00:36:48.199
the growth in your career. And that's exactly
00:36:48.199 --> 00:36:51.420
where we are with engineering. And so I try to
00:36:51.420 --> 00:36:53.039
focus on the upside of that because I think there
00:36:53.039 --> 00:36:56.820
is enormous upside there for people who do lean
00:36:56.820 --> 00:36:59.219
in and who do become the experts. And so that's,
00:36:59.260 --> 00:37:00.760
I guess, what I would take away is wherever you're
00:37:00.760 --> 00:37:03.820
in the stack, step back and just realize this
00:37:03.820 --> 00:37:06.800
ain't gonna happen again. There'll be other stuff.
00:37:06.900 --> 00:37:08.579
It'll be interesting. It'll be fascinating. But
00:37:08.579 --> 00:37:11.380
the first time's the first time. And that's what
00:37:11.380 --> 00:37:13.840
we're in for engineering with code. Absolutely.
00:37:14.539 --> 00:37:16.900
Well, where can people find more about Flux and
00:37:16.900 --> 00:37:21.510
you? askflux.ai. We've completely revamped the
00:37:21.510 --> 00:37:23.349
website. We have the research survey on there.
00:37:23.489 --> 00:37:26.110
We have a great case study about effective AI
00:37:26.110 --> 00:37:28.530
transformations for engineering leaders that
00:37:28.530 --> 00:37:30.829
I think would be a fun read for anybody to get
00:37:30.829 --> 00:37:34.789
sort of a case study. That's what it is for like
00:37:34.789 --> 00:37:36.510
how you could do it or maybe see where you're
00:37:36.510 --> 00:37:37.789
at in your journey and what you can learn from
00:37:37.789 --> 00:37:41.090
it. Very cool. Ted, thank you so much for your
00:37:41.090 --> 00:37:42.329
time. Thank you for coming on. Really appreciate
00:37:42.329 --> 00:37:44.949
it. Thanks a ton, Brian. It was a pleasure. That
00:37:44.949 --> 00:37:48.230
was my conversation with Ted Julian from Flux.
00:37:48.780 --> 00:37:51.500
The biggest takeaway for me is that writing the
00:37:51.500 --> 00:37:54.900
code isn't the hard part anymore. Seeing and
00:37:54.900 --> 00:37:58.679
trusting what it changes is. When Ted's survey
00:37:58.679 --> 00:38:02.179
asked what was holding teams back, the answers
00:38:02.179 --> 00:38:06.219
were about trust. Complexity that spirals into
00:38:06.219 --> 00:38:09.519
an outage. A new library with a vulnerability.
00:38:10.079 --> 00:38:13.420
Regulated data nobody has really scrutinized.
00:38:13.639 --> 00:38:17.219
Code review breaks first. Ted calls it act two.
00:38:17.670 --> 00:38:20.869
and it's a people problem as much as a tooling
00:38:20.869 --> 00:38:25.170
problem. AI review helps, but he doesn't think
00:38:25.170 --> 00:38:28.590
you take humans out of the loop. And the bigger
00:38:28.590 --> 00:38:31.969
bill comes later. When someone has to understand
00:38:31.969 --> 00:38:36.170
code nobody really wrote. His once-in-a-career
00:38:36.170 --> 00:38:39.869
hack is to start measuring features versus maintenance
00:38:39.869 --> 00:38:44.929
versus bug fixes now, backfill it, and show the
00:38:44.929 --> 00:38:51.079
before and after. One customer went from 9% features to 43%.
00:38:51.079 --> 00:38:55.019
And it ended up as a chart in their board meeting.
00:38:55.199 --> 00:38:59.460
On ownership, it's who feeds the cat. Ted says
00:38:59.460 --> 00:39:02.800
it has to be the head of engineering and ultimately
00:39:02.800 --> 00:39:06.900
the CIO. And for your career, tech disruption
00:39:06.900 --> 00:39:10.360
is the best chance you'll get to move up. Because
00:39:10.360 --> 00:39:14.250
in the early days, there are no experts. You
00:39:14.250 --> 00:39:19.050
can find Ted and Flux at askflux.ai, along with
00:39:19.050 --> 00:39:22.190
the full report and a case study on AI transformations
00:39:22.190 --> 00:39:25.150
for engineering leaders through the links in
00:39:25.150 --> 00:39:28.170
the show notes. Follow or subscribe to Ship It
00:39:28.170 --> 00:39:30.889
Weekly wherever you listen, and find previous
00:39:30.889 --> 00:39:34.610
episodes at shipitweekly.fm. Thanks for listening,
00:39:34.670 --> 00:39:36.710
and I'll see you Friday for the news.