TRANSCRIPTION 🔗
WEBVTT
00:00:00.000 --> 00:00:03.100
Drop someone in the middle of New York City with
00:00:03.100 --> 00:00:07.280
no directions and they'll get lost. Lots of teams
00:00:07.280 --> 00:00:11.380
do that to AI agents. They hand over some context,
00:00:11.640 --> 00:00:16.440
maybe an agents.md file, and call it done. My
00:00:16.440 --> 00:00:19.960
guest today says that's not enough. The agent
00:00:19.960 --> 00:00:23.760
needs to know what to do, how to do it, how it'll
00:00:23.760 --> 00:00:27.059
know it's done, and what it's allowed to touch.
00:00:27.609 --> 00:00:30.629
And if your platform can't do that for humans
00:00:30.629 --> 00:00:34.369
yet, she'll tell you to pump the brakes. As one
00:00:34.369 --> 00:00:37.509
of her workshop attendees put it, if we can't
00:00:37.509 --> 00:00:40.829
even write a README for our repos, how are we
00:00:40.829 --> 00:00:44.189
going to write one for our agents? I'm Brian
00:00:44.189 --> 00:00:47.436
Teller from Teller's Tech and this is Ship It Weekly.
00:01:04.307 --> 00:01:06.510
Welcome back to Ship It Weekly, where
00:01:06.510 --> 00:01:09.170
I filter the noise and focus on what matters
00:01:09.540 --> 00:01:11.819
when you're the one running infrastructure and
00:01:11.819 --> 00:01:15.200
owning reliability. Most weeks, it's a quick
00:01:15.200 --> 00:01:19.500
DevOps, SRE, platform, cloud, and security news
00:01:19.500 --> 00:01:23.040
recap. In between those, I do conversation episodes
00:01:23.040 --> 00:01:26.159
with people building and operating these systems.
00:01:26.920 --> 00:01:30.680
Ship It Weekly turns one this November. To celebrate
00:01:30.680 --> 00:01:33.599
through mid-December, you are getting two conversations
00:01:33.599 --> 00:01:37.480
a week, Mondays and Wednesdays, with the news
00:01:37.480 --> 00:01:41.230
every Friday. Then we're back to our usual rhythm.
00:01:41.370 --> 00:01:44.849
Today, I'm joined by Mallory, Head of Platform
00:01:44.849 --> 00:01:48.549
Education and Advocacy at PlatformEngineering.org.
00:01:48.549 --> 00:01:52.890
Mallory works with platform and DevEx teams
00:01:52.890 --> 00:01:57.189
on how platforms actually succeed or fail inside
00:01:57.189 --> 00:02:00.950
engineering organizations. And lately, on what
00:02:00.950 --> 00:02:04.430
changes when AI agents enter the software delivery
00:02:04.430 --> 00:02:08.360
workflow. We talk about... why context delivery
00:02:08.360 --> 00:02:12.439
alone isn't enough, the harness and governance
00:02:12.439 --> 00:02:16.199
layers that actually run agents on a platform,
00:02:16.199 --> 00:02:20.460
the three kinds of paths agents work on, why she
00:02:20.460 --> 00:02:24.479
tells some teams to fix human governance before
00:02:24.479 --> 00:02:28.939
adding agents, and where a platform team should
00:02:28.939 --> 00:02:32.251
actually start. All right, let's jump in.
00:02:36.795 --> 00:02:39.060
Mallory, thank you for joining me. Thanks for having me,
00:02:39.060 --> 00:02:41.580
Brian. This is really exciting. So I'm interested
00:02:41.580 --> 00:02:45.360
to hear one line that you had mentioned was context
00:02:45.360 --> 00:02:48.620
delivery alone isn't enough. What do you mean
00:02:48.620 --> 00:02:50.939
by that? I try to think of a lot of the things
00:02:50.939 --> 00:02:53.419
with AI in general in the same way we think about
00:02:53.419 --> 00:02:55.819
interacting with humans. So context delivery
00:02:55.819 --> 00:02:59.159
with a human isn't alone. I can put you in the
00:02:59.159 --> 00:03:00.979
middle of New York City and say you're in the
00:03:00.979 --> 00:03:03.039
middle of New York City. But if I don't give
00:03:03.039 --> 00:03:04.800
you any clear direction on where I'm expecting
00:03:04.800 --> 00:03:07.539
you to go or what I'm expecting you to do, and
00:03:07.539 --> 00:03:08.979
I just took you out of the middle of your day
00:03:08.979 --> 00:03:11.409
and put you there chances are you might get
00:03:11.409 --> 00:03:12.789
yourself into some trouble or you're going to
00:03:12.789 --> 00:03:16.129
get lost. It's the same thing with agents. So
00:03:16.129 --> 00:03:18.750
when I talk about context delivery on its own
00:03:18.750 --> 00:03:20.610
just not being enough, there's other things we
00:03:20.610 --> 00:03:22.710
have to actually give that agent to. It's things
00:03:22.710 --> 00:03:25.289
like, what exactly are you supposed to be doing?
00:03:25.590 --> 00:03:28.409
How exactly should you be looking at doing it?
00:03:28.469 --> 00:03:30.969
And how are you going to know whether or not
00:03:30.969 --> 00:03:34.550
it was done and done correctly? Then there's
00:03:34.550 --> 00:03:36.930
also this other piece, which comes to governance
00:03:36.930 --> 00:03:39.990
and security, which sits alongside that. That
00:03:39.990 --> 00:03:42.210
says, okay, here's how you do your work, and
00:03:42.210 --> 00:03:45.389
here's how you're allowed to do it, where you're
00:03:45.389 --> 00:03:47.509
allowed to do it, and the constraints of that
00:03:47.509 --> 00:03:50.229
entire operation. That's what I mean. You just
00:03:50.229 --> 00:03:52.469
give something context. You give it an agent.md
00:03:52.469 --> 00:03:54.830
file. That's not enough. There needs to be
00:03:54.830 --> 00:03:58.150
a lot more. And what is that more? Is it a Kyverno
00:03:58.150 --> 00:04:00.930
rule? What do you mean by that? Definitely part
00:04:00.930 --> 00:04:05.439
of it. We talk about this agent infrastructure
00:04:05.439 --> 00:04:07.740
layer. And that's really what runs agents inside
00:04:07.740 --> 00:04:09.800
of a platform. And there's two pieces. So there's
00:04:09.800 --> 00:04:11.879
the harness. That's how the agent does the work.
00:04:12.099 --> 00:04:15.699
So that's an execution component. There's context,
00:04:15.939 --> 00:04:19.040
capability, that evaluation rubric, basically.
00:04:19.060 --> 00:04:21.959
So like, did I do a good job? Am I done? But
Load full transcript
00:04:21.959 --> 00:04:23.860
the second piece is that governance layer. So
00:04:23.860 --> 00:04:26.819
what's allowed? What's safe? What is actually
00:04:26.819 --> 00:04:28.459
being observed here? So that's things like identity.
00:04:28.920 --> 00:04:31.779
So whether or not that is something like... OPA,
00:04:32.000 --> 00:04:35.639
Kyverno, just regular kind of RBAC roles applied
00:04:35.639 --> 00:04:39.120
in a slightly different way. It's security parameters
00:04:39.120 --> 00:04:43.339
and context too. So things like codified security
00:04:43.339 --> 00:04:44.879
standards, whatever that might look like for
00:04:44.879 --> 00:04:46.740
you, or even just some sort of vulnerability
00:04:46.740 --> 00:04:49.220
scanning and testing. And then there's also just
00:04:49.220 --> 00:04:52.769
its unique requirements for observability. Agent
00:04:52.769 --> 00:04:54.509
observability is something that people are talking
00:04:54.509 --> 00:04:56.970
about a lot. And that's a big part of that governance
00:04:56.970 --> 00:04:59.230
piece. But both of those harness and governance
00:04:59.230 --> 00:05:01.709
layers make up agent infrastructure as a whole.
00:05:01.850 --> 00:05:04.949
So when you talk about observability with agents,
00:05:05.329 --> 00:05:08.769
what does that mean? Like what kind of analytics
00:05:08.769 --> 00:05:10.550
are we trying to get out of the agent or what's
00:05:10.550 --> 00:05:14.410
valuable to pass into like, I don't know, Grafana,
00:05:14.509 --> 00:05:19.240
whatever? Yep. And I think there's a lot of different
00:05:19.240 --> 00:05:20.620
ways people are talking about this. But again,
00:05:20.699 --> 00:05:22.819
I come back to like, how do we think about observability
00:05:22.819 --> 00:05:26.199
with humans? A good platform at the end of the
00:05:26.199 --> 00:05:30.819
day is very auditable. So I can see what actions
00:05:30.819 --> 00:05:33.839
were taken either by an agent or a human, so
00:05:33.839 --> 00:05:37.819
an actor inside of that platform, or the platform
00:05:37.819 --> 00:05:41.040
making a decision itself. And what we like to
00:05:41.040 --> 00:05:43.100
think about is if the platform makes a decision,
00:05:43.220 --> 00:05:45.410
that should be part of the audit trail as well.
00:05:45.750 --> 00:05:48.110
So you're observing actions, but you're also
00:05:48.110 --> 00:05:50.769
observing outcomes and system health and performance.
00:05:51.009 --> 00:05:52.569
You know, there's always that discussion too
00:05:52.569 --> 00:05:56.850
about observability isn't just looking at system
00:05:56.850 --> 00:05:59.250
health. It's also that system performance aspect.
00:05:59.329 --> 00:06:02.209
So for agents, you know, you're looking at token
00:06:02.209 --> 00:06:04.889
spend, you're looking at, you know, the time
00:06:04.889 --> 00:06:06.870
to completion, to getting to that evaluation
00:06:06.870 --> 00:06:08.990
piece. So if there is a loop, for instance, that
00:06:08.990 --> 00:06:11.350
it goes through a retry loop, how many times
00:06:11.350 --> 00:06:13.009
is it going through that loop? You should, by
00:06:13.009 --> 00:06:15.069
the way, set a limit on that. But how many times
00:06:15.069 --> 00:06:17.069
did it go through it before it got to that acceptance
00:06:17.069 --> 00:06:21.629
criteria? There's always pieces too with just
00:06:21.629 --> 00:06:24.509
sort of monitoring the step-by-step process.
00:06:24.889 --> 00:06:27.689
So was this gate passed? Was this gate failed?
00:06:28.250 --> 00:06:30.629
What triggered this particular event? So it's
00:06:30.629 --> 00:06:32.569
that bigger picture observability that we talk
00:06:32.569 --> 00:06:35.129
about in platform engineering as one of the pieces
00:06:35.129 --> 00:06:38.050
people tend to get wrong really quickly because
00:06:38.050 --> 00:06:40.490
they just focus in on like... performance metrics
00:06:40.490 --> 00:06:43.850
or, you know, system performance, like, oh, we have
00:06:43.850 --> 00:06:47.490
a whole bunch of 400s. That's a problem. Um, it's
00:06:47.490 --> 00:06:48.709
a little bit different and a little bit more
00:06:48.709 --> 00:06:51.490
nuanced than that. So it's efficiency, but it's
00:06:51.490 --> 00:06:54.089
also, like, correctness? Are we actually validating
00:06:54.089 --> 00:06:56.529
for correctness, and how are we doing that? Or
00:06:56.529 --> 00:06:59.949
is the user providing that input, or how are we
00:06:59.949 --> 00:07:02.410
determining the correctness from the agent? I
00:07:02.410 --> 00:07:04.110
think that's a good question because it really
00:07:04.110 --> 00:07:06.589
depends on the circumstance in which you're using
00:07:06.589 --> 00:07:09.850
that agent to begin with. And this is something
00:07:09.850 --> 00:07:11.990
that I work a lot with teams on is sort of deciding
00:07:11.990 --> 00:07:14.730
when are you going to bring an agent into here
00:07:14.730 --> 00:07:16.750
and why? What are you trying to specifically
00:07:16.750 --> 00:07:19.430
improve? So with anything in platform engineering,
00:07:19.670 --> 00:07:22.230
you want to have a clear why. Why are you doing
00:07:22.230 --> 00:07:24.810
it? And that should guide you towards what it
00:07:24.810 --> 00:07:26.589
is you actually need to measure to explain that
00:07:26.589 --> 00:07:29.550
you're successful. So whether or not that's some
00:07:29.550 --> 00:07:31.310
sort of throughput metric, whether or not it's
00:07:31.310 --> 00:07:34.449
something around uptime, you know, mean time recovery,
00:07:34.629 --> 00:07:37.480
whatever other DORA metric you want to use. That
00:07:37.480 --> 00:07:40.699
stuff is all really important, but you have to
00:07:40.699 --> 00:07:43.259
be able to actually articulate that against your
00:07:43.259 --> 00:07:44.920
why. So why are you even doing this to begin
00:07:44.920 --> 00:07:48.060
with? It's nice to say, great, you reduced our
00:07:48.060 -->
00:07:50.620
downtime, but why did we need to do that? Did
00:07:50.620 --> 00:07:53.019
we have a problem? And it's the same with the
00:07:53.019 --> 00:07:55.100
agentic pieces too. So we want to put that agent
00:07:55.100 --> 00:07:57.699
in to actually make something better, make something
00:07:57.699 --> 00:08:02.879
faster, be more innovative, reduce toil on developers
00:08:02.879 --> 00:08:08.149
or operations folks, move at a speed that regular
00:08:08.149 --> 00:08:10.730
humans just can't do. So we have to be able to
00:08:10.730 --> 00:08:12.850
say to ourselves, what do we need to observe, and
00:08:12.850 --> 00:08:14.629
what do we need to actually think about in order
00:08:14.629 --> 00:08:18.250
to say, hey, is this actually working? That is on
00:08:18.250 --> 00:08:21.649
top of the definition of success for the particular
00:08:21.649 --> 00:08:23.670
task that the agent is asked to do. So whether
00:08:23.670 --> 00:08:26.410
or not that's, you know, here's a ticket in Jira,
00:08:26.410 --> 00:08:28.910
go through this, the success criteria defined
00:08:28.910 --> 00:08:31.470
at the end, and when you meet all of that success
00:08:31.470 --> 00:08:35.440
criteria, plus you meet these testing success
00:08:35.440 --> 00:08:39.179
or fail points, then you're done. Then you can
00:08:39.179 --> 00:08:41.259
consider this shipped and you can hand it off
00:08:41.259 --> 00:08:44.299
agent to the next step. That's the kind of stuff
00:08:44.299 --> 00:08:46.940
we're thinking about. Makes sense. That's also
00:08:46.940 --> 00:08:48.700
assuming, I guess, the definition of done is
00:08:48.700 --> 00:08:53.580
well articulated in the ticket. Well, this is
00:08:53.580 --> 00:08:57.779
why I think agentic integration as a whole is
00:08:57.779 --> 00:08:59.340
making people very uncomfortable. And it's not
00:08:59.340 --> 00:09:01.220
just because of the AI bad conversation that
00:09:01.220 --> 00:09:03.799
people like to have. It's also the Whole thing
00:09:03.799 --> 00:09:05.740
with AI is it's forcing us to confront our own
00:09:05.740 --> 00:09:09.159
humanity in a very strange way. And that is a
00:09:09.159 --> 00:09:11.480
good example of it. The definition of done sucks.
00:09:11.980 --> 00:09:14.059
It doesn't matter whether or not it's a human
00:09:14.059 --> 00:09:17.220
or a machine. It still sucks. It's just the machine
00:09:17.220 --> 00:09:20.440
is a lot more capable of creating a lot more
00:09:20.440 --> 00:09:22.539
chaos than the human is because they can move
00:09:22.539 --> 00:09:25.539
that much quicker. When people say IDPs are becoming
00:09:25.539 --> 00:09:29.279
context layers for agents, what are they getting
00:09:29.279 --> 00:09:32.059
right? What they're getting right is that An
00:09:32.059 --> 00:09:38.980
IDP is an abstraction. And in a way, a context
00:09:38.980 --> 00:09:40.600
layer is an abstraction. It's an abstraction
00:09:40.600 --> 00:09:45.440
on knowledge and practice
and, you know, behavior
00:09:45.440 --> 00:09:49.259
expectation. And what IDPs are at their core
00:09:49.259 --> 00:09:54.679
is engineering behavior. The idea of the golden
00:09:54.679 --> 00:09:56.940
path as a whole is to direct human behavior in
00:09:56.940 --> 00:09:59.200
a certain way. And we can extend that path to
00:09:59.200 --> 00:10:03.330
agents to direct their behavior. The IDP is the
00:10:03.330 --> 00:10:05.590
foundation in which the agentic platform is built,
00:10:05.690 --> 00:10:07.710
regardless as to what direction your agentic
00:10:07.710 --> 00:10:09.649
platform goes or whether or not, you know, you're
00:10:09.649 --> 00:10:12.330
only bringing a few autonomous pieces in, really
00:10:12.330 --> 00:10:14.250
gradually scaling up, whatever that looks like.
00:10:14.929 --> 00:10:17.169
That IDP is the foundation and that has to be
00:10:17.169 --> 00:10:19.110
right. And it has to work for the human users
00:10:19.110 --> 00:10:21.750
in your organization because they are the ones
00:10:21.750 --> 00:10:23.529
that will determine whether or not it's built
00:10:23.529 --> 00:10:27.509
contextually, to overuse that word, correct for
00:10:27.509 --> 00:10:29.740
the organization. And the things that they have
00:10:29.740 --> 00:10:32.440
to do inside of it and their day to day. So that's
00:10:32.440 --> 00:10:36.419
like, it's partially right. And it kind of scratches,
00:10:36.500 --> 00:10:39.700
I think, at multiple layers. But there's a lot
00:10:39.700 --> 00:10:41.759
more to it. Are you seeing that they're missing
00:10:41.759 --> 00:10:44.740
on the flip side of that, conversely? Yeah. And
00:10:44.740 --> 00:10:46.039
that's a good question. This is the question
00:10:46.039 --> 00:10:49.440
I think everybody asks me. So that's one layer.
00:10:49.850 --> 00:10:52.250
in an agentic platform. And really that's sort
00:10:52.250 --> 00:10:53.769
of your tooling layer. It's your foundational
00:10:53.769 --> 00:10:56.350
substrate that everything else is built on. But
00:10:56.350 --> 00:10:58.769
there's other pieces too. So there's that agent
00:10:58.769 --> 00:11:00.629
infrastructure piece that I talked about earlier.
00:11:00.750 --> 00:11:03.169
So that harness, how the agent does the work,
00:11:03.190 --> 00:11:05.549
it's the governance piece, who's allowed, what's
00:11:05.549 --> 00:11:07.409
allowed to do it, what's safe, how is it watched.
00:11:07.929 --> 00:11:10.029
But then there's this other piece that's at the
00:11:10.029 --> 00:11:11.730
middle of these that allows them to interact.
00:11:11.909 --> 00:11:14.830
And this is the path specifications. So you can
00:11:14.830 --> 00:11:17.600
think of this as looking at golden paths and
00:11:17.600 --> 00:11:20.639
sort of, you know, 100x-ing them. What this
00:11:20.639 --> 00:11:23.919
really is, is the repeatable actions and the
00:11:23.919 --> 00:11:26.139
rules around when agents are called and when
00:11:26.139 --> 00:11:28.940
they aren't. So there's three kinds, realistically,
00:11:29.159 --> 00:11:32.100
that we are primarily thinking about. The first
00:11:32.100 --> 00:11:35.820
is that probabilistic set of paths that are very
00:11:35.820 --> 00:11:38.470
agent-driven. There's also the deterministic
00:11:38.470 --> 00:11:40.909
ones that are pipeline-driven. And then there's
00:11:40.909 --> 00:11:43.850
hybrids that are that agent and a gate kind of
00:11:43.850 --> 00:11:46.730
combined together. You know, if I think of a
00:11:46.730 --> 00:11:48.809
probabilistic path that's agent driven, that's
00:11:48.809 --> 00:11:50.950
an agent deciding what to do at each step where
00:11:50.950 --> 00:11:52.870
the output is fairly non-deterministic. So it's
00:11:52.870 --> 00:11:56.509
like reviewing a PR, posting comments against
00:11:56.509 --> 00:11:59.230
whatever your standards are that it has access
00:11:59.230 --> 00:12:01.549
to. And, you know, an easy one to start with
00:12:01.549 --> 00:12:04.149
is that PR review piece because it's fairly risk
00:12:04.149 --> 00:12:06.730
averse. It could just be read only. Doesn't touch
00:12:06.730 --> 00:12:09.090
anything. The deterministic piece is what we're
00:12:09.090 --> 00:12:10.990
fairly used to in platform engineering. That's
00:12:10.990 --> 00:12:12.470
pipeline driven. So it's something like a CI
00:12:12.470 --> 00:12:15.690
build. It runs same way every time it's called.
00:12:15.870 --> 00:12:18.850
There's no agent involved in its execution. Maybe
00:12:18.850 --> 00:12:21.429
it's triggered by an agent, but the path itself
00:12:21.429 --> 00:12:25.149
just, it runs. It's the way it is. You know,
00:12:25.149 --> 00:12:26.830
and it's incredibly predictable in that way.
00:12:27.250 --> 00:12:30.929
The hybrid would look more like a validate change
00:12:30.929 --> 00:12:33.529
type path where there's this agent and gate piece
00:12:33.529 --> 00:12:36.509
in there. So agent generates output. Let's say
00:12:36.509 --> 00:12:39.230
it's PR. It's the common one. But then there's
00:12:39.230 --> 00:12:41.570
that deterministic gate that validates it's a
00:12:41.570 --> 00:12:44.409
CI build, essentially. And if it fails, the agent
00:12:44.409 --> 00:12:47.210
has a retry loop it can go through. And let's
00:12:47.210 --> 00:12:48.909
just say at the stage you're at, a human review
00:12:48.909 --> 00:12:53.019
on that pass of the loop. It can't do the merge,
00:12:53.019 --> 00:12:56.980
the agent. The CI says whether or not it is appropriate
00:12:56.980 --> 00:12:58.779
to be merged, and the human does the final deciding.
00:12:58.779 --> 00:13:02.279
But that's a hybrid path at its core. So those
00:13:02.279 --> 00:13:04.480
are the three layers that we really need to think
00:13:04.480 --> 00:13:07.320
about. The IDP is just a piece. So is that agent
00:13:07.320 --> 00:13:10.000
infrastructure. But it's these paths that have
00:13:10.000 --> 00:13:13.159
to be almost redesigned with agents
in mind.
00:13:13.200 --> 00:13:16.000
Sometimes you take the human path and you're
00:13:16.000 --> 00:13:18.159
augmenting it and you're also making sure it
00:13:18.159 --> 00:13:20.879
can work at much faster speeds and in parallel
00:13:20.879 --> 00:13:23.980
at the same time. How do you keep an agent from
00:13:23.980 --> 00:13:26.679
just looping forever or optimizing the wrong
00:13:26.679 --> 00:13:30.059
thing in that model? Yeah. So that's a good question
00:13:30.059 --> 00:13:32.940
too. The really important thing is setting limits.
00:13:33.580 --> 00:13:37.620
If you saw a developer that was just going and
00:13:38.860 --> 00:13:42.940
repushing a thousand PRs with similar messaging
00:13:42.940 --> 00:13:44.659
and eventually get frustrated and their messaging
00:13:44.659 --> 00:13:46.720
says things like, please work, oh my God, make
00:13:46.720 --> 00:13:50.379
it stop. You'd step in and you'd say, whoa, whoa,
00:13:50.419 --> 00:13:53.559
whoa, let's look at what we need to change. Same
00:13:53.559 --> 00:13:55.700
thing with an agent, but put a limit in, put
00:13:55.700 --> 00:13:58.440
in three to five, I would say is your magic number
00:13:58.440 --> 00:14:00.919
to start with. Why is it the magic number? I
00:14:00.919 --> 00:14:03.240
don't really know, just seems to work. I think
00:14:03.240 --> 00:14:07.600
humans just like those numbers allow the retries
00:14:07.600 --> 00:14:10.799
to also be learning experiences. So you can make
00:14:10.799 --> 00:14:12.940
sure when you're setting it up that each retry
00:14:12.940 --> 00:14:15.120
gives that agent information that it can learn
00:14:15.120 --> 00:14:18.220
from, or that, you know, comes back as feedback
00:14:18.220 --> 00:14:20.120
to the human that's built this platform to say,
00:14:20.200 --> 00:14:23.879
hey, maybe the kind of agent infrastructure layer
00:14:23.879 --> 00:14:25.960
needs to be adjusted here, or for lack of better
00:14:25.960 --> 00:14:27.620
terms, your system prompt needs reconfigured.
00:14:27.759 --> 00:14:30.840
There's something not quite right. Because often...
00:14:31.460 --> 00:14:34.460
The failure is a learning experience for either
00:14:34.460 --> 00:14:36.759
the human or the machine, and you might as well
00:14:36.759 --> 00:14:38.960
take advantage of that. So what should the platform
00:14:38.960 --> 00:14:43.299
score or evaluate in that loop, in that process?
00:14:43.940 --> 00:14:48.960
Yeah, it does depend on the path type. So let's
00:14:48.960 --> 00:14:50.779
just look specifically at a probabilistic path.
00:14:51.259 --> 00:14:55.139
What we really want to be evaluating here is
00:14:55.139 --> 00:15:00.480
the decisions that the agent took. Maybe how
00:15:00.480 --> 00:15:02.740
many decisions it had to take, how many tokens
00:15:02.740 --> 00:15:06.960
it used in order to do this, whether like how
00:15:06.960 --> 00:15:09.659
many retry loops it had to go through in order
00:15:09.659 --> 00:15:13.460
to get to that definition of success. And, you
00:15:13.460 --> 00:15:15.139
know, there's time components here. You could
00:15:15.139 --> 00:15:17.399
be looking at costs as well as tokens. You can
00:15:17.399 --> 00:15:21.340
look at it either way. And honestly, just kind
00:15:21.340 --> 00:15:24.320
of the measure of success, the measure of the
00:15:24.320 --> 00:15:27.559
behavior that the human's job at this stage becomes
00:15:27.559 --> 00:15:30.220
monitoring behavior and monitoring outcomes.
00:15:30.820 --> 00:15:32.940
You know, one thing I'll talk to teams about
00:15:32.940 --> 00:15:35.440
quite a bit is if you're running an agent constantly
00:15:35.440 --> 00:15:37.299
and it's always doing the same thing with the
00:15:37.299 --> 00:15:39.419
same outcome, maybe you need to revisit whether
00:15:39.419 --> 00:15:41.679
or not an agent is appropriate there. So you
00:15:41.679 --> 00:15:45.200
need to keep track of outcomes. Do you have to
00:15:45.200 --> 00:15:48.799
be calling through a crazy amount of tokens every
00:15:48.799 --> 00:15:51.220
single time you run this when it's a fairly simple
00:15:51.220 --> 00:15:53.720
process that's readily executed in the same way?
00:15:54.080 --> 00:15:56.379
You could probably reduce the number of tokens
00:15:56.379 --> 00:15:59.179
you're using and get the agent to be a lot more
00:15:59.179 --> 00:16:02.460
checklisty because you have it so standardized.
00:16:02.740 --> 00:16:05.080
And that's the things to watch. You know, if
00:16:05.080 --> 00:16:07.500
you're always getting the same output, you're
00:16:07.500 --> 00:16:09.480
probably wasting money if you're just letting
00:16:09.480 --> 00:16:11.840
an agent, you know, kind of free reign through
00:16:11.840 --> 00:16:13.820
everything. So we talked a little bit earlier
00:16:13.820 --> 00:16:18.120
about governance. So one thing that I've definitely
00:16:18.120 --> 00:16:21.740
realized in the industry is platform teams already
00:16:21.740 --> 00:16:24.759
struggle with governance when humans are doing
00:16:24.759 --> 00:16:28.000
the work. How do we... change that with agents?
00:16:28.159 --> 00:16:29.980
And it seems like it gets harder with agents,
00:16:30.159 --> 00:16:32.299
at least in my opinion, what I've come across.
00:16:32.440 --> 00:16:34.860
How do we handle that? Yeah, it absolutely does
00:16:34.860 --> 00:16:37.159
get harder. And I think it really comes down
00:16:37.159 --> 00:16:41.039
to the reason why is because we're confronting
00:16:41.039 --> 00:16:44.000
our own lack of trust in ourselves and in a system
00:16:44.000 --> 00:16:46.379
we've built. Fundamentally, at the end of the
00:16:46.379 --> 00:16:49.779
day, platform engineering, regardless of whether
00:16:49.779 --> 00:16:53.559
or not there's agents in this loop, is a practice
00:16:53.559 --> 00:16:56.220
about building trust. It's organizational trust
00:16:56.220 --> 00:16:58.799
in the system. It's organizational trust in the
00:16:58.799 --> 00:17:00.840
people who have access to the system or the actors
00:17:00.840 --> 00:17:03.700
who have access to it. So that foundation has
00:17:03.700 --> 00:17:06.400
to be good and it has to be trustworthy. And,
00:17:06.440 --> 00:17:10.500
you know, I, in the last three weeks alone, have
00:17:10.500 --> 00:17:12.779
told multiple platform teams to pump the brakes
00:17:12.779 --> 00:17:15.640
on putting an agent in because they can't even
00:17:15.640 --> 00:17:18.559
get human governance correct. They are still
00:17:18.559 --> 00:17:21.420
struggling. And this is where, you know. We'll
00:17:21.420 --> 00:17:24.099
really sit down and understand where this is
00:17:24.099 --> 00:17:25.819
happening. Where is this trust breaking down?
00:17:25.940 --> 00:17:28.720
And is it just because we haven't codified the
00:17:28.720 --> 00:17:32.200
governance yet, that identity security and observability
00:17:32.200 --> 00:17:35.799
piece? Or is it unclear what you're supposed
00:17:35.799 --> 00:17:37.799
to be doing? In which case you have to work with
00:17:37.799 --> 00:17:39.660
security. You have to work with the governance
00:17:39.660 --> 00:17:41.960
pieces of your org to really make sure you're
00:17:41.960 --> 00:17:45.220
getting this right. And, you know, if their trust
00:17:45.220 --> 00:17:46.900
in the system isn't there and they're able to
00:17:46.900 --> 00:17:49.200
sort of pull the plug on this stuff, that's where
00:17:49.200 --> 00:17:52.359
you need to focus. We focus so much on developer
00:17:52.359 --> 00:17:54.259
experience, we forget that there are other stakeholders
00:17:54.259 --> 00:17:56.420
we have to care about. And that often is governance
00:17:56.420 --> 00:17:59.759
security teams, you know, having to feel like
00:17:59.759 --> 00:18:01.700
they have a say in how these platforms are designed.
00:18:02.160 --> 00:18:05.700
And yeah, it gets even messier when agents come
00:18:05.700 --> 00:18:07.619
in, especially if that foundational trust just
00:18:07.619 --> 00:18:11.160
isn't there. How can you trust a machine, multiple
00:18:11.160 --> 00:18:14.049
machines in parallel, if you can't even trust
00:18:14.049 --> 00:18:17.470
a person to go through a process without someone
00:18:17.470 --> 00:18:20.099
standing over their shoulder constantly? So how
00:18:20.099 --> 00:18:22.619
should teams think about agent identity versus
00:18:22.619 --> 00:18:27.119
human identity? Yeah, I don't often think of
00:18:27.119 --> 00:18:31.180
them that differently. The reason why, I think,
00:18:31.240 --> 00:18:35.660
is because it does separate that trust a little
00:18:35.660 --> 00:18:38.019
bit. If you trust your system to allow humans
00:18:38.019 --> 00:18:41.539
to do the right kind of thing, agents can probably
00:18:41.539 --> 00:18:43.819
work within the same boundaries. The difference
00:18:43.819 --> 00:18:46.099
is the parallelization piece, so you can have
00:18:46.099 --> 00:18:50.150
multiple running at the same time. And in that
00:18:50.150 --> 00:18:53.049
case, my recommendation is always, you're not
00:18:53.049 --> 00:18:55.230
looking at generalists, you're looking at specialists.
00:18:55.569 --> 00:18:58.069
So you need to constrain around that speciality.
00:19:00.190 --> 00:19:05.170
Rather than, we used to constrain a lot, or not constrain a lot, we used to split people into, you were a front
00:19:05.170 --> 00:19:07.150
-end engineer, you were a back-end engineer.
00:19:07.369 --> 00:19:09.670
Never shall the two meet. We probably spent a
00:19:09.670 --> 00:19:11.170
good 10 years doing this when it was unnecessary.
00:19:11.609 --> 00:19:16.890
Full stack is very useful. And that's really
00:19:16.890 --> 00:19:18.829
not how we necessarily want agents to think,
00:19:18.869 --> 00:19:20.670
especially when we're trying to do this in the
00:19:20.670 --> 00:19:23.470
enterprise at large scale. We want to be specialized.
00:19:23.769 --> 00:19:28.109
Have an agent for Terraform specifically, maybe
00:19:28.109 --> 00:19:32.609
inside of this particular product team. Don't
00:19:32.609 --> 00:19:35.809
allow the scope to go beyond that. It's way easier
00:19:35.809 --> 00:19:37.950
to control the smaller the scope is, and it's
00:19:37.950 --> 00:19:40.779
also way easier to monitor. And, you know, you
00:19:40.779 --> 00:19:43.420
can't control what you can't see and what you
00:19:43.420 --> 00:19:46.450
can't understand. So the smaller the chunk with
00:19:46.450 --> 00:19:48.930
identity, the easier it is to manage. But start
00:19:48.930 --> 00:19:50.930
with the foundational layer you have for humans.
00:19:51.029 --> 00:19:53.529
You just have to augment it. So you probably
00:19:53.529 --> 00:19:55.710
have RBAC in place already. You probably have
00:19:55.710 --> 00:19:57.450
some level of role binding or an inheritance
00:19:57.450 --> 00:20:01.849
pattern in place. Base it off that just now into
00:20:01.849 --> 00:20:04.349
the specificity of whatever that agent's specialty
00:20:04.349 --> 00:20:08.490
is. Do you have like the same like time bound
00:20:08.490 --> 00:20:12.829
access that you do for humans or? I don't think
00:20:12.829 --> 00:20:15.130
there's a standard set for this yet. And it depends
00:20:15.130 --> 00:20:18.549
on the context. Like I'm a big fan of lower environments,
00:20:18.630 --> 00:20:21.509
a little more relaxed. It's also where you should
00:20:21.509 --> 00:20:23.390
start really in terms of building in governance
00:20:23.390 --> 00:20:26.450
and security because it's easier to codify those
00:20:26.450 --> 00:20:28.490
standards in a more relaxed environment and then
00:20:28.490 --> 00:20:32.250
harden them as you go. But yeah, I would approach
00:20:32.250 --> 00:20:35.650
it at the same kind of way you approach the human
00:20:35.650 --> 00:20:37.569
side of things, just obviously considering that
00:20:37.569 --> 00:20:40.609
these things can move quicker. If you're relying
00:20:40.609 --> 00:20:42.690
on an external model, there's always the questions
00:20:42.690 --> 00:20:46.829
of what happens if that model is somehow compromised.
00:20:47.369 --> 00:20:49.569
And obviously that happens internally too with
00:20:49.569 --> 00:20:52.250
internally hosted models, but it's that sort
00:20:52.250 --> 00:20:54.410
of thing you have to consider. But time constraint,
00:20:54.549 --> 00:20:56.410
definitely have it in place. You can make it
00:20:56.410 --> 00:20:57.910
shorter. It's not the end of the world. These
00:20:57.910 --> 00:21:02.250
things run quickly. But start somewhere, measure
00:21:02.250 --> 00:21:04.829
and see how you feel about things and evaluate
00:21:04.829 --> 00:21:06.829
the organizational trust in those things at that
00:21:06.829 --> 00:21:10.119
base level and then make changes as you need
00:21:10.119 --> 00:21:14.180
to. So we talked about agents involved in generating
00:21:14.180 --> 00:21:17.359
code or being on the developer side of things.
00:21:17.579 --> 00:21:22.559
How do you view using agents on the SRE incident
00:21:22.559 --> 00:21:27.279
response side of a platform in an IDP? What are
00:21:27.279 --> 00:21:29.579
your thoughts on that? It's one of my favorite
00:21:29.579 --> 00:21:32.940
places to put these things because you have the
00:21:32.940 --> 00:21:35.579
ability to have a very powerful, essentially
00:21:35.579 --> 00:21:38.880
like emergency dispatch system or early warning
00:21:38.880 --> 00:21:43.500
system. You know, a really good client of mine
00:21:43.500 --> 00:21:47.279
talked about how they've brought an agent into
00:21:47.279 --> 00:21:50.059
kind of their log ingestion and they're getting
00:21:50.059 --> 00:21:52.039
terabytes of logs per day. This is one of the
00:21:52.039 --> 00:21:53.819
biggest financial services companies in the US.
00:21:54.200 --> 00:21:56.900
Terabytes of logs. I don't think a human can
00:21:56.900 --> 00:22:01.019
go through that, but their agents do. And they're
00:22:01.019 --> 00:22:03.640
able to recognize potential problems before they
00:22:03.640 --> 00:22:07.559
even surface. And they don't do auto remediation
00:22:07.559 --> 00:22:10.400
just yet. I mean, yeah, they aren't quite there
00:22:10.400 --> 00:22:12.960
yet. You know, we're still working with them
00:22:12.960 --> 00:22:15.279
to build in the foundations of what auto remediation
00:22:15.279 --> 00:22:18.259
needs to look like. But it's making sure that
00:22:18.259 --> 00:22:20.380
it's cataloging the issue. It's alerting the
00:22:20.380 --> 00:22:23.819
right people in the right channels at the right
00:22:23.819 --> 00:22:28.940
time. So don't go send a PagerDuty page to Bob
00:22:28.940 --> 00:22:34.019
who's on PTO. Send it to Dave who's in the office
00:22:34.019 --> 00:22:37.190
and on call right now. And it knows that. And
00:22:37.190 --> 00:22:39.630
that's really helped them get ahead of some really
00:22:39.630 -->
00:22:42.369
large issues. But also when there is something
00:22:42.369 --> 00:22:44.730
that is major happening that maybe wasn't seen
00:22:44.730 --> 00:22:47.589
or it's kind of that instant everything's on
00:22:47.589 --> 00:22:50.089
fire, they have a much cleaner picture of where
00:22:50.089 --> 00:22:52.349
it is. And they're not sifting through terabytes
00:22:52.349 --> 00:22:54.730
of logs to decide what's relevant and what isn't.
00:22:55.069 --> 00:22:57.130
Agents are fairly able to recognize those patterns
00:22:57.130 --> 00:23:01.089
based on how they've built that context layer,
00:23:01.289 --> 00:23:03.630
that governance layer, that agent architecture
00:23:03.630 --> 00:23:06.160
around that get them the information they need.
00:23:06.299 --> 00:23:10.180
Yeah. I think identifying the gaps in metrics
00:23:10.180 --> 00:23:12.220
or alarms that you have is just as important
00:23:12.220 --> 00:23:15.339
as identifying the alert fatigue patterns that
00:23:15.339 --> 00:23:19.240
you may have. And yeah, I'm sure agents can help
00:23:19.240 --> 00:23:21.880
with both of those. Yeah. And an agent fatigue
00:23:21.880 --> 00:23:25.619
pattern, this is something your base platform
00:23:25.619 --> 00:23:29.140
should be handling too. If you want to do platform
00:23:29.140 --> 00:23:31.220
engineering right and really make your SREs have
00:23:31.220 --> 00:23:35.490
good days, or anybody that has to see 6,000
00:23:35.490 --> 00:23:37.150
alerts that don't mean anything every single
00:23:37.150 --> 00:23:40.809
day, consider how the platform can help that.
00:23:41.329 --> 00:23:44.390
Because you can make sure things are routed.
00:23:44.430 --> 00:23:47.329
You can make sure things are relevant to that
00:23:47.329 --> 00:23:51.549
person's context window. You could probably get
00:23:51.549 --> 00:23:53.730
my opinion on what I think about AIs in terms
00:23:53.730 --> 00:23:58.829
of how they exist in the world. But it's really
00:23:58.829 --> 00:24:00.170
important to address that for humans. Because
00:24:00.170 --> 00:24:02.150
alert fatigue is real. I cannot tell you the
00:24:02.150 --> 00:24:04.779
amount of times, even on my own stuff, And I'm
00:24:04.779 --> 00:24:06.519
probably the worst for actually fixing this because
00:24:06.519 --> 00:24:09.180
it's just me on my own projects that I get an
00:24:09.180 --> 00:24:11.180
alert. I'm like, yeah, whatever. And then I'll
00:24:11.180 --> 00:24:14.099
get a bunch of users pinging me like, dude, everything's
00:24:14.099 --> 00:24:19.640
broken. So fix that for the humans first. Okay,
00:24:19.680 --> 00:24:22.960
so I guess speaking to the human side, do platform
00:24:22.960 --> 00:24:26.059
teams need to focus more like on the design for
00:24:26.059 --> 00:24:29.500
agents as another kind of user? Or philosophically,
00:24:29.500 --> 00:24:30.920
do we think about that differently when we're
00:24:30.920 --> 00:24:34.150
designing the platform? I love that
question
00:24:34.150 --> 00:24:37.809
because it's really the foundational thesis of
00:24:37.809 --> 00:24:40.630
what I'm talking about right now. What we're
00:24:40.630 --> 00:24:42.869
talking about, what we're writing books about,
00:24:43.069 --> 00:24:45.490
conference main stage talks and everything. And
00:24:45.490 --> 00:24:48.650
it is that at the end of the day, if your underlying
00:24:48.650 --> 00:24:52.930
platform, that IDP is not strong, is not mature,
00:24:53.150 --> 00:24:57.089
is not delivering value, is not actually reducing
00:24:57.089 --> 00:25:00.910
toil, isn't adopted, good luck putting an agent
00:25:00.910 --> 00:25:03.779
in there, or any, you know, kind of AI at all. And
00:25:03.779 --> 00:25:07.460
don't. I wouldn't do it, honestly. So you need to
00:25:07.460 --> 00:25:09.880
get that foundational design for humans first.
00:25:09.880 --> 00:25:16.220
Then your design has to shift to really thinking
00:25:16.220 --> 00:25:20.920
about codifying the things that make us understand
00:25:20.920 --> 00:25:24.339
how to do our job. You know, that's that's that
00:25:24.339 --> 00:25:26.920
part of that agent infrastructure piece. It's
00:25:26.920 --> 00:25:29.339
so important, and it's something that I've been
00:25:29.339 --> 00:25:32.049
talking a lot about recently, is like, you have
00:25:32.049 --> 00:25:34.509
to now codify every single assumption you make
00:25:34.509 --> 00:25:38.960
as an organization. Every single cultural piece
00:25:38.960 --> 00:25:41.460
of working on that team. Like, I know I need
00:25:41.460 --> 00:25:44.960
to go talk to Andrew because he knows exactly
00:25:44.960 --> 00:25:46.799
how to write Terraform and I'm an idiot at it.
00:25:47.319 --> 00:25:50.059
It's like, I keep breaking it. Or I know that
00:25:50.059 --> 00:25:51.900
if I need this information, I jump into Confluence
00:25:51.900 --> 00:25:56.549
to go get it. Or the assumption that I'm a junior
00:25:56.549 --> 00:25:58.289
engineer, so I'm not going to get all the...
00:25:58.690 --> 00:26:00.710
really, you know, hardcore tickets. I'm going to
00:26:00.710 --> 00:26:04.309
get kind of the grunt work. Those are things that
00:26:04.309 --> 00:26:08.309
we've never written down, generally, and now we
00:26:08.309 --> 00:26:11.630
have to. So it does, often. One of the most interesting
00:26:11.630 --> 00:26:13.950
things I think from like a psychological perspective
00:26:13.950 --> 00:26:16.089
to see is teams are now going through this process
00:26:16.089 --> 00:26:20.519
of doing anthropology on their own organizations
00:26:20.519 --> 00:26:24.180
and writing, like, an ethnography on, you know, how
00:26:24.180 --> 00:26:27.059
they work, how they approach problems. And having
00:26:27.059 --> 00:26:30.559
to document that's really hard. So that is the
00:26:30.559 --> 00:26:32.740
next step. But honestly, it's a good exercise
00:26:32.740 --> 00:26:35.119
to do regardless because it'll make your platform
00:26:35.119 --> 00:26:38.319
better for humans, but also make it the right
00:26:38.319 --> 00:26:41.140
place to throw agents. Yeah, I think anyone that's
00:26:41.140 --> 00:26:44.509
onboarded with a new company just realize it might
00:26:44.509 --> 00:26:46.990
even be outside of the engineering organization,
00:26:47.130 --> 00:26:48.869
but there's this tribal knowledge that like,
00:26:48.890 --> 00:26:51.349
what does this acronym mean? You know, you have
00:26:51.349 --> 00:26:52.809
this imposter syndrome where it's like, wait, is
00:26:52.809 --> 00:26:54.589
this an acronym I should know? No, this is... I don't
00:26:54.589 --> 00:26:56.950
want to ask someone. Yeah, yeah, no, this is an internal
00:26:56.950 --> 00:27:00.309
acronym that, you know, it's just tribal knowledge
00:27:00.309 --> 00:27:02.990
that people have. Or even just like, how, who do
00:27:02.990 --> 00:27:05.829
you go to when you have a question about something
00:27:05.829 --> 00:27:09.410
specific? Or, well, I need access to this, but you
00:27:09.410 --> 00:27:11.650
got to go ask, like, John in finance. He'll get
00:27:11.650 --> 00:27:13.589
that for you. Well, who was going to tell me that
00:27:13.589 --> 00:27:17.140
until I ran into it? Um... Yeah, like just think
00:27:17.140 --> 00:27:19.299
about onboarding a new human. How long does it
00:27:19.299 --> 00:27:20.819
take to get them comfortable and actually doing
00:27:20.819 --> 00:27:22.480
stuff? But there's things you don't even think
00:27:22.480 --> 00:27:25.740
about in that too. So it sounds like engineers
00:27:25.740 --> 00:27:29.880
trusting agents less. It's not so much because
00:27:29.880 --> 00:27:32.079
they're risky, but maybe because the platform
00:27:32.079 --> 00:27:34.180
around them is unclear. Does that sound right?
00:27:34.500 --> 00:27:36.440
I mean, there is going to be that element of
00:27:36.440 --> 00:27:39.440
humans not trusting AI. And you know what? I
00:27:39.440 --> 00:27:43.400
think it's valid. But if they trust the platform,
00:27:44.160 --> 00:27:46.420
to have constraints because they've seen it with
00:27:46.420 --> 00:27:49.099
people they've seen it not only help people work
00:27:49.099 --> 00:27:51.619
faster but work safer they're going to feel more
00:27:51.619 --> 00:27:53.619
comfortable about agents coming into that system
00:27:53.619 --> 00:27:58.099
and working within the same guardrails and working
00:27:58.099 --> 00:28:00.839
within the same patterns the patterns are clear
00:28:00.839 --> 00:28:06.579
they're observable and they are value generating
00:28:06.579 --> 00:28:08.980
to begin with and it's clear to see how they
00:28:08.980 --> 00:28:11.799
do that I try to avoid saying value generating
00:28:11.799 --> 00:28:14.000
for a lot of engineering conversations because
00:28:14.000 --> 00:28:15.900
often the eyes start glazing over and they look
00:28:15.900 --> 00:28:18.059
at me like, how dare you? But it's true. It's
00:28:18.059 --> 00:28:22.640
what we're doing. And it really is important
00:28:22.640 --> 00:28:25.660
to start thinking that way when you want to go
00:28:25.660 --> 00:28:29.480
down the agentic route anyway. Yeah, we have
00:28:29.480 --> 00:28:32.519
to build that foundation and we have to have
00:28:32.519 --> 00:28:35.559
that foundational trust there before we can introduce
00:28:35.559 --> 00:28:40.660
something that runs a lot faster. Before you
00:28:40.660 --> 00:28:42.400
can ride a bike, you start with training wheels,
00:28:42.400 --> 00:28:45.579
then you take the training wheels off. Look, now
00:28:45.579 --> 00:28:47.500
you're steady enough and stable enough, you can
00:28:47.500 --> 00:28:49.460
go mountain biking, or you can get on a race bike,
00:28:49.460 --> 00:28:52.200
or, um, heck, you're crazy enough, you want to go
00:28:52.200 --> 00:28:55.180
do motocross or something. You know, it's it's
00:28:55.180 --> 00:28:58.019
you don't just start by jumping in the Formula
00:28:58.019 --> 00:29:02.539
One car. Okay, so I'm a platform engineer. I have
00:29:02.539 --> 00:29:05.140
an IDP. I'm running Backstage. I have a service
00:29:05.140 --> 00:29:09.289
catalog. I, I've been told by my director of engineering
00:29:09.289 --> 00:29:12.210
that I need to have an agentic workflows built
00:29:12.210 --> 00:29:15.609
into my platform. Where do I start? What do I
00:29:15.609 --> 00:29:19.369
start with? First thing you do is ask why. It's
00:29:19.369 --> 00:29:21.710
like 90% of my job, I think, is telling people
00:29:21.710 --> 00:29:24.630
they need to ask why more. Okay, why? What are
00:29:24.630 --> 00:29:26.809
we trying to achieve by doing this? Are we doing
00:29:26.809 --> 00:29:28.029
AI for the sake of AI? Engineers move faster,
00:29:28.089 --> 00:29:30.289
velocity, right? That's the answer I get, right?
00:29:30.509 --> 00:29:33.769
That's better than just because someone on LinkedIn
00:29:33.769 --> 00:29:36.130
said I should, which believe it or not happens.
00:29:36.769 --> 00:29:38.690
You know, okay, so we've established why. We
00:29:38.690 --> 00:29:41.849
want velocity. So we have a platform. It's mature.
00:29:41.970 --> 00:29:45.529
It's adopted. Why are we talking about velocity
00:29:45.529 --> 00:29:47.470
being slow? Is it just because we want to unlock
00:29:47.470 --> 00:29:50.450
the potential of AI? Then great. We will look
00:29:50.450 --> 00:29:52.750
at the platform that we currently have in front
00:29:52.750 --> 00:29:55.210
of us and say, what makes the most sense for
00:29:55.210 --> 00:29:58.789
an agent? And this is where there's three paths
00:29:58.789 --> 00:30:00.789
I want people to usually start with here. And
00:30:00.789 --> 00:30:03.809
the first is that... very, very probabilistic
00:30:03.809 --> 00:30:07.690
agent-driven PR review task where it's that
00:30:07.690 --> 00:30:10.250
read-only output. You know, they go through,
00:30:10.490 --> 00:30:14.430
they add comments based on
the standards of the
00:30:14.430 --> 00:30:18.009
organization, your ADRs, whatever you have in
00:30:18.009 --> 00:30:21.190
place. There's no modification of code happening,
00:30:21.269 --> 00:30:23.130
but that is the first place to start because
00:30:23.130 --> 00:30:28.259
let's be honest, code review is a... very time
00:30:28.259 --> 00:30:31.279
consuming process. A lot of the time, it's one
00:30:31.279 --> 00:30:34.640
of those things where I have 25 PRs to review
00:30:34.640 --> 00:30:37.880
and I'll get to them when I get to them. Or I
00:30:37.880 --> 00:30:39.559
have to set aside a specific day to go ahead
00:30:39.559 --> 00:30:41.559
and do that. But what if that PR could happen
00:30:41.559 --> 00:30:45.480
really quickly, like within a couple minutes?
00:30:45.940 --> 00:30:49.000
That would speed us up exponentially. So that's
00:30:49.000 --> 00:30:51.019
a nice one to do first because to be honest,
00:30:51.119 --> 00:30:54.400
it's very low risk. You as a human can go in
00:30:54.400 --> 00:30:56.420
and make sure that what it is saying, it's not
00:30:56.420 --> 00:30:58.319
hallucinating, that it is correct against the
00:30:58.319 --> 00:31:00.819
standards that you have. You can review the behavior.
00:31:01.579 --> 00:31:04.140
But at the end of the day, like that's still
00:31:04.140 --> 00:31:06.180
way faster because if it's let's say it's four
00:31:06.180 --> 00:31:08.579
or five hundred lines of code, that PR review
00:31:08.579 --> 00:31:12.019
would take a human a decent amount of time. It's
00:31:12.019 --> 00:31:14.200
also a good exercise to make sure that teams
00:31:14.200 --> 00:31:19.519
are doing PRs in a way that they are able to
00:31:19.519 --> 00:31:21.640
explain the code that they've generated, whether
00:31:21.640 --> 00:31:24.759
or not they've had. Claude do it for them you
00:31:24.759 --> 00:31:26.720
know I think you still need to be able to say
00:31:26.720 --> 00:31:29.339
this is the thing that I've built or that my
00:31:29.339 --> 00:31:32.279
agent and I have built and this is why it's important
00:31:32.279 --> 00:31:34.839
to this ticket and this is why we think it's
00:31:34.839 --> 00:31:38.299
the right approach. So I think that's a good place
00:31:38.299 --> 00:31:42.299
to start. After that, you know, you could really
00:31:42.299 --> 00:31:44.579
start looking at that hybrid piece, so that validate
00:31:44.579 --> 00:31:47.759
change component, once you're comfortable with
00:31:47.759 --> 00:31:51.539
that PR review on its own. So, you know, the agent
00:31:51.539 --> 00:31:55.200
does its PR, its own PR, sorry, and then there's
00:31:55.200 --> 00:31:58.420
some sort of, um, CI or deterministic gate of another
00:31:58.420 --> 00:32:00.519
type that's called to validate it, and there's
00:32:00.519 --> 00:32:02.380
that retry loop until it gets it right. That's
00:32:02.380 --> 00:32:04.480
another one to start with because I think it's
00:32:04.480 --> 00:32:10.769
quite safe. Um, the, you know, the the baseline you
00:32:10.769 --> 00:32:14.630
can move from with both of those is very strong.
00:32:14.630 --> 00:32:16.730
You know, have a human review the evidence that
00:32:16.730 --> 00:32:19.190
it was successful. And then once you're happy
00:32:19.190 --> 00:32:22.130
with that, go into security scanning. Then, hey,
00:32:22.170 --> 00:32:24.150
we're confident running this in dev all the time,
00:32:24.309 --> 00:32:27.910
maybe even staging. That's an easy velocity game.
00:32:28.049 --> 00:32:31.509
And it's a fairly quick one to put in place as
00:32:31.509 --> 00:32:33.109
long as you have the foundational platform there.
00:32:33.410 --> 00:32:36.970
Makes sense. So you had recently talked at PlatformCon.
00:32:37.589 --> 00:32:39.349
Is there anything you'd like to leave our audience
00:32:39.349 --> 00:32:41.950
with? Any thoughts from that conference or your
00:32:41.950 --> 00:32:44.509
talks in general? Yeah, I think there's two pieces.
00:32:45.069 --> 00:32:48.470
And it's at the end of the day, the kind of overarching
00:32:48.470 --> 00:32:50.410
story is that as a platform engineer, you're
00:32:50.410 --> 00:32:53.369
so valuable to your organization, technically,
00:32:53.549 --> 00:32:55.650
not just when it comes to building that IDP,
00:32:55.690 --> 00:32:57.490
but building that agentic development platform
00:32:57.490 --> 00:33:01.829
or that agentic finance, marketing, communications
00:33:01.829 --> 00:33:04.170
platform, whatever it might be, because you are
00:33:04.170 --> 00:33:09.859
good at taking requirements and translating them
00:33:09.859 --> 00:33:13.900
into value and being able to talk between different
00:33:13.900 --> 00:33:16.720
traditionally siloed teams and build a product
00:33:16.720 --> 00:33:19.440
that actually works for them. So that's kind
00:33:19.440 --> 00:33:21.720
of the overarching thing, but there's two really
00:33:21.720 --> 00:33:24.299
key components here. And the first one is, remember
00:33:24.299 --> 00:33:28.480
this, and this is a quote, actually. Person in
00:33:28.480 --> 00:33:30.000
one of the workshops I did at PlatformCon New
00:33:30.000 --> 00:33:32.500
York this year said, and it's possibly my favorite
00:33:32.500 --> 00:33:33.839
thing I've heard in a long time, and it's that
00:33:33.839 --> 00:33:37.480
if we can't even write a readme.md file for
00:33:37.480 --> 00:33:40.119
our repos, how the hell are we ever going to
00:33:40.119 --> 00:33:43.380
write an agent.md for our agents? So keep that
00:33:43.380 --> 00:33:46.859
in mind. You know, that is so important foundationally
00:33:46.859 --> 00:33:49.119
to what we're doing here. And the other piece
00:33:49.119 --> 00:33:53.259
that I just want to kind of give people is that,
00:33:53.259 --> 00:33:58.369
you know, change is scary sometimes that scary
00:33:58.369 --> 00:34:00.210
change puts us on the precipice of something
00:34:00.210 --> 00:34:03.690
really
special. And I think this is that time
00:34:03.690 --> 00:34:06.250
for platform engineers to be at that gateway
00:34:06.250 --> 00:34:10.769
into something really impactful. And AI has a
00:34:10.769 --> 00:34:13.690
lot of ethical questions around it and has a
00:34:13.690 --> 00:34:16.889
lot of pluses and minuses, but you're in the
00:34:16.889 --> 00:34:20.130
place where you can actually not just implement
00:34:20.130 --> 00:34:22.809
this cool technology, but build something really
00:34:22.809 --> 00:34:25.710
powerful. And that is this base of knowledge.
00:34:25.710 --> 00:34:31.030
This is putting on paper the culture of your
00:34:31.030 --> 00:34:33.329
organization, and it it won't just help agents,
00:34:33.329 --> 00:34:36.269
it will help the people that work there. I promise
00:34:36.269 --> 00:34:39.409
you it will. So yeah, that's like, that's my big
00:34:39.409 --> 00:34:41.829
things right now. And and I'm really excited in
00:34:41.829 --> 00:34:43.989
this space. I think it's really, really exciting
00:34:43.989 --> 00:34:46.150
to be a part of and to be able to speak about
00:34:46.150 --> 00:34:48.090
it and work with teams directly on what they're
00:34:48.090 --> 00:34:51.090
doing, um, and kind of everything I say comes from
00:34:51.090 --> 00:34:53.650
doing that and actually working with organizations
00:34:53.650 --> 00:34:56.429
so I'm not just kind of making it up as I go
00:34:56.429 --> 00:34:59.630
along. Well said. Where can people find out more
00:34:59.630 --> 00:35:02.349
about you, follow you, that sort of thing? Yeah,
00:35:02.369 --> 00:35:03.989
so just find me on LinkedIn, sort of the easiest
00:35:03.989 --> 00:35:05.829
place to find me. And I tend to post a lot of
00:35:05.829 --> 00:35:08.550
stuff there, whether I'm writing on Substack,
00:35:08.630 --> 00:35:10.929
stuff on YouTube, podcasts like this one, the
00:35:10.929 --> 00:35:13.769
stuff my colleagues are working on. So some colleagues
00:35:13.769 --> 00:35:16.369
of mine recently. Released a book, which is called
00:35:16.369 --> 00:35:17.869
Thinking in Platforms, and it's really about
00:35:17.869 --> 00:35:20.110
taking platform engineering as an ideology and
00:35:20.110 --> 00:35:23.269
expanding it beyond software, outside of IT even,
00:35:23.409 --> 00:35:26.429
into the rest of the world and sort of the philosophy
00:35:26.429 --> 00:35:30.349
on what guides us and how we do things. PlatformEngineering.org,
00:35:30.349 --> 00:35:32.429
that's where you can also find me, not
00:35:32.429 --> 00:35:35.949
just in the community abroad, but inside of
00:35:35.949 --> 00:35:37.610
the Platform Engineering University. You can
00:35:37.610 --> 00:35:40.150
see me in a lot of the courses. I do teach quite
00:35:40.150 --> 00:35:41.449
a few of them. So if you want to get certified,
00:35:41.710 --> 00:35:44.349
you get to see my face a whole bunch and hear
00:35:44.349 --> 00:35:47.050
my voice. And then also Platform Engineering
00:35:47.050 --> 00:35:50.409
Consulting. So pec.platformengineering.org.
00:35:50.849 --> 00:35:53.489
And that is how you can get what we like to jokingly
00:35:53.489 --> 00:35:56.550
call Mallory as a service. Awesome. What all
00:35:56.550 --> 00:35:58.929
do you provide in that consulting? I'm just curious.
00:35:59.409 --> 00:36:03.929
Yeah. It started jokingly getting called a platform
00:36:03.929 --> 00:36:05.869
therapist because I spent most of my time feeling
00:36:05.869 --> 00:36:07.690
like I was giving people therapy or group therapy.
00:36:08.769 --> 00:36:12.329
But it's not really a joke anymore. A lot of
00:36:12.329 --> 00:36:14.630
what we're doing there is really helping teams
00:36:14.630 --> 00:36:16.610
be successful with the frameworks that we've
00:36:16.610 --> 00:36:18.849
developed for platform engineering. You know,
00:36:18.869 --> 00:36:21.250
I will go train teams on the best practices.
00:36:21.550 --> 00:36:23.769
And it's not, by the way, like, here's the tools
00:36:23.769 --> 00:36:25.510
you're going to use here. And here is how you're
00:36:25.510 --> 00:36:28.170
going to write a Pulumi. And here's how you're
00:36:28.170 --> 00:36:29.510
going to work with Kubernetes. It's not that
00:36:29.510 --> 00:36:33.210
at all. The cultural, the human side, but it's
00:36:33.210 --> 00:36:36.949
also this business case, metrics evaluation,
00:36:37.849 --> 00:36:42.210
roadmap development, building golden paths. It's
00:36:42.210 --> 00:36:43.849
all these pieces. There's this agentic stuff
00:36:43.849 --> 00:36:47.670
now as well. So I'm just a coach. There is the
00:36:47.670 --> 00:36:49.989
element of therapy. We do joke about that quite
00:36:49.989 --> 00:36:52.829
a bit because, like I said earlier, it's so much
00:36:52.829 --> 00:36:54.570
of me asking people, but have you asked why?
00:36:56.190 --> 00:36:58.530
And yeah, building their standard of practice
00:36:58.530 --> 00:37:00.829
for platform engineering, really. That's awesome.
00:37:01.309 --> 00:37:03.210
Thank you, Mallory, so much for coming on. Really,
00:37:03.269 --> 00:37:05.130
really appreciate your time. No problem, Brian.
00:37:05.230 --> 00:37:07.130
Thanks so much. It's been great. And yeah, I
00:37:07.130 --> 00:37:08.949
look forward to chatting with you again. That
00:37:08.949 --> 00:37:12.690
was my conversation with Mallory from PlatformEngineering.org.
00:37:12.690 --> 00:37:16.650
The biggest takeaway for me is that context
00:37:16.650 --> 00:37:21.550
alone isn't enough. An agents.md file is like
00:37:21.550 --> 00:37:23.809
dropping someone in the middle of New York City.
00:37:24.250 --> 00:37:27.949
The agent also needs to know what to do, how
00:37:27.949 --> 00:37:31.260
to do it, how it'll know it's done and what
00:37:31.260 --> 00:37:35.019
it's allowed to touch. I also liked how she breaks
00:37:35.019 --> 00:37:39.219
down paths. Probabilistic paths are agent-driven
00:37:39.219 --> 00:37:43.380
like a read-only PR review. Deterministic paths
00:37:43.380 --> 00:37:48.280
are your pipelines like a CI build. Hybrid paths
00:37:48.280 --> 00:37:52.420
pair an agent with a gate. The agent opens the
00:37:52.420 --> 00:37:57.239
PR, CI checks it, the agent retries, and a human
00:37:57.239 --> 00:38:01.039
still makes the call. And cap those retries.
00:38:01.360 --> 00:38:05.179
Three to five is her magic number. The part that
00:38:05.179 --> 00:38:08.199
stuck with me most is that platform engineering
00:38:08.199 --> 00:38:12.880
is really about trust. She's told teams to pump
00:38:12.880 --> 00:38:16.340
the brakes on agents because they can't get human
00:38:16.340 --> 00:38:19.400
governance right yet. Start with the identity
00:38:19.400 --> 00:38:24.239
and RBAC you already have and give agents narrow
00:38:24.239 --> 00:38:28.719
specialties. So where do you start? Ask why.
00:38:29.320 --> 00:38:34.039
Then try a read-only PR review agent, move to
00:38:34.039 --> 00:38:38.239
that hybrid path, and write down the tribal knowledge
00:38:38.239 --> 00:38:42.519
your org never documented. Your agents need it,
00:38:42.599 --> 00:38:46.880
and so do your humans. You can find Mallory on
00:38:46.880 --> 00:38:50.880
LinkedIn, plus PlatformEngineering.org, through
00:38:50.880 --> 00:38:53.940
the links in the show notes. Follow or subscribe
00:38:53.940 --> 00:38:58.070
to Ship It Weekly wherever you listen, find previous
00:38:58.070 --> 00:39:02.110
episodes at shipitweekly.fm. Thanks for listening,
00:39:02.190 --> 00:39:04.329
and I'll see you Friday for the news.