WEBVTT
00:00:00.000 --> 00:00:04.000
This week, containerd disclosed a stack of CRI
00:00:04.000 --> 00:00:06.820
plugin vulnerabilities in the runtime layer,
00:00:07.080 --> 00:00:10.800
a huge number of Kubernetes nodes trust to start
00:00:10.800 --> 00:00:14.759
your containers. Datadog ran a PostgreSQL
00:00:14.759 --> 00:00:17.660
gameday and learned their database could fail over
00:00:17.660 --> 00:00:21.300
just fine. It just couldn't do it safely. AWS
00:00:21.300 --> 00:00:25.039
DevOps Agent and Datadog's MCP Server are both
00:00:25.039 --> 00:00:28.660
now generally available. And the new AWS integration
00:00:28.660 --> 00:00:33.179
means AI incident response just graduated from
00:00:33.179 --> 00:00:37.500
demo to on-call rotation. And EKS will now route
00:00:37.500 --> 00:00:40.060
your Kubernetes control plane's outbound traffic
00:00:40.060 --> 00:00:44.020
through your own VPC, which is great, right up
00:00:44.020 --> 00:00:47.439
until a stale route table quietly kills your
00:00:47.439 --> 00:00:50.280
admission webhooks. Put those together and the
00:00:50.280 --> 00:00:53.259
shape of the episode is pretty clear. The control
00:00:53.259 --> 00:00:56.859
plane keeps getting wider. Runtimes. Databases.
00:00:57.079 --> 00:01:00.899
Incident agents. API-server egress. credentials,
00:01:01.039 --> 00:01:04.219
even the cloud console. One by one, they are
00:01:04.219 --> 00:01:07.140
all sliding into your production blast radius.
00:01:07.439 --> 00:01:10.159
And here's the part that matters. Your users
00:01:10.159 --> 00:01:13.700
don't care which control plane failed. They just
00:01:13.700 --> 00:01:16.799
feel the wait. I'm Brian Teller from Teller's
00:01:16.799 --> 00:01:36.859
Tech, and this is Ship It Weekly. Welcome back
00:01:36.859 --> 00:01:39.959
to Ship It Weekly, the show about the DevOps,
00:01:40.280 --> 00:01:45.000
SRE, cloud, platform, and security stories that
00:01:45.000 --> 00:01:47.980
actually matter when you are the person who has
00:01:47.980 --> 00:01:51.219
to keep the thing running at 3 a.m. If you are
00:01:51.219 --> 00:01:54.540
new here, follow or subscribe wherever you are
00:01:54.540 --> 00:01:57.519
watching or listening. And if you want the weekly
00:01:57.519 --> 00:02:01.219
story list and source links, check out OnCallBrief.com
00:02:01.219 --> 00:02:05.069
For past episodes, full show notes, and
00:02:05.069 --> 00:02:08.069
more from the show, head over to ShipItWeekly.fm
00:02:08.069 --> 00:02:12.469
We open with the containerd CRI plugin vulnerabilities,
00:02:13.009 --> 00:02:16.650
because your node runtime is the trust boundary
00:02:16.650 --> 00:02:20.530
underneath the trust boundary. Then, Datadog's
00:02:20.530 --> 00:02:24.430
PostgreSQL HA gameday, where the scary discovery
00:02:24.430 --> 00:02:28.530
wasn't that failover was hard, it was that failover
00:02:28.530 --> 00:02:32.800
was unsafe. After that, AWS DevOps Agent and
00:02:32.800 --> 00:02:36.639
Datadog MCP Server going GA. And what it means
00:02:36.639 --> 00:02:40.080
when an AI agent gets a seat near your control
00:02:40.080 --> 00:02:43.680
plane. Then, EKS customer-routed control-plane
00:02:43.680 --> 00:02:47.599
egress. Because your API server is now part of
00:02:47.599 --> 00:02:50.180
your network perimeter, whether you plan for
00:02:50.180 --> 00:02:53.680
it or not. In the lightning round, GitHub Credential
00:02:53.680 --> 00:02:57.840
Revocation. AWS Console Private Access. Vercel
00:02:57.840 --> 00:03:01.780
Connect, and S3 annotations. And we close with
00:03:01.780 --> 00:03:05.620
Marc Brooker on waiting, on why your customers
00:03:05.620 --> 00:03:08.860
live in the tail of your latency distribution,
00:03:09.240 --> 00:03:12.360
even when your dashboards swear everything's
00:03:12.360 --> 00:03:20.039
fine. Let's get into it. First up, containerd
00:03:20.039 --> 00:03:23.780
has a batch of CRI plugin vulnerabilities. And
00:03:23.780 --> 00:03:26.949
if you run Kubernetes, this one's yours. AWS
00:03:26.949 --> 00:03:30.030
published a security bulletin spanning
00:03:30.030 --> 00:03:35.030
containerd branches 1.7 through 2.3. And the list is
00:03:35.030 --> 00:03:38.490
not a fun read. Image cache poisoning through
00:03:38.490 --> 00:03:41.770
checkpoint image references. Host command execution.
00:03:42.490 --> 00:03:45.949
through unsanitized image labels, CDI annotation
00:03:45.949 --> 00:03:49.710
handling that can inject devices and host mounts,
00:03:50.030 --> 00:03:53.050
host file reads through symlinked container
00:03:53.050 --> 00:03:57.150
log paths during checkpoint restore, and a denial
00:03:57.150 --> 00:04:01.110
of service from crafted images that exhaust memory.
00:04:01.370 --> 00:04:05.319
So not exactly a relaxing Patch Tuesday. Here's
00:04:05.319 --> 00:04:08.840
why it matters. containerd sits underneath an
00:04:08.840 --> 00:04:12.099
enormous number of clusters, and we spend almost
00:04:12.099 --> 00:04:15.219
all of our security attention on the layers above
00:04:15.219 --> 00:04:18.800
it. Pod specs, admission control, image scanning,
00:04:19.040 --> 00:04:23.220
RBAC, network policy, runtime classes, all the
00:04:23.220 --> 00:04:26.180
familiar Kubernetes machinery. But eventually,
00:04:26.519 --> 00:04:29.680
something has to actually pull the image, unpack
00:04:29.680 --> 00:04:34.100
it, restore it, wire up devices. handle the logs,
00:04:34.180 --> 00:04:37.699
and start the container. That layer is a trust
00:04:37.699 --> 00:04:41.040
boundary too. And in some ways, it's the more
00:04:41.040 --> 00:04:44.019
dangerous one. Because by the time a workload
00:04:44.019 --> 00:04:47.399
reaches the runtime, the rest of the system has
00:04:47.399 --> 00:04:50.800
already decided this thing is allowed to exist.
00:04:51.160 --> 00:04:54.240
That's why the boring fields turn out to matter.
00:04:54.459 --> 00:04:57.939
Labels, annotations, checkpoint and restore paths,
00:04:58.319 --> 00:05:02.459
CDI, log paths, every field. that feels like
00:05:02.459 --> 00:05:05.819
plumbing can become an input to privileged behavior
00:05:05.819 --> 00:05:09.920
on the node. A malicious image isn't just application
00:05:09.920 --> 00:05:14.079
code. It's metadata, build time weirdness, and
00:05:14.079 --> 00:05:17.240
a set of assumptions the runtime makes about
00:05:17.240 --> 00:05:20.800
what it can trust. The takeaway is direct. Patch
00:05:20.800 --> 00:05:23.980
containerd. Check your managed node groups,
00:05:24.240 --> 00:05:28.519
your self-managed nodes, your AMIs, your Bottlerocket
00:05:28.519 --> 00:05:31.899
versions, your distro packages, anything
00:05:31.899 --> 00:05:35.319
that controls the runtime. If you lean on checkpoint
00:05:35.319 --> 00:05:39.899
restore, CDI devices, or GPU workloads, look
00:05:39.899 --> 00:05:43.079
harder. And if you don't use any of that, don't
00:05:43.079 --> 00:05:46.699
relax. At least one of these issues doesn't need
00:05:46.699 --> 00:05:49.620
checkpoint and restore turned on at all. Your
00:05:49.620 --> 00:05:53.379
node runtime is the trust boundary under the
00:05:53.379 --> 00:05:56.139
trust boundary. Stop treating it like invisible
00:05:56.139 --> 00:06:03.699
plumbing. Second story. Datadog published a genuinely
00:06:03.699 --> 00:06:07.160
good engineering write-up on running high availability
00:06:07.160 --> 00:06:11.500
PostgreSQL on Kubernetes. And it's one of those
00:06:11.500 --> 00:06:14.839
pieces that sounds boring until the real problem
00:06:14.839 --> 00:06:17.980
comes into focus. The problem wasn't that the
00:06:17.980 --> 00:06:20.779
database couldn't fail over. It was that it couldn't
00:06:20.779 --> 00:06:24.189
fail over safely. During a gameday, Datadog
00:06:24.189 --> 00:06:27.269
simulated a zonal failure. That added network
00:06:27.269 --> 00:06:30.930
latency, replication lag grew, and when the cluster
00:06:30.930 --> 00:06:34.209
needed a new primary, Patroni couldn't safely
00:06:34.209 --> 00:06:37.410
promote a standby without risking data loss.
00:06:37.730 --> 00:06:40.930
So the system got stuck in the worst possible
00:06:40.930 --> 00:06:44.889
spot. The old primary was unhealthy. The standbys
00:06:44.889 --> 00:06:47.550
weren't safe to promote, and the only correct
00:06:47.550 --> 00:06:50.199
move was to wait. That's the kind of failure
00:06:50.199 --> 00:06:53.680
mode that ages every SRE in the room about three
00:06:53.680 --> 00:06:56.680
years. Because on paper, you have everything.
00:06:56.939 --> 00:07:00.519
Multiple nodes, standbys, Kubernetes, automation,
00:07:01.040 --> 00:07:04.079
failover machinery. And then the actual failure
00:07:04.079 --> 00:07:07.699
arrives and the system says, yes, but not safely.
00:07:07.920 --> 00:07:10.839
Which, honestly, is the right answer. Promoting
00:07:10.839 --> 00:07:14.120
a stale standby might hand you a writable primary
00:07:14.120 --> 00:07:17.569
faster. But if it costs you data loss, split
00:07:17.569 --> 00:07:21.110
brain, or a broken consistency guarantee, you
00:07:21.110 --> 00:07:23.829
haven't fixed the outage. You've traded it for
00:07:23.829 --> 00:07:26.470
a corruption event. That's not an improvement.
00:07:26.769 --> 00:07:29.910
It's just a different postmortem. The real lesson
00:07:29.910 --> 00:07:33.009
is that HA isn't only about whether the service
00:07:33.009 --> 00:07:35.829
comes back. It's about whether the recovery path
00:07:35.829 --> 00:07:39.329
itself is safe. Can you fail over without losing
00:07:39.329 --> 00:07:42.509
writes? Can you prove which standby is safe to
00:07:42.509 --> 00:07:45.060
promote? Can your automation tell the difference
00:07:45.060 --> 00:07:47.959
between available and correct? And does your
00:07:47.959 --> 00:07:51.139
whole team agree on which one it should prefer
00:07:51.139 --> 00:07:54.899
before the incident call is on fire? Datadog's
00:07:54.899 --> 00:07:57.360
answer was to move toward synchronous replication
00:07:57.360 --> 00:08:01.300
and stronger Patroni guardrails. So a promoted
00:08:01.300 --> 00:08:05.019
standby is guaranteed to have the writes it needs.
00:08:05.300 --> 00:08:07.899
And that's the part that's worth copying. They
00:08:07.899 --> 00:08:11.040
didn't just ask how to recover faster. They asked
00:08:11.040 --> 00:08:14.480
how to recover safely. So test your database
00:08:14.480 --> 00:08:18.079
HA against real constraints, not the easy ones.
00:08:18.360 --> 00:08:22.019
Ask what happens under replication lag. Ask what
00:08:22.019 --> 00:08:25.259
happens during a zone failure. Ask what happens
00:08:25.259 --> 00:08:28.579
when the network is slow instead of cleanly dead.
00:08:28.860 --> 00:08:32.039
Ask what happens when every standby is behind.
00:08:32.399 --> 00:08:35.419
And ask whether your automation prefers safety
00:08:35.419 --> 00:08:38.600
or availability. And whether everyone actually
00:08:38.600 --> 00:08:41.779
agrees with that choice. Because failover is
00:08:41.779 --> 00:08:44.860
useless, if the only safe option is waiting.
00:08:45.039 --> 00:08:52.919
But unsafe failover can be a lot worse. Third
00:08:52.919 --> 00:08:56.679
story, AWS DevOps Agent is now generally available
00:08:56.679 --> 00:09:00.820
and Datadog's MCP Server is GA as a standard
00:09:00.820 --> 00:09:04.379
way for AI agents to reach Datadog monitoring
00:09:04.379 --> 00:09:07.379
data. This is one of those announcements. where
00:09:07.379 --> 00:09:10.019
the slide says autonomous incident resolution
00:09:10.019 --> 00:09:13.919
and the operator says, cool, but what exactly
00:09:13.919 --> 00:09:17.720
is it allowed to touch? The idea is solid. AWS
00:09:17.720 --> 00:09:21.820
DevOps Agent can work through Datadog MCP Server
00:09:21.820 --> 00:09:26.179
to investigate an incident across logs, metrics,
00:09:26.419 --> 00:09:30.340
traces, deployment events, and AWS infrastructure
00:09:30.340 --> 00:09:33.980
context. Instead of one engineer bouncing between
00:09:33.980 --> 00:09:38.679
CloudWatch, Datadog, deploy history, traces, dashboards,
00:09:38.679 --> 00:09:42.700
and Slack, the agent correlates the signals and
00:09:42.700 --> 00:09:45.779
helps push the incident forward and nobody wants
00:09:45.779 --> 00:09:48.620
to spend the first 30 minutes of an outage doing
00:09:48.620 --> 00:09:51.879
browser-tab archaeology if an agent can gather
00:09:51.879 --> 00:09:55.460
context, summarize what changed, flag a suspicious
00:09:55.460 --> 00:09:59.259
deploy and propose likely causes that's real
00:09:59.259 --> 00:10:02.820
time saved but this is also the moment AI incident
00:10:02.820 --> 00:10:06.549
response stops being a chatbot and becomes a
00:10:06.549 --> 00:10:10.289
production workflow. It's an agent reading operational
00:10:10.289 --> 00:10:13.549
telemetry, interpreting signals, recommending
00:10:13.549 --> 00:10:17.909
fixes, and potentially wired into Slack, PagerDuty,
00:10:18.049 --> 00:10:21.669
ServiceNow, your code, your deploys, and your
00:10:21.669 --> 00:10:24.549
runbooks. That puts it right next to the control
00:10:24.549 --> 00:10:27.870
plane. And once something sits next to the control
00:10:27.870 --> 00:10:31.029
plane, the question stops being, is it smart?
00:10:31.210 --> 00:10:34.840
And becomes, what authority does it have? Can
00:10:34.840 --> 00:10:38.240
it only read? Can it write? Can it open tickets?
00:10:38.539 --> 00:10:42.200
Trigger automation? Roll back a deploy? Restart
00:10:42.200 --> 00:10:46.679
a service? Change config? Page a human at 4 a.m.?
00:10:46.940 --> 00:10:51.120
Can it make things worse quickly and very confidently?
00:10:51.460 --> 00:10:54.919
That last one is the whole game. Incident response
00:10:54.919 --> 00:10:58.840
isn't about speed. It's about safe speed. So
00:10:58.840 --> 00:11:02.659
treat AI incident tooling like any other production
00:11:02.659 --> 00:11:05.759
automation. Give it the least privilege that
00:11:05.759 --> 00:11:09.059
still leaves it useful. Log what it sees and
00:11:09.059 --> 00:11:11.860
what it does. Make the human approval boundary
00:11:11.860 --> 00:11:15.879
impossible to miss. And draw a hard line between
00:11:15.879 --> 00:11:19.299
what it can recommend and what it can execute.
00:11:19.600 --> 00:11:23.279
Have rollback rules. Know what happens when it's
00:11:23.279 --> 00:11:26.419
wrong. And don't grade it only on time to answer.
00:11:26.580 --> 00:11:29.480
Grade it on whether the answer was safe, auditable,
00:11:29.620 --> 00:11:33.100
and actually useful under pressure. AI incident
00:11:33.100 --> 00:11:36.679
response is moving from demo to production. That's
00:11:36.679 --> 00:11:44.080
exciting. Production just needs guardrails. Fourth
00:11:44.080 --> 00:11:48.120
story. Amazon EKS now supports customer-routed
00:11:48.120 --> 00:11:51.940
control-plane egress. That's a very AWS phrase.
00:11:52.240 --> 00:11:55.000
So here's the human version. The Kubernetes API
00:11:55.000 --> 00:11:58.940
server sometimes needs to call outward to admission
00:11:58.940 --> 00:12:02.940
webhooks, OIDC providers, aggregated API servers,
00:12:03.159 --> 00:12:05.940
other endpoints that you control. Historically,
00:12:06.019 --> 00:12:09.639
that outbound traffic took AWS managed egress
00:12:09.639 --> 00:12:12.679
paths. Now you can route it through your own
00:12:12.679 --> 00:12:16.299
VPC, which hands platform teams control over
00:12:16.299 --> 00:12:19.700
routing, inspection, firewalls, NAT, private
00:12:19.700 --> 00:12:22.419
connectivity, and compliance boundaries. For
00:12:22.419 --> 00:12:25.519
regulated environments, that's a real win. It
00:12:25.519 --> 00:12:28.320
also makes the control plane feel a lot more
00:12:28.320 --> 00:12:30.899
like part of your network. which of course it
00:12:30.899 --> 00:12:34.000
always was. The difference is that now you own
00:12:34.000 --> 00:12:37.559
the outbound path and AWS is blunt about what
00:12:37.559 --> 00:12:40.379
that ownership means. In customer routed mode,
00:12:40.639 --> 00:12:43.639
you are responsible for making sure the control
00:12:43.639 --> 00:12:46.700
plane can reach the endpoints it needs. Wrong
00:12:46.700 --> 00:12:50.100
route table, too-tight security group, a NACL
00:12:50.100 --> 00:12:52.940
that blocks the wrong thing, a broken firewall
00:12:52.940 --> 00:12:55.980
hop, and control plane operations start failing.
00:12:56.120 --> 00:13:00.139
That includes admission webhook calls, and OIDC
00:13:00.139 --> 00:13:03.700
authentication. So yes, great feature. But it
00:13:03.700 --> 00:13:06.580
isn't a checkbox. It's a failure mode change.
00:13:06.919 --> 00:13:10.480
If your API server can't reach an admission webhook,
00:13:10.539 --> 00:13:14.460
do pod creates fail? Do deploys hang? Does authentication
00:13:14.460 --> 00:13:18.039
break? Does your incident response now depend
00:13:18.039 --> 00:13:21.779
on a firewall path some other team owns? And
00:13:21.779 --> 00:13:25.019
do you have a metric? a test, and a name on the
00:13:25.019 --> 00:13:28.120
pager for when it breaks? This is a feature you
00:13:28.120 --> 00:13:31.100
bring to a design review. Not because it's risky,
00:13:31.259 --> 00:13:34.580
but because it's powerful. Map the traffic. Map
00:13:34.580 --> 00:13:38.440
the dependencies. Test the webhooks. Test OIDC.
00:13:38.580 --> 00:13:41.679
Test the failure modes. Make the routing visible.
00:13:41.899 --> 00:13:44.679
And write the runbook before the control plane
00:13:44.679 --> 00:13:47.820
starts failing in creative ways. The Kubernetes
00:13:47.820 --> 00:13:50.659
control plane is becoming part of your network
00:13:50.659 --> 00:14:00.740
perimeter. Treat it like one. Quick lightning
00:14:00.740 --> 00:14:03.980
round. First, GitHub added self-service credential
00:14:03.980 --> 00:14:07.059
revocation for incident response. Enterprise
00:14:07.059 --> 00:14:11.159
owners now get a break-glass capability to revoke
00:14:11.159 --> 00:14:14.179
a compromised user's credentials in one move.
00:14:14.399 --> 00:14:17.200
This matters because credential cleanup should
00:14:17.200 --> 00:14:20.200
never be a scavenger hunt. You do not want to
00:14:20.200 --> 00:14:22.639
be hand-hunting through SSO authorizations,
00:14:22.899 --> 00:14:27.149
personal access tokens, SSH keys, and OAuth grants
00:14:27.149 --> 00:14:30.490
while everyone argues in Slack. Revocation is
00:14:30.490 --> 00:14:33.169
incident response infrastructure. Know who can
00:14:33.169 --> 00:14:35.889
trigger it, know what it kills, know what it
00:14:35.889 --> 00:14:39.169
logs, and put it in the compromised-account runbook.
00:14:39.370 --> 00:14:42.429
Second, AWS Management Console private access
00:14:42.429 --> 00:14:45.529
now works without internet connectivity. Console
00:14:45.529 --> 00:14:48.490
traffic for supported services can flow over
00:14:48.490 --> 00:14:51.549
VPC endpoints instead of the public internet.
00:14:51.899 --> 00:14:54.460
It's a strong story for regulated environments.
00:14:54.899 --> 00:14:57.399
Even the console is getting pulled behind private
00:14:57.399 --> 00:15:00.259
network boundaries. The lesson? Console access
00:15:00.259 --> 00:15:03.179
is part of your control plane too. And private
00:15:03.179 --> 00:15:06.779
link, endpoint policies, and known-account restrictions
00:15:06.779 --> 00:15:10.360
are becoming cloud operations, not just app networking.
00:15:10.879 --> 00:15:13.960
Third, Vercel shipped Vercel Connect. And the
00:15:13.960 --> 00:15:17.179
idea worth catching is runtime credential exchange.
00:15:17.659 --> 00:15:20.419
Instead of stashing a long-lived provider token
00:15:20.419 --> 00:15:23.799
for an agent, The app proves its identity and
00:15:23.799 --> 00:15:26.500
gets a short-lived task-scoped credential.
00:15:26.799 --> 00:15:28.980
That's the pattern that we've been tracking for
00:15:28.980 --> 00:15:31.720
weeks. Agent credentials moving from store this
00:15:31.720 --> 00:15:35.679
token forever to prove who you are and get scoped
00:15:35.679 --> 00:15:38.600
access when you need it. Short-lived credentials
00:15:38.600 --> 00:15:41.720
don't solve every agent security problem, but
00:15:41.720 --> 00:15:44.360
they beat long-lived secrets sitting around
00:15:44.360 --> 00:15:47.879
waiting to become next quarter's incident. Fourth,
00:15:48.039 --> 00:15:52.009
Amazon S3 annotations are here. mutable, queryable
00:15:52.009 --> 00:15:56.149
context attached directly to S3 objects. Sounds
00:15:56.149 --> 00:15:59.909
dull, but object metadata has driven a lot of
00:15:59.909 --> 00:16:02.850
awkward platform design over the years. Side
00:16:02.850 --> 00:16:06.470
tables, DynamoDB metadata stores, Lambda sync
00:16:06.470 --> 00:16:10.490
jobs, custom catalogs, and constant drift between
00:16:10.490 --> 00:16:14.029
the object and whatever's describing it. If annotations
00:16:14.029 --> 00:16:16.720
shrink that glue layer, That's worth watching.
00:16:16.820 --> 00:16:20.259
Object metadata is quietly becoming a first-class
00:16:20.259 --> 00:16:24.000
platform layer, especially for data, AI, search,
00:16:24.200 --> 00:16:27.799
and agent workflows that need to know what an
00:16:27.799 --> 00:16:38.779
object is, not just where it lives. The human
00:16:38.779 --> 00:16:42.340
closer this week comes from a Marc Brooker post
00:16:42.340 --> 00:16:47.370
about waiting, latency, MTTR, and why averages
00:16:47.370 --> 00:16:50.990
can lie. The point is that your users don't experience
00:16:50.990 --> 00:16:54.070
your averages the way your dashboards report
00:16:54.070 --> 00:16:57.570
them. You measure mean latency, mean time to
00:16:57.570 --> 00:17:00.529
recovery, average outage duration. But people
00:17:00.529 --> 00:17:03.590
are far more likely to land in the long waits
00:17:03.590 --> 00:17:07.289
simply because long waits take up more of the
00:17:07.289 --> 00:17:10.970
time. That's the inspection paradox. A 10-minute
00:17:10.970 --> 00:17:15.019
outage catches a few users. A 10-hour outage
00:17:15.019 --> 00:17:18.000
catches a lot of them. Your incident tracker
00:17:18.000 --> 00:17:22.000
counts both as one outage. Your dashboard says
00:17:22.000 --> 00:17:26.279
MTTR looks fine. Your users say they spent all
00:17:26.279 --> 00:17:29.140
morning waiting. Both are true. And that's the
00:17:29.140 --> 00:17:31.460
whole episode, really. When the system breaks,
00:17:31.740 --> 00:17:34.460
nobody experiences your architecture diagram.
00:17:34.759 --> 00:17:37.859
They experience waiting. Waiting for a request.
00:17:38.160 --> 00:17:40.890
Waiting for recovery. Waiting for a credential
00:17:40.890 --> 00:17:44.150
to get revoked. Waiting for a deploy to stop
00:17:44.150 --> 00:17:46.950
failing. Waiting for the control plane to come
00:17:46.950 --> 00:17:50.190
back. Waiting for someone to find the right context.
00:17:50.549 --> 00:17:53.930
So here's the takeaway. Don't only measure the
00:17:53.930 --> 00:17:56.990
system from the server side. Measure it from
00:17:56.990 --> 00:18:00.650
the waiting side. Because your users don't live
00:18:00.650 --> 00:18:04.309
in your average. They live in the tail. And the
00:18:04.309 --> 00:18:07.789
tail is usually where the real reliability story
00:18:07.789 --> 00:18:10.799
is hiding. That's it for this week of Ship It
00:18:10.799 --> 00:18:13.519
Weekly. We covered containerd runtime risk,
00:18:13.799 --> 00:18:17.000
Postgres failover safety, AI incident response,
00:18:17.460 --> 00:18:20.839
EKS control-plane egress, and why your users
00:18:20.839 --> 00:18:23.819
feel the wait more than your dashboards show.
00:18:24.039 --> 00:18:27.019
If this episode was useful, follow or subscribe
00:18:27.019 --> 00:18:29.960
wherever you are watching or listening. If you're
00:18:29.960 --> 00:18:32.940
on YouTube, hit subscribe. If you're in a podcast
00:18:32.940 --> 00:18:36.250
app, follow the show there. And if you know someone
00:18:36.250 --> 00:18:39.109
wrestling with Kubernetes runtime security, database
00:18:39.109 --> 00:18:42.750
failover, AI incident response, or platform control
00:18:42.750 --> 00:18:45.930
planes, send them this one. It genuinely helps
00:18:45.930 --> 00:18:48.549
the show grow, and it helps me keep making this
00:18:48.549 --> 00:18:51.549
for people who actually live with these systems.
00:18:51.849 --> 00:18:54.430
You can find the weekly brief at OnCallBrief.com
00:18:54.430 --> 00:18:57.549
and the full show notes, links, and past
00:18:57.549 --> 00:19:01.230
episodes at ShipItWeekly.fm. I'm Brian Teller
00:19:01.230 --> 00:19:03.670
from Teller's Tech. Thanks for listening. And
00:19:03.670 --> 00:19:06.140
remember, your dashboards measure the average.
00:19:06.339 --> 00:19:08.220
Your users feel the wait.
00:00:00.000 --> 00:00:04.000
This week, containerd disclosed a stack of CRI
00:00:04.000 --> 00:00:06.820
plugin vulnerabilities in the runtime layer,
00:00:07.080 --> 00:00:10.800
a huge number of Kubernetes nodes trust to start
00:00:10.800 --> 00:00:14.759
your containers. Datadog ran a PostgreSQL
00:00:14.759 --> 00:00:17.660
gameday and learned their database could fail over
00:00:17.660 --> 00:00:21.300
just fine. It just couldn't do it safely. AWS
00:00:21.300 --> 00:00:25.039
DevOps Agent and Datadog's MCP Server are both
00:00:25.039 --> 00:00:28.660
now generally available. And the new AWS integration
00:00:28.660 --> 00:00:33.179
means AI incident response just graduated from
00:00:33.179 --> 00:00:37.500
demo to on-call rotation. And EKS will now route
00:00:37.500 --> 00:00:40.060
your Kubernetes control plane's outbound traffic
00:00:40.060 --> 00:00:44.020
through your own VPC, which is great, right up
00:00:44.020 --> 00:00:47.439
until a stale route table quietly kills your
00:00:47.439 --> 00:00:50.280
admission webhooks. Put those together and the
00:00:50.280 --> 00:00:53.259
shape of the episode is pretty clear. The control
00:00:53.259 --> 00:00:56.859
plane keeps getting wider. Runtimes. Databases.
00:00:57.079 --> 00:01:00.899
Incident agents. API-server egress. credentials,
00:01:01.039 --> 00:01:04.219
even the cloud console. One by one, they are
00:01:04.219 --> 00:01:07.140
all sliding into your production blast radius.
00:01:07.439 --> 00:01:10.159
And here's the part that matters. Your users
00:01:10.159 --> 00:01:13.700
don't care which control plane failed. They just
00:01:13.700 --> 00:01:16.799
feel the wait. I'm Brian Teller from Teller's
00:01:16.799 --> 00:01:36.859
Tech, and this is Ship It Weekly. Welcome back
00:01:36.859 --> 00:01:39.959
to Ship It Weekly, the show about the DevOps,
00:01:40.280 --> 00:01:45.000
SRE, cloud, platform, and security stories that
00:01:45.000 --> 00:01:47.980
actually matter when you are the person who has
00:01:47.980 --> 00:01:51.219
to keep the thing running at 3 a.m. If you are
00:01:51.219 --> 00:01:54.540
new here, follow or subscribe wherever you are
00:01:54.540 --> 00:01:57.519
watching or listening. And if you want the weekly
00:01:57.519 --> 00:02:01.219
story list and source links, check out OnCallBrief.com
00:02:01.219 --> 00:02:05.069
For past episodes, full show notes, and
00:02:05.069 --> 00:02:08.069
more from the show, head over to ShipItWeekly.fm
00:02:08.069 --> 00:02:12.469
We open with the containerd CRI plugin vulnerabilities,
00:02:13.009 --> 00:02:16.650
because your node runtime is the trust boundary
00:02:16.650 --> 00:02:20.530
underneath the trust boundary. Then, Datadog's
00:02:20.530 --> 00:02:24.430
PostgreSQL HA gameday, where the scary discovery
00:02:24.430 --> 00:02:28.530
wasn't that failover was hard, it was that failover
00:02:28.530 --> 00:02:32.800
was unsafe. After that, AWS DevOps Agent and
00:02:32.800 --> 00:02:36.639
Datadog MCP Server going GA. And what it means
00:02:36.639 --> 00:02:40.080
when an AI agent gets a seat near your control
00:02:40.080 --> 00:02:43.680
plane. Then, EKS customer-routed control-plane
00:02:43.680 --> 00:02:47.599
egress. Because your API server is now part of
00:02:47.599 --> 00:02:50.180
your network perimeter, whether you plan for
00:02:50.180 --> 00:02:53.680
it or not. In the lightning round, GitHub Credential
00:02:53.680 --> 00:02:57.840
Revocation. AWS Console Private Access. Vercel
00:02:57.840 --> 00:03:01.780
Connect, and S3 annotations. And we close with
00:03:01.780 --> 00:03:05.620
Marc Brooker on waiting, on why your customers
00:03:05.620 --> 00:03:08.860
live in the tail of your latency distribution,
00:03:09.240 --> 00:03:12.360
even when your dashboards swear everything's
00:03:12.360 --> 00:03:20.039
fine. Let's get into it. First up, containerd
00:03:20.039 --> 00:03:23.780
has a batch of CRI plugin vulnerabilities. And
00:03:23.780 --> 00:03:26.949
if you run Kubernetes, this one's yours. AWS
00:03:26.949 --> 00:03:30.030
published a security bulletin spanning
00:03:30.030 --> 00:03:35.030
containerd branches 1.7 through 2.3. And the list is
00:03:35.030 --> 00:03:38.490
not a fun read. Image cache poisoning through
00:03:38.490 --> 00:03:41.770
checkpoint image references. Host command execution.
00:03:42.490 --> 00:03:45.949
through unsanitized image labels, CDI annotation
00:03:45.949 --> 00:03:49.710
handling that can inject devices and host mounts,
00:03:50.030 --> 00:03:53.050
host file reads through symlinked container
00:03:53.050 --> 00:03:57.150
log paths during checkpoint restore, and a denial
00:03:57.150 --> 00:04:01.110
of service from crafted images that exhaust memory.
00:04:01.370 --> 00:04:05.319
So not exactly a relaxing Patch Tuesday. Here's
00:04:05.319 --> 00:04:08.840
why it matters. containerd sits underneath an
00:04:08.840 --> 00:04:12.099
enormous number of clusters, and we spend almost
00:04:12.099 --> 00:04:15.219
all of our security attention on the layers above
00:04:15.219 --> 00:04:18.800
it. Pod specs, admission control, image scanning,
00:04:19.040 --> 00:04:23.220
RBAC, network policy, runtime classes, all the
00:04:23.220 --> 00:04:26.180
familiar Kubernetes machinery. But eventually,
00:04:26.519 --> 00:04:29.680
something has to actually pull the image, unpack
00:04:29.680 --> 00:04:34.100
it, restore it, wire up devices. handle the logs,
00:04:34.180 --> 00:04:37.699
and start the container. That layer is a trust
00:04:37.699 --> 00:04:41.040
boundary too. And in some ways, it's the more
00:04:41.040 --> 00:04:44.019
dangerous one. Because by the time a workload
00:04:44.019 --> 00:04:47.399
reaches the runtime, the rest of the system has
00:04:47.399 --> 00:04:50.800
already decided this thing is allowed to exist.
00:04:51.160 --> 00:04:54.240
That's why the boring fields turn out to matter.
00:04:54.459 --> 00:04:57.939
Labels, annotations, checkpoint and restore paths,
00:04:58.319 --> 00:05:02.459
CDI, log paths, every field. that feels like
00:05:02.459 --> 00:05:05.819
plumbing can become an input to privileged behavior
00:05:05.819 --> 00:05:09.920
on the node. A malicious image isn't just application
00:05:09.920 --> 00:05:14.079
code. It's metadata, build time weirdness, and
00:05:14.079 --> 00:05:17.240
a set of assumptions the runtime makes about
00:05:17.240 --> 00:05:20.800
what it can trust. The takeaway is direct. Patch
00:05:20.800 --> 00:05:23.980
containerd. Check your managed node groups,
00:05:24.240 --> 00:05:28.519
your self-managed nodes, your AMIs, your Bottlerocket
00:05:28.519 --> 00:05:31.899
versions, your distro packages, anything
00:05:31.899 --> 00:05:35.319
that controls the runtime. If you lean on checkpoint
00:05:35.319 --> 00:05:39.899
restore, CDI devices, or GPU workloads, look
00:05:39.899 --> 00:05:43.079
harder. And if you don't use any of that, don't
00:05:43.079 --> 00:05:46.699
relax. At least one of these issues doesn't need
00:05:46.699 --> 00:05:49.620
checkpoint and restore turned on at all. Your
00:05:49.620 --> 00:05:53.379
node runtime is the trust boundary under the
00:05:53.379 --> 00:05:56.139
trust boundary. Stop treating it like invisible
00:05:56.139 --> 00:06:03.699
plumbing. Second story. Datadog published a genuinely
00:06:03.699 --> 00:06:07.160
good engineering write-up on running high availability
00:06:07.160 --> 00:06:11.500
PostgreSQL on Kubernetes. And it's one of those
00:06:11.500 --> 00:06:14.839
pieces that sounds boring until the real problem
00:06:14.839 --> 00:06:17.980
comes into focus. The problem wasn't that the
00:06:17.980 --> 00:06:20.779
database couldn't fail over. It was that it couldn't
00:06:20.779 --> 00:06:24.189
fail over safely. During a gameday, Datadog
00:06:24.189 --> 00:06:27.269
simulated a zonal failure. That added network
00:06:27.269 --> 00:06:30.930
latency, replication lag grew, and when the cluster
00:06:30.930 --> 00:06:34.209
needed a new primary, Patroni couldn't safely
00:06:34.209 --> 00:06:37.410
promote a standby without risking data loss.
00:06:37.730 --> 00:06:40.930
So the system got stuck in the worst possible
00:06:40.930 --> 00:06:44.889
spot. The old primary was unhealthy. The standbys
00:06:44.889 --> 00:06:47.550
weren't safe to promote, and the only correct
00:06:47.550 --> 00:06:50.199
move was to wait. That's the kind of failure
00:06:50.199 --> 00:06:53.680
mode that ages every SRE in the room about three
00:06:53.680 --> 00:06:56.680
years. Because on paper, you have everything.
00:06:56.939 --> 00:07:00.519
Multiple nodes, standbys, Kubernetes, automation,
00:07:01.040 --> 00:07:04.079
failover machinery. And then the actual failure
00:07:04.079 --> 00:07:07.699
arrives and the system says, yes, but not safely.
00:07:07.920 --> 00:07:10.839
Which, honestly, is the right answer. Promoting
00:07:10.839 --> 00:07:14.120
a stale standby might hand you a writable primary
00:07:14.120 --> 00:07:17.569
faster. But if it costs you data loss, split
00:07:17.569 --> 00:07:21.110
brain, or a broken consistency guarantee, you
00:07:21.110 --> 00:07:23.829
haven't fixed the outage. You've traded it for
00:07:23.829 --> 00:07:26.470
a corruption event. That's not an improvement.
00:07:26.769 --> 00:07:29.910
It's just a different postmortem. The real lesson
00:07:29.910 --> 00:07:33.009
is that HA isn't only about whether the service
00:07:33.009 --> 00:07:35.829
comes back. It's about whether the recovery path
00:07:35.829 --> 00:07:39.329
itself is safe. Can you fail over without losing
00:07:39.329 --> 00:07:42.509
writes? Can you prove which standby is safe to
00:07:42.509 --> 00:07:45.060
promote? Can your automation tell the difference
00:07:45.060 --> 00:07:47.959
between available and correct? And does your
00:07:47.959 --> 00:07:51.139
whole team agree on which one it should prefer
00:07:51.139 --> 00:07:54.899
before the incident call is on fire? Datadog's
00:07:54.899 --> 00:07:57.360
answer was to move toward synchronous replication
00:07:57.360 --> 00:08:01.300
and stronger Patroni guardrails. So a promoted
00:08:01.300 --> 00:08:05.019
standby is guaranteed to have the writes it needs.
00:08:05.300 --> 00:08:07.899
And that's the part that's worth copying. They
00:08:07.899 --> 00:08:11.040
didn't just ask how to recover faster. They asked
00:08:11.040 --> 00:08:14.480
how to recover safely. So test your database
00:08:14.480 --> 00:08:18.079
HA against real constraints, not the easy ones.
00:08:18.360 --> 00:08:22.019
Ask what happens under replication lag. Ask what
00:08:22.019 --> 00:08:25.259
happens during a zone failure. Ask what happens
00:08:25.259 --> 00:08:28.579
when the network is slow instead of cleanly dead.
00:08:28.860 --> 00:08:32.039
Ask what happens when every standby is behind.
00:08:32.399 --> 00:08:35.419
And ask whether your automation prefers safety
00:08:35.419 --> 00:08:38.600
or availability. And whether everyone actually
00:08:38.600 --> 00:08:41.779
agrees with that choice. Because failover is
00:08:41.779 --> 00:08:44.860
useless, if the only safe option is waiting.
00:08:45.039 --> 00:08:52.919
But unsafe failover can be a lot worse. Third
00:08:52.919 --> 00:08:56.679
story, AWS DevOps Agent is now generally available
00:08:56.679 --> 00:09:00.820
and Datadog's MCP Server is GA as a standard
00:09:00.820 --> 00:09:04.379
way for AI agents to reach Datadog monitoring
00:09:04.379 --> 00:09:07.379
data. This is one of those announcements. where
00:09:07.379 --> 00:09:10.019
the slide says autonomous incident resolution
00:09:10.019 --> 00:09:13.919
and the operator says, cool, but what exactly
00:09:13.919 --> 00:09:17.720
is it allowed to touch? The idea is solid. AWS
00:09:17.720 --> 00:09:21.820
DevOps Agent can work through Datadog MCP Server
00:09:21.820 --> 00:09:26.179
to investigate an incident across logs, metrics,
00:09:26.419 --> 00:09:30.340
traces, deployment events, and AWS infrastructure
00:09:30.340 --> 00:09:33.980
context. Instead of one engineer bouncing between
00:09:33.980 --> 00:09:38.679
CloudWatch, Datadog, deploy history, traces, dashboards,
00:09:38.679 --> 00:09:42.700
and Slack, the agent correlates the signals and
00:09:42.700 --> 00:09:45.779
helps push the incident forward and nobody wants
00:09:45.779 --> 00:09:48.620
to spend the first 30 minutes of an outage doing
00:09:48.620 --> 00:09:51.879
browser-tab archaeology if an agent can gather
00:09:51.879 --> 00:09:55.460
context, summarize what changed, flag a suspicious
00:09:55.460 --> 00:09:59.259
deploy and propose likely causes that's real
00:09:59.259 --> 00:10:02.820
time saved but this is also the moment AI incident
00:10:02.820 --> 00:10:06.549
response stops being a chatbot and becomes a
00:10:06.549 --> 00:10:10.289
production workflow. It's an agent reading operational
00:10:10.289 --> 00:10:13.549
telemetry, interpreting signals, recommending
00:10:13.549 --> 00:10:17.909
fixes, and potentially wired into Slack, PagerDuty,
00:10:18.049 --> 00:10:21.669
ServiceNow, your code, your deploys, and your
00:10:21.669 --> 00:10:24.549
runbooks. That puts it right next to the control
00:10:24.549 --> 00:10:27.870
plane. And once something sits next to the control
00:10:27.870 --> 00:10:31.029
plane, the question stops being, is it smart?
00:10:31.210 --> 00:10:34.840
And becomes, what authority does it have? Can
00:10:34.840 --> 00:10:38.240
it only read? Can it write? Can it open tickets?
00:10:38.539 --> 00:10:42.200
Trigger automation? Roll back a deploy? Restart
00:10:42.200 --> 00:10:46.679
a service? Change config? Page a human at 4 a.m.?
00:10:46.940 --> 00:10:51.120
Can it make things worse quickly and very confidently?
00:10:51.460 --> 00:10:54.919
That last one is the whole game. Incident response
00:10:54.919 --> 00:10:58.840
isn't about speed. It's about safe speed. So
00:10:58.840 --> 00:11:02.659
treat AI incident tooling like any other production
00:11:02.659 --> 00:11:05.759
automation. Give it the least privilege that
00:11:05.759 --> 00:11:09.059
still leaves it useful. Log what it sees and
00:11:09.059 --> 00:11:11.860
what it does. Make the human approval boundary
00:11:11.860 --> 00:11:15.879
impossible to miss. And draw a hard line between
00:11:15.879 --> 00:11:19.299
what it can recommend and what it can execute.
00:11:19.600 --> 00:11:23.279
Have rollback rules. Know what happens when it's
00:11:23.279 --> 00:11:26.419
wrong. And don't grade it only on time to answer.
00:11:26.580 --> 00:11:29.480
Grade it on whether the answer was safe, auditable,
00:11:29.620 --> 00:11:33.100
and actually useful under pressure. AI incident
00:11:33.100 --> 00:11:36.679
response is moving from demo to production. That's
00:11:36.679 --> 00:11:44.080
exciting. Production just needs guardrails. Fourth
00:11:44.080 --> 00:11:48.120
story. Amazon EKS now supports customer-routed
00:11:48.120 --> 00:11:51.940
control-plane egress. That's a very AWS phrase.
00:11:52.240 --> 00:11:55.000
So here's the human version. The Kubernetes API
00:11:55.000 --> 00:11:58.940
server sometimes needs to call outward to admission
00:11:58.940 --> 00:12:02.940
webhooks, OIDC providers, aggregated API servers,
00:12:03.159 --> 00:12:05.940
other endpoints that you control. Historically,
00:12:06.019 --> 00:12:09.639
that outbound traffic took AWS managed egress
00:12:09.639 --> 00:12:12.679
paths. Now you can route it through your own
00:12:12.679 --> 00:12:16.299
VPC, which hands platform teams control over
00:12:16.299 --> 00:12:19.700
routing, inspection, firewalls, NAT, private
00:12:19.700 --> 00:12:22.419
connectivity, and compliance boundaries. For
00:12:22.419 --> 00:12:25.519
regulated environments, that's a real win. It
00:12:25.519 --> 00:12:28.320
also makes the control plane feel a lot more
00:12:28.320 --> 00:12:30.899
like part of your network. which of course it
00:12:30.899 --> 00:12:34.000
always was. The difference is that now you own
00:12:34.000 --> 00:12:37.559
the outbound path and AWS is blunt about what
00:12:37.559 --> 00:12:40.379
that ownership means. In customer routed mode,
00:12:40.639 --> 00:12:43.639
you are responsible for making sure the control
00:12:43.639 --> 00:12:46.700
plane can reach the endpoints it needs. Wrong
00:12:46.700 --> 00:12:50.100
route table, too-tight security group, a NACL
00:12:50.100 --> 00:12:52.940
that blocks the wrong thing, a broken firewall
00:12:52.940 --> 00:12:55.980
hop, and control plane operations start failing.
00:12:56.120 --> 00:13:00.139
That includes admission webhook calls, and OIDC
00:13:00.139 --> 00:13:03.700
authentication. So yes, great feature. But it
00:13:03.700 --> 00:13:06.580
isn't a checkbox. It's a failure mode change.
00:13:06.919 --> 00:13:10.480
If your API server can't reach an admission webhook,
00:13:10.539 --> 00:13:14.460
do pod creates fail? Do deploys hang? Does authentication
00:13:14.460 --> 00:13:18.039
break? Does your incident response now depend
00:13:18.039 --> 00:13:21.779
on a firewall path some other team owns? And
00:13:21.779 --> 00:13:25.019
do you have a metric? a test, and a name on the
00:13:25.019 --> 00:13:28.120
pager for when it breaks? This is a feature you
00:13:28.120 --> 00:13:31.100
bring to a design review. Not because it's risky,
00:13:31.259 --> 00:13:34.580
but because it's powerful. Map the traffic. Map
00:13:34.580 --> 00:13:38.440
the dependencies. Test the webhooks. Test OIDC.
00:13:38.580 --> 00:13:41.679
Test the failure modes. Make the routing visible.
00:13:41.899 --> 00:13:44.679
And write the runbook before the control plane
00:13:44.679 --> 00:13:47.820
starts failing in creative ways. The Kubernetes
00:13:47.820 --> 00:13:50.659
control plane is becoming part of your network
00:13:50.659 --> 00:14:00.740
perimeter. Treat it like one. Quick lightning
00:14:00.740 --> 00:14:03.980
round. First, GitHub added self-service credential
00:14:03.980 --> 00:14:07.059
revocation for incident response. Enterprise
00:14:07.059 --> 00:14:11.159
owners now get a break-glass capability to revoke
00:14:11.159 --> 00:14:14.179
a compromised user's credentials in one move.
00:14:14.399 --> 00:14:17.200
This matters because credential cleanup should
00:14:17.200 --> 00:14:20.200
never be a scavenger hunt. You do not want to
00:14:20.200 --> 00:14:22.639
be hand-hunting through SSO authorizations,
00:14:22.899 --> 00:14:27.149
personal access tokens, SSH keys, and OAuth grants
00:14:27.149 --> 00:14:30.490
while everyone argues in Slack. Revocation is
00:14:30.490 --> 00:14:33.169
incident response infrastructure. Know who can
00:14:33.169 --> 00:14:35.889
trigger it, know what it kills, know what it
00:14:35.889 --> 00:14:39.169
logs, and put it in the compromised-account runbook.
00:14:39.370 --> 00:14:42.429
Second, AWS Management Console private access
00:14:42.429 --> 00:14:45.529
now works without internet connectivity. Console
00:14:45.529 --> 00:14:48.490
traffic for supported services can flow over
00:14:48.490 --> 00:14:51.549
VPC endpoints instead of the public internet.
00:14:51.899 --> 00:14:54.460
It's a strong story for regulated environments.
00:14:54.899 --> 00:14:57.399
Even the console is getting pulled behind private
00:14:57.399 --> 00:15:00.259
network boundaries. The lesson? Console access
00:15:00.259 --> 00:15:03.179
is part of your control plane too. And private
00:15:03.179 --> 00:15:06.779
link, endpoint policies, and known-account restrictions
00:15:06.779 --> 00:15:10.360
are becoming cloud operations, not just app networking.
00:15:10.879 --> 00:15:13.960
Third, Vercel shipped Vercel Connect. And the
00:15:13.960 --> 00:15:17.179
idea worth catching is runtime credential exchange.
00:15:17.659 --> 00:15:20.419
Instead of stashing a long-lived provider token
00:15:20.419 --> 00:15:23.799
for an agent, The app proves its identity and
00:15:23.799 --> 00:15:26.500
gets a short-lived task-scoped credential.
00:15:26.799 --> 00:15:28.980
That's the pattern that we've been tracking for
00:15:28.980 --> 00:15:31.720
weeks. Agent credentials moving from store this
00:15:31.720 --> 00:15:35.679
token forever to prove who you are and get scoped
00:15:35.679 --> 00:15:38.600
access when you need it. Short-lived credentials
00:15:38.600 --> 00:15:41.720
don't solve every agent security problem, but
00:15:41.720 --> 00:15:44.360
they beat long-lived secrets sitting around
00:15:44.360 --> 00:15:47.879
waiting to become next quarter's incident. Fourth,
00:15:48.039 --> 00:15:52.009
Amazon S3 annotations are here. mutable, queryable
00:15:52.009 --> 00:15:56.149
context attached directly to S3 objects. Sounds
00:15:56.149 --> 00:15:59.909
dull, but object metadata has driven a lot of
00:15:59.909 --> 00:16:02.850
awkward platform design over the years. Side
00:16:02.850 --> 00:16:06.470
tables, DynamoDB metadata stores, Lambda sync
00:16:06.470 --> 00:16:10.490
jobs, custom catalogs, and constant drift between
00:16:10.490 --> 00:16:14.029
the object and whatever's describing it. If annotations
00:16:14.029 --> 00:16:16.720
shrink that glue layer, That's worth watching.
00:16:16.820 --> 00:16:20.259
Object metadata is quietly becoming a first-class
00:16:20.259 --> 00:16:24.000
platform layer, especially for data, AI, search,
00:16:24.200 --> 00:16:27.799
and agent workflows that need to know what an
00:16:27.799 --> 00:16:38.779
object is, not just where it lives. The human
00:16:38.779 --> 00:16:42.340
closer this week comes from a Marc Brooker post
00:16:42.340 --> 00:16:47.370
about waiting, latency, MTTR, and why averages
00:16:47.370 --> 00:16:50.990
can lie. The point is that your users don't experience
00:16:50.990 --> 00:16:54.070
your averages the way your dashboards report
00:16:54.070 --> 00:16:57.570
them. You measure mean latency, mean time to
00:16:57.570 --> 00:17:00.529
recovery, average outage duration. But people
00:17:00.529 --> 00:17:03.590
are far more likely to land in the long waits
00:17:03.590 --> 00:17:07.289
simply because long waits take up more of the
00:17:07.289 --> 00:17:10.970
time. That's the inspection paradox. A 10-minute
00:17:10.970 --> 00:17:15.019
outage catches a few users. A 10-hour outage
00:17:15.019 --> 00:17:18.000
catches a lot of them. Your incident tracker
00:17:18.000 --> 00:17:22.000
counts both as one outage. Your dashboard says
00:17:22.000 --> 00:17:26.279
MTTR looks fine. Your users say they spent all
00:17:26.279 --> 00:17:29.140
morning waiting. Both are true. And that's the
00:17:29.140 --> 00:17:31.460
whole episode, really. When the system breaks,
00:17:31.740 --> 00:17:34.460
nobody experiences your architecture diagram.
00:17:34.759 --> 00:17:37.859
They experience waiting. Waiting for a request.
00:17:38.160 --> 00:17:40.890
Waiting for recovery. Waiting for a credential
00:17:40.890 --> 00:17:44.150
to get revoked. Waiting for a deploy to stop
00:17:44.150 --> 00:17:46.950
failing. Waiting for the control plane to come
00:17:46.950 --> 00:17:50.190
back. Waiting for someone to find the right context.
00:17:50.549 --> 00:17:53.930
So here's the takeaway. Don't only measure the
00:17:53.930 --> 00:17:56.990
system from the server side. Measure it from
00:17:56.990 --> 00:18:00.650
the waiting side. Because your users don't live
00:18:00.650 --> 00:18:04.309
in your average. They live in the tail. And the
00:18:04.309 --> 00:18:07.789
tail is usually where the real reliability story
00:18:07.789 --> 00:18:10.799
is hiding. That's it for this week of Ship It
00:18:10.799 --> 00:18:13.519
Weekly. We covered containerd runtime risk,
00:18:13.799 --> 00:18:17.000
Postgres failover safety, AI incident response,
00:18:17.460 --> 00:18:20.839
EKS control-plane egress, and why your users
00:18:20.839 --> 00:18:23.819
feel the wait more than your dashboards show.
00:18:24.039 --> 00:18:27.019
If this episode was useful, follow or subscribe
00:18:27.019 --> 00:18:29.960
wherever you are watching or listening. If you're
00:18:29.960 --> 00:18:32.940
on YouTube, hit subscribe. If you're in a podcast
00:18:32.940 --> 00:18:36.250
app, follow the show there. And if you know someone
00:18:36.250 --> 00:18:39.109
wrestling with Kubernetes runtime security, database
00:18:39.109 --> 00:18:42.750
failover, AI incident response, or platform control
00:18:42.750 --> 00:18:45.930
planes, send them this one. It genuinely helps
00:18:45.930 --> 00:18:48.549
the show grow, and it helps me keep making this
00:18:48.549 --> 00:18:51.549
for people who actually live with these systems.
00:18:51.849 --> 00:18:54.430
You can find the weekly brief at OnCallBrief.com
00:18:54.430 --> 00:18:57.549
and the full show notes, links, and past
00:18:57.549 --> 00:19:01.230
episodes at ShipItWeekly.fm. I'm Brian Teller
00:19:01.230 --> 00:19:03.670
from Teller's Tech. Thanks for listening. And
00:19:03.670 --> 00:19:06.140
remember, your dashboards measure the average.
00:19:06.339 --> 00:19:08.220
Your users feel the wait.
