AWS Retires DevOps Guru: What the End of Support Means, Kubernetes Cross-Namespace CVE-2026-2270, Node.js Undici WebSocket DoS & Cloudflare’s New CLI for AI Agents

WEBVTT
00:00:00.000 --> 00:00:03.680
AWS is retiring DevOps Guru and pointing customers
00:00:03.680 --> 00:00:07.339
toward a new generation of AI-assisted operations.
00:00:08.099 --> 00:00:11.140
A Kubernetes controller vulnerability can let
00:00:11.140 --> 00:00:15.019
namespace-scoped permissions create a pod somewhere
00:00:15.019 --> 00:00:18.820
they should not be able to. And Cloudflare says
00:00:18.820 --> 00:00:22.100
AI agents now account for almost half of the
00:00:22.100 --> 00:00:25.960
usage of its Wrangler CLI. I’m Brian Teller from
00:00:25.960 --> 00:00:45.539
Teller's Tech, and this is Ship It Weekly. Welcome
00:00:45.539 --> 00:00:48.579
back to Ship It Weekly, the show about the DevOps,
00:00:48.899 --> 00:00:52.700
SRE, cloud, platform, and security stories that
00:00:52.700 --> 00:00:54.859
matter when you are the person keeping the thing
00:00:54.859 --> 00:00:57.740
running at three in the morning. For the weekly
00:00:57.740 --> 00:01:01.219
story list and source links, check out
00:01:01.219 --> 00:01:04.620
OnCallBrief.com. For past episodes and show notes, head
00:01:04.620 --> 00:01:08.859
over to ShipItWeekly.fm. This is episode 69.
00:01:09.120 --> 00:01:12.560
Nice. Anyway, quick programming note before we
00:01:12.560 --> 00:01:15.760
get into it. I’ll be at AWS headquarters in Arlington
00:01:15.760 --> 00:01:20.780
on Friday for AWS DMV Community Day. I’m speaking
00:01:20.780 --> 00:01:22.900
and recording some interviews while I’m there,
00:01:23.019 --> 00:01:25.760
so you’ll see some of those conversations coming
00:01:25.760 --> 00:01:29.760
to the show soon. This week, AWS is officially
00:01:29.760 --> 00:01:33.540
putting an end date on DevOps Guru and recommending
00:01:33.540 --> 00:01:36.760
customers move toward CloudWatch and the newer
00:01:36.760 --> 00:01:40.500
Amazon DevOps Agent. Kubernetes disclosed a controller
00:01:40.500 --> 00:01:43.870
vulnerability that can cross a namespace boundary
00:01:43.870 --> 00:01:48.030
under some specific permissions. We have a Node.js
00:01:48.030 --> 00:01:52.069
and Undici WebSocket issue where a malicious
00:01:52.069 --> 00:01:56.510
server can crash the client process. And Cloudflare
00:01:56.510 --> 00:02:00.390
has launched a new CLI as AI agents become major
00:02:00.390 --> 00:02:03.890
consumers of its developer tooling. Then we have
00:02:03.890 --> 00:02:06.969
a quick lightning round and a human closer from
00:02:06.969 --> 00:02:10.449
SRE Weekly about why some categories of production
00:02:10.449 --> 00:02:14.330
failure may never go away. Let's get into it.
00:02:18.949 --> 00:02:23.189
First up, AWS is retiring Amazon DevOps Guru.
00:02:23.449 --> 00:02:26.090
The service will stop accepting new customers
00:02:26.090 --> 00:02:30.710
on October 29, 2026, and reach end of support
00:02:30.710 --> 00:02:35.090
on September 30, 2027. Existing customers can
00:02:35.090 --> 00:02:37.689
continue using it until then, but there will
00:02:37.689 --> 00:02:41.150
be no new features. After end of support, the
00:02:41.150 --> 00:02:44.949
console and APIs go away and previously generated
00:02:44.949 --> 00:02:49.050
insights will no longer be retrievable. DevOps
00:02:49.050 --> 00:02:54.210
Guru was AWS’s earlier generation of AI for operations.
00:02:54.590 --> 00:02:57.969
It used machine learning to analyze operational
00:02:57.969 --> 00:03:01.990
data, identify abnormal behavior, correlate signals,
00:03:02.330 --> 00:03:05.949
and help engineers investigate problems. AWS
00:03:05.949 --> 00:03:09.729
is now telling customers to split those capabilities
00:03:09.729 --> 00:03:14.090
into two directions. For monitoring, alerting,
00:03:14.150 --> 00:03:17.669
and anomaly detection, move toward CloudWatch.
00:03:17.990 --> 00:03:22.150
For AI-assisted investigation, AWS recommends
00:03:22.150 --> 00:03:26.530
looking at Amazon DevOps Agent. So if you use
00:03:26.530 --> 00:03:29.610
DevOps Guru today, this is more than swapping
00:03:29.610 --> 00:03:33.310
one AWS service for another. You need to figure
00:03:33.310 --> 00:03:36.009
out which capabilities you are actually using
00:03:36.539 --> 00:03:40.580
and where each one moves. There is also an infrastructure-as-code
00:03:40.580 --> 00:03:44.860
problem to deal with. If you have DevOps
00:03:44.860 --> 00:03:49.259
Guru resources declared in CloudFormation, CDK,
00:03:49.639 --> 00:03:53.479
Terraform, or automation built around its API,
00:03:53.979 --> 00:03:56.900
you cannot leave those definitions sitting around
00:03:56.900 --> 00:04:01.210
after the service disappears. AWS specifically
00:04:01.210 --> 00:04:04.990
warns that CloudFormation operations involving
00:04:04.990 --> 00:04:09.270
withdrawn resource types can fail after end of
00:04:09.270 --> 00:04:13.530
support. You have a year, so this is not an emergency.
00:04:13.870 --> 00:04:17.730
But managed services are still dependencies.
00:04:18.170 --> 00:04:20.949
You may not maintain the software underneath
00:04:20.949 --> 00:04:24.089
them, but you still build infrastructure, automation,
00:04:24.370 --> 00:04:28.230
and operational processes around their APIs.
00:04:29.079 --> 00:04:37.910
Eventually, some of those APIs disappear. Next,
00:04:38.170 --> 00:04:44.149
Kubernetes disclosed CVE-2026-2270. This is
00:04:44.149 --> 00:04:47.470
a vulnerability involving the StatefulSet controller
00:04:47.470 --> 00:04:51.050
and ControllerRevisions. Under specific conditions,
00:04:51.490 --> 00:04:54.430
somebody with namespace-scoped write permissions
00:04:54.430 --> 00:04:58.089
for StatefulSets and ControllerRevisions can
00:04:58.089 --> 00:05:01.629
cause the controller to create a pod in another
00:05:01.629 --> 00:05:04.980
namespace. That sounds particularly bad because
00:05:04.980 --> 00:05:07.920
namespace-scoped permissions are supposed to
00:05:07.920 --> 00:05:11.720
stay inside the namespace. There are some important
00:05:11.720 --> 00:05:15.040
limitations. The resulting pod will normally
00:05:15.040 --> 00:05:18.180
be deleted by Kubernetes garbage collection unless
00:05:18.180 --> 00:05:21.139
the attacker can construct a valid StatefulSet
00:05:21.139 --> 00:05:25.139
owner reference. To do that, they need the UID
00:05:25.139 --> 00:05:28.699
of an existing StatefulSet in the target namespace.
00:05:29.240 --> 00:05:33.300
The vulnerability is rated Medium with a CVSS
00:05:33.300 --> 00:05:37.639
score of 5.9 and requires substantial existing
00:05:37.639 --> 00:05:40.939
permissions. So this is not somebody outside
00:05:40.939 --> 00:05:44.660
the cluster suddenly getting arbitrary pod creation.
00:05:45.079 --> 00:05:48.079
Kubernetes describes it as a confused deputy
00:05:48.079 --> 00:05:51.540
attack. The user does not directly have permission
00:05:51.540 --> 00:05:55.019
to create the pod in the target namespace. The
00:05:55.019 --> 00:05:58.279
controller does. The user manipulates an object
00:05:58.279 --> 00:06:01.199
they are allowed to control, and the controller
00:06:01.199 --> 00:06:04.220
performs the more privileged operation for them.
00:06:04.379 --> 00:06:07.519
That is worth considering when reviewing Kubernetes
00:06:07.519 --> 00:06:11.379
RBAC. Permissions on an object do not always
00:06:11.379 --> 00:06:15.480
end with that object. Controllers react to resources
00:06:15.480 --> 00:06:18.860
and perform additional operations using their
00:06:18.860 --> 00:06:22.579
own authority. Sometimes, the important question
00:06:22.579 --> 00:06:26.860
is not just, “What can this identity create?” It
00:06:26.860 --> 00:06:29.740
is, “What can this identity convince a controller
00:06:29.740 --> 00:06:38.170
to create?” Third, we have a denial-of-service
00:06:38.170 --> 00:06:42.350
vulnerability in Undici. Undici is the HTTP client
00:06:42.350 --> 00:06:46.769
used by Node.js, and its WebSocket implementation
00:06:46.769 --> 00:06:51.550
backs Node's built -in globalThis.WebSocket.
00:06:51.790 --> 00:06:57.910
CVE-2026-85024 involves WebSocket compression.
00:06:58.550 --> 00:07:02.269
A malicious or compromised WebSocket server can
00:07:02.269 --> 00:07:05.269
send a specially constructed compressed message
00:07:05.269 --> 00:07:08.970
that exceeds the decompressed payload limit and
00:07:08.970 --> 00:07:13.110
contains malformed DEFLATE data. Inside Undici,
00:07:13.410 --> 00:07:17.110
cleanup for the size limit removes an error listener
00:07:17.110 --> 00:07:21.110
from an internal InflateRaw stream. That stream
00:07:21.110 --> 00:07:24.949
can still generate an error. When it does, Node
00:07:24.949 --> 00:07:28.430
sees an unhandled error event and terminates
00:07:28.430 --> 00:07:31.589
the process. Adding an error handler to your
00:07:31.589 --> 00:07:35.170
application's WebSocket does not fix this. The
00:07:35.170 --> 00:07:38.310
failure occurs inside the decompression object,
00:07:38.610 --> 00:07:42.810
so the public WebSocket error and close handlers
00:07:42.810 --> 00:07:46.670
never get an opportunity to catch it. The vulnerability
00:07:46.670 --> 00:07:52.410
is rated Medium with a CVSS score of 5.9. The
00:07:52.410 --> 00:07:56.470
advisory says that roughly 130 kilobytes sent
00:07:56.470 --> 00:07:59.670
over the network can expand beyond the internal
00:07:59.670 --> 00:08:03.589
limit and terminate the process. If the application
00:08:03.589 --> 00:08:06.490
automatically reconnects, it could potentially
00:08:06.490 --> 00:08:10.550
end up in a crash loop. Affected Undici branches
00:08:10.550 --> 00:08:18.829
have fixes in 6.28.1, 7.29.1, and 8.10.2.
00:08:19.350 --> 00:08:22.029
The operational point here is pretty simple.
00:08:22.250 --> 00:08:25.389
Your application, being the client, does not
00:08:25.389 --> 00:08:28.670
make the server trusted. Clients still parse,
00:08:28.889 --> 00:08:33.289
decompress, deserialize, and process data coming
00:08:33.289 --> 00:08:36.750
from the other side. If you use WebSockets from
00:08:36.750 --> 00:08:39.769
Node, especially against endpoints outside your
00:08:39.769 --> 00:08:43.049
control, check which version of Undici you are
00:08:43.049 --> 00:08:50.679
running. Fourth, Cloudflare has launched a new
00:08:50.679 --> 00:08:54.990
CLI called `cf`. Normally, another CLI would not
00:08:54.990 --> 00:08:57.789
make the main story list. The numbers behind
00:08:57.789 --> 00:09:02.330
this one are why it did. Cloudflare says AI agents
00:09:02.330 --> 00:09:07.090
accounted for about 25 percent of Wrangler usage in
00:09:07.090 --> 00:09:11.649
March. Last week, that reached 48 percent. Agents also
00:09:11.649 --> 00:09:14.570
use almost twice as many distinct commands per
00:09:14.570 --> 00:09:18.350
day and are nearly four times as likely to use
00:09:18.350 --> 00:09:21.980
six or more commands. Wrangler was primarily
00:09:21.980 --> 00:09:25.960
built for people. It exposes around 280 operations,
00:09:26.379 --> 00:09:29.620
while Cloudflare has thousands of API operations.
00:09:30.139 --> 00:09:33.179
Instead of manually adding commands for all of
00:09:33.179 --> 00:09:37.279
those, Cloudflare built `cf` around its API definitions.
00:09:37.960 --> 00:09:41.860
Cloudflare also open-sourced Forge, the generation
00:09:41.860 --> 00:09:45.919
system behind the tooling. We have spent a lot
00:09:45.919 --> 00:09:48.960
of time talking about agents using infrastructure
00:09:48.960 --> 00:09:52.559
tools. Now we are starting to see infrastructure
00:09:52.559 --> 00:09:56.679
tools change because agents are using them. If
00:09:56.679 --> 00:10:00.019
almost half of your CLI usage is coming from
00:10:00.019 --> 00:10:03.419
software, things like consistent commands, structured
00:10:03.419 --> 00:10:07.500
output, discoverability, and complete API coverage
00:10:07.500 --> 00:10:11.799
start becoming more important. The CLI is no
00:10:11.799 --> 00:10:15.460
longer only an interface for the engineer sitting
00:10:15.460 --> 00:10:18.960
at the terminal. It is also becoming an interface
00:10:18.960 --> 00:10:22.360
for software operating on the engineer's behalf.
00:10:29.850 --> 00:10:33.529
Quick lightning round. First, Kubernetes disclosed
00:10:33.529 --> 00:10:36.889
another vulnerability affecting Windows nodes.
00:10:37.549 --> 00:10:43.850
CVE-2026-76654 involves a pod volume mount
00:10:43.850 --> 00:10:47.889
using `subPath` with a symbolic link pointing to
00:10:47.889 --> 00:10:51.029
an attacker-controlled network share. The Windows
00:10:51.029 --> 00:10:55.210
kubelet can follow that link to a UNC path and
00:10:55.210 --> 00:10:59.129
attempt NTLM authentication to the remote share.
00:10:59.450 --> 00:11:04.370
That can expose the NetNTLMv2 hash of the account
00:11:04.370 --> 00:11:06.789
running the kubelet, which could potentially
00:11:06.789 --> 00:11:10.929
be cracked or relayed if the node is domain joined.
00:11:11.169 --> 00:11:16.090
It is rated Medium with a CVSS score of 5.8.
00:11:16.590 --> 00:11:19.970
Second, GitHub now lets repository administrators
00:11:19.970 --> 00:11:24.090
configure custom runners for Dependabot version
00:11:24.090 --> 00:11:27.750
and security updates. You can select the runner
00:11:27.750 --> 00:11:31.090
type, a custom label, and an optional runner
00:11:31.090 --> 00:11:34.509
group. That is useful when Dependabot needs access
00:11:34.509 --> 00:11:38.230
to private package registries or other resources
00:11:38.230 --> 00:11:42.070
only reachable from your own environment. And
00:11:42.070 --> 00:11:45.649
third, GitHub has added external custom properties
00:11:45.649 --> 00:11:49.169
for repositories. An external system of record
00:11:49.690 --> 00:11:53.590
like a CMDB or internal developer portal, can
00:11:53.590 --> 00:11:56.389
now push information like ownership, service
00:11:56.389 --> 00:12:00.330
tier, lifecycle stage, or compliance status into
00:12:00.330 --> 00:12:04.470
GitHub. Those values remain read-only in GitHub
00:12:04.470 --> 00:12:07.870
while the external system stays the source of
00:12:07.870 --> 00:12:11.590
truth. Because custom properties can also be used
00:12:11.590 --> 00:12:14.230
with filtering and rulesets. Service catalog
00:12:14.230 --> 00:12:17.429
metadata can now feed directly into repository
00:12:17.429 --> 00:12:28.789
governance. The human closer this week comes
00:12:28.789 --> 00:12:31.850
from Lorin Hochstein and a piece featured in
00:12:31.850 --> 00:12:35.690
SRE Weekly called Omnipresent Availability Risks
00:12:35.690 --> 00:12:39.289
in Cloud Software. The argument is that there
00:12:39.289 --> 00:12:42.309
are some categories of production incidents we
00:12:42.309 --> 00:12:45.750
probably are not going to engineer away. Take
00:12:45.750 --> 00:12:50.429
saturation. Every resource is finite. CPU, memory,
00:12:50.750 --> 00:12:54.450
disk, database connections, queue depth, network
00:12:54.450 --> 00:12:58.159
capacity. Eventually, something can hit a limit.
00:12:58.360 --> 00:13:02.740
Then there is networking. Cloud software is distributed
00:13:02.740 --> 00:13:06.039
software, and distributed software depends on
00:13:06.039 --> 00:13:10.179
networks. Networking failures can also have enormous
00:13:10.179 --> 00:13:14.039
blast radius. Security creates another conflict.
00:13:14.460 --> 00:13:17.919
Availability says legitimate users should be
00:13:17.919 --> 00:13:21.519
able to access something. Security says illegitimate
00:13:21.519 --> 00:13:27.500
users should not. Sometimes the security control doing exactly what it was designed
00:13:27.500 --> 00:13:31.539
to do can cause the outage. An expired certificate
00:13:31.539 --> 00:13:35.919
is probably the simplest example. Then there
00:13:35.919 --> 00:13:39.559
are changes you cannot simply stop making. Sometimes
00:13:39.559 --> 00:13:43.399
you have to migrate the database. Sometimes you
00:13:43.399 --> 00:13:46.480
have to replace infrastructure. And sometimes
00:13:46.480 --> 00:13:49.799
production is already broken and making another
00:13:49.799 --> 00:13:53.419
change is the only way to recover it. Even the
00:13:53.419 --> 00:13:57.179
systems we build to improve reliability add complexity.
00:13:57.779 --> 00:14:01.820
Retries, failover, replication, health checks,
00:14:02.080 --> 00:14:05.539
circuit breakers. They help us recover from failures,
00:14:05.720 --> 00:14:09.179
but they also create more states and interactions
00:14:09.179 --> 00:14:12.620
that can fail. That changes how I think about
00:14:12.620 --> 00:14:15.159
one of the questions that comes up after almost
00:14:15.159 --> 00:14:18.440
every incident. How do we make sure this never
00:14:18.440 --> 00:14:21.879
happens again? For a specific preventable failure,
00:14:22.320 --> 00:14:25.700
absolutely. Fix it. Add the guardrails. Improve
00:14:25.700 --> 00:14:28.960
the alert. Remove the dangerous manual step.
00:14:29.279 --> 00:14:32.679
But we are not going to eliminate finite resources.
00:14:33.159 --> 00:14:36.659
We are not going to eliminate networks. We are
00:14:36.659 --> 00:14:40.179
not going to eliminate security boundaries. And
00:14:40.179 --> 00:14:42.740
we are definitely not going to eliminate change.
00:14:43.179 --> 00:14:46.879
Incident response itself has to be part of reliability
00:14:46.879 --> 00:14:50.480
engineering. How quickly do we notice the problem?
00:14:51.019 --> 00:14:53.840
How quickly can we understand the blast radius?
00:14:54.159 --> 00:14:58.120
Can we get the right people involved? Can we
00:14:58.120 --> 00:15:01.399
safely make changes while the system is already
00:15:01.399 --> 00:15:05.460
behaving unpredictably? And can we learn from
00:15:05.460 --> 00:15:08.559
what happened afterward? Sometimes improving
00:15:08.559 --> 00:15:12.039
reliability means preventing the next incident.
00:15:12.139 --> 00:15:15.600
Sometimes it means getting much better at handling
00:15:15.600 --> 00:15:19.470
the incidents you cannot prevent. That is it for
00:15:19.470 --> 00:15:23.370
this week's Ship It Weekly. We covered AWS retiring
00:15:23.370 --> 00:15:27.470
DevOps Guru and pointing customers toward CloudWatch
00:15:27.470 --> 00:15:30.950
and DevOps Agent. A Kubernetes vulnerability
00:15:30.950 --> 00:15:35.409
that can cross namespace boundaries. A WebSocket
00:15:35.409 --> 00:15:40.809
bug that can crash a Node.js process. And Cloudflare
00:15:40.809 --> 00:15:45.509
redesigning its CLI as AI agents become a major
00:15:45.509 --> 00:15:49.360
part of its usage. Plus, another Kubernetes security
00:15:49.360 --> 00:15:53.720
issue on Windows nodes, custom Dependabot runners,
00:15:54.019 --> 00:15:58.299
and GitHub pulling service-catalog metadata into
00:15:58.299 --> 00:16:01.799
repository governance. Follow or subscribe wherever
00:16:01.799 --> 00:16:04.779
you are watching or listening. You can find the
00:16:04.779 --> 00:16:08.200
weekly story list and source links at OnCallBrief
00:16:08.200 --> 00:16:13.179
.com and past episodes and show notes at ShipItWeekly
00:16:13.179 --> 00:16:16.820
.fm. I’m Brian Teller from Teller's Tech. Thanks
00:16:16.820 --> 00:16:19.830
for listening. And remember, you cannot engineer
00:16:19.830 --> 00:16:23.649
away every failure, but you can get a lot better
00:16:23.649 --> 00:16:25.190
at what happens next.