전사 🔗
WEBVTT
00:00:00.000 --> 00:00:03.680
An attacker used Qubole's trusted access into
00:00:03.680 --> 00:00:07.379
customer AWS accounts to read secrets and download
00:00:07.379 --> 00:00:11.619
whole S3 buckets. Claude Code's Don't Ask Again
00:00:11.619 --> 00:00:14.619
button has been saving passwords and API keys
00:00:14.619 --> 00:00:18.120
into a file that keeps turning up on GitHub and
00:00:18.120 --> 00:00:22.379
Docker Hub. And at KubeCon, sandboxing AI agents
00:00:22.379 --> 00:00:26.100
became its own Kubernetes layer. I'm Brian Teller
00:00:26.100 --> 00:00:28.399
from Teller's Tech, and this is Ship It Weekly.
00:00:45.289 --> 00:00:48.170
Welcome back to Ship It Weekly, the show about
00:00:48.170 --> 00:00:52.289
the DevOps, SRE, cloud, platform, and security
00:00:52.289 --> 00:00:54.670
stories that matter when you are the person keeping
00:00:54.670 --> 00:00:57.369
the thing running at three in the morning. For
00:00:57.369 --> 00:00:59.969
the weekly story list and source links, check
00:00:59.969 --> 00:01:03.609
out oncallbrief.com. For past episodes and show
00:01:03.609 --> 00:01:07.250
notes, head over to shipitweekly.fm. This week,
00:01:07.310 --> 00:01:11.349
AWS roles that trusted Qubole, the Claude Code
00:01:11.349 --> 00:01:15.709
file that leaks secrets. KubeCon's new agent
00:01:15.709 --> 00:01:21.170
sandboxes, and a WorkOS breach traced to a researcher's
00:01:21.170 --> 00:01:24.689
AI agent. Then we have a quick lightning round
00:01:24.689 --> 00:01:27.750
and a human closer about why incident resolution
00:01:27.750 --> 00:01:31.937
time is the wrong thing to target. Let's get into it.
00:01:36.686 --> 00:01:39.480
First up, AWS accounts breached through
00:01:39.480 --> 00:01:42.909
a vendor's trusted access. Wiz reported Monday
00:01:42.909 --> 00:01:46.870
that between October 14th and October 26th, an
00:01:46.870 --> 00:01:50.349
attacker used IAM credentials from an AWS account
00:01:50.349 --> 00:01:54.609
tied to Qubole, a data lake platform, to assume
00:01:54.609 --> 00:01:57.609
the cross-account roles customers had created
00:01:57.609 --> 00:02:01.430
to trust it. Once inside, the attacker went straight
00:02:01.430 --> 00:02:04.349
for credentials, Lambda environment variables,
00:02:04.849 --> 00:02:09.030
Secrets Manager values, and SSM parameters. It
00:02:09.030 --> 00:02:12.810
also downloaded S3 data, sometimes entire buckets
00:02:12.810 --> 00:02:16.430
in a single session. How it got Qubole's credentials
00:02:16.430 --> 00:02:20.090
hasn't been disclosed. The interesting part is
00:02:20.090 --> 00:02:24.050
the role trust itself. Every assume role call
00:02:24.050 --> 00:02:27.409
Wiz tied to the attacker included the external
00:02:27.409 --> 00:02:32.310
ID and every one succeeded. The external ID was
00:02:32.310 --> 00:02:35.550
never meant to stop this. It protects against
00:02:35.550 --> 00:02:39.460
the confused deputy problem, where another one
00:02:39.460 --> 00:02:42.360
of the vendor's customers gets the vendor to
00:02:42.360 --> 00:02:45.819
act on your account. It isn't a secret either.
00:02:45.979 --> 00:02:49.300
It shows up in plain text in CloudTrail. If the
00:02:49.300 --> 00:02:52.379
vendor's own credentials are stolen, the external
00:02:52.379 --> 00:02:56.439
ID comes along for free. In most of these environments,
00:02:56.840 --> 00:03:00.120
the attacker's calls were the only assume role
00:03:00.120 --> 00:03:04.319
activity from that account in October. The integration
00:03:04.319 --> 00:03:08.460
was idle. The trust policy was still live. So
00:03:08.460 --> 00:03:12.939
find every IAM role whose trust policy names
00:03:12.939 --> 00:03:16.419
an outside account and ask whether you still
00:03:16.419 --> 00:03:19.639
use that vendor. If you don't, delete the role.
00:03:19.879 --> 00:03:23.240
If you do, narrow the trust to the vendor-specific
00:03:23.240 --> 00:03:27.520
role where they support it. Add a source IP or
00:03:27.520 --> 00:03:31.719
VPC endpoint condition. Wiz says a source IP
00:03:31.719 --> 00:03:35.039
condition alone would have blocked every role
00:03:35.039 --> 00:03:38.460
assumption they saw. If you ever connected Qubole,
00:03:38.580 --> 00:03:42.379
check CloudTrail from October 1st for its assume
00:03:42.379 --> 00:03:46.479
role calls. Then rotate everything those roles
00:03:46.479 --> 00:03:50.259
could have read and revoke active sessions. A
00:03:50.259 --> 00:03:53.659
trust policy keeps working long after you've
00:03:53.659 --> 00:03:55.049
stopped thinking about it.
00:03:59.798 --> 00:04:01.240
Next, Claude Code has
00:04:01.240 --> 00:04:04.419
been writing secrets into a file that ends up
00:04:04.419 --> 00:04:07.969
in public repos and container images. When Claude Code
00:04:07.969 --> 00:04:11.210
asks to run a shell command, and you pick yes
00:04:11.210 --> 00:04:14.889
and don't ask again, it saves an allow rule to
00:04:14.889 --> 00:04:19.930
.claude/settings.local.json in your project
00:04:19.930 --> 00:04:24.430
folder. That rule is the full command, so any
00:04:24.430 --> 00:04:28.050
password or token in it now sits there in plain
00:04:28.050 --> 00:04:32.029
text. On November 6th, Apiiro reported that out
00:04:32.029 --> 00:04:35.689
of about 104,000 public copies of that file
00:04:35.689 --> 00:04:42.040
on GitHub, 1,122 held at least one secret. Those
00:04:42.040 --> 00:04:47.899
included 379 database passwords and 23 AWS keys.
00:04:48.259 --> 00:04:52.500
On Monday, GitGuardian counted more than 31,000
00:04:52.500 --> 00:04:56.300
secrets pushed in that file since January 2025
00:04:56.300 --> 00:05:01.220
and more than 5,300 worked when it found them.
00:05:01.399 --> 00:05:05.240
The part I'd look at closely is Docker. GitGuardian
00:05:05.240 --> 00:05:09.819
found 613 secrets in public Docker Hub images,
00:05:10.079 --> 00:05:14.500
99 of them valid. If your Dockerfile does COPY
00:05:14.500 --> 00:05:18.019
Load full transcript
dot dot, it copies your working tree, not your
00:05:18.019 --> 00:05:22.680
Git index. So a file that Git ignores still ships
00:05:22.680 --> 00:05:26.939
in the image, unless .dockerignore excludes
00:05:26.939 --> 00:05:30.439
it. Claude Code does add the file to your global
00:05:30.439 --> 00:05:34.519
Git excludes, but that only covers your own laptop,
00:05:35.040 --> 00:05:39.000
not a teammate's machine, a CI runner, or an
00:05:39.000 --> 00:05:42.980
image build. And Apiiro says none of the related
00:05:42.980 --> 00:05:46.379
bug reports have gotten a reply from Anthropic.
00:05:46.680 --> 00:05:51.360
So rotate first, because git history, forks,
00:05:51.379 --> 00:05:56.019
and image layers all keep copies. Then git rm
00:05:56.019 --> 00:06:00.920
dash dash cached the file. Add .claude to your
00:06:00.920 --> 00:06:06.139
.gitignore and .dockerignore and delete any
00:06:06.139 --> 00:06:09.779
allow role with a literal secret in it. After
00:06:09.779 --> 00:06:12.699
that, use environment variables for credentials
00:06:12.699 --> 00:06:17.494
and approve command patterns instead of exact commands.
00:06:22.243 --> 00:06:25.120
Third, KubeCon turned sandboxing AI
00:06:25.120 --> 00:06:29.139
agents into a standard Kubernetes layer. At KubeCon
00:06:29.139 --> 00:06:33.019
this week, the CNCF took in two new sandbox projects
00:06:33.019 --> 00:06:36.980
for agents. Agent Substrate, contributed by Google,
00:06:37.519 --> 00:06:41.740
is a zero-trust runtime for AI agents on Kubernetes.
00:06:42.199 --> 00:06:46.779
OpenShell, led by NVIDIA, enforces policy on
00:06:46.779 --> 00:06:50.620
which files, networks, credentials, and model
00:06:50.620 --> 00:06:54.600
endpoints an agent can reach. And gVisor, Google's
00:06:54.600 --> 00:06:57.959
application kernel, was contributed too. AWS
00:06:57.959 --> 00:07:02.180
launched Nitro Sandboxes for EKS the same week.
00:07:02.579 --> 00:07:06.079
Set a pod's runtime class name to isolated sandbox,
00:07:06.399 --> 00:07:09.720
and that pod runs in its own lightweight virtual
00:07:09.720 --> 00:07:13.519
machine on Nitro with its own kernel. It needs
00:07:13.519 --> 00:07:17.759
Kubernetes 1.36 or later, and it's built into
00:07:17.759 --> 00:07:22.519
EKS Auto Mode. AWS calls it available today without
00:07:22.519 --> 00:07:25.879
saying GA or preview. Here's why this matters.
00:07:26.100 --> 00:07:29.660
A normal container shares the host kernel. And
00:07:29.660 --> 00:07:32.259
that's been fine for code your team wrote and
00:07:32.259 --> 00:07:35.800
reviewed. An agent runs code nobody reviewed.
00:07:36.079 --> 00:07:40.060
And a container escape puts it on the host. Next
00:07:40.060 --> 00:07:44.560
to everyone else's pods. gVisor puts a user space
00:07:44.560 --> 00:07:49.019
kernel in between. And Nitro and microVMs use
00:07:49.019 --> 00:07:52.680
a separate virtual machine. Either way, the boundary
00:07:52.680 --> 00:07:55.860
moves off the shared kernel. Two
cautions though.
00:07:56.180 --> 00:08:01.569
Sandbox stage at the CNCF means early. So treat
00:08:01.569 --> 00:08:05.649
Agent Substrate and OpenShell as things to evaluate,
00:08:06.009 --> 00:08:10.189
not deploy. And isolation limits what the agent
00:08:10.189 --> 00:08:13.670
can do to the host, not what it can reach with
00:08:13.670 --> 00:08:16.629
the credentials you hand it. If you run coding
00:08:16.629 --> 00:08:20.470
agents or untrusted CI on Kubernetes, the default
00:08:20.470 --> 00:08:23.709
runtime plus a broad service account plus open
00:08:23.709 --> 00:08:27.035
egress is the combination to fix first.
00:08:31.784 --> 00:08:36.490
Fourth, a breach that WorkOS traced to an AI agent. On
00:08:36.490 --> 00:08:39.370
November 2nd, WorkOS, the developer platform,
00:08:39.830 --> 00:08:43.330
disclosed unauthorized access to an internal
00:08:43.330 --> 00:08:47.190
sales tool. The first access was October 25th,
00:08:47.210 --> 00:08:49.789
and nobody noticed until records were pulled
00:08:49.789 --> 00:08:53.289
in bulk on October 30th. The data was sales and
00:08:53.289 --> 00:08:56.289
support material, including excerpts of sales
00:08:56.289 --> 00:08:59.539
calls and customer Slack conversations. Then
00:08:59.539 --> 00:09:02.799
on November 6th, a security researcher came forward.
00:09:03.019 --> 00:09:05.580
They said they were running benchmark evaluations
00:09:05.580 --> 00:09:09.559
with ARTEX, an AI offensive security framework,
00:09:09.679 --> 00:09:13.179
when their agent escaped its isolated test environment
00:09:13.179 --> 00:09:18.340
and ended up at WorkOS. WorkOS and an independent
00:09:18.340 --> 00:09:21.240
forensics firm matched the researcher's evidence
00:09:21.240 --> 00:09:24.899
to its logs, and they're confident it was this
00:09:24.899 --> 00:09:28.360
person. The escape itself is the researcher's
00:09:28.360 --> 00:09:31.629
account. Nobody has said how it happened, and
00:09:31.629 --> 00:09:34.450
the claim that the data was deleted hasn't been
00:09:34.450 --> 00:09:37.409
independently verified. The more useful part
00:09:37.409 --> 00:09:42.889
is WorkOS's own list of what let it in, and none
00:09:42.889 --> 00:09:46.909
of it needed a clever agent. The tool's API was
00:09:46.909 --> 00:09:50.330
reachable from the internet. It accepted a predictable
00:09:50.330 --> 00:09:53.509
credential meant for local development. There
00:09:53.509 --> 00:09:58.049
was no rate limiting, and five days went by before
00:09:58.049 --> 00:10:01.549
anyone noticed. It was a legacy internal tool
00:10:01.549 --> 00:10:05.549
and most of us have one of those. WorkOS has
00:10:05.549 --> 00:10:08.809
since cut all of its internal tools off from
00:10:08.809 --> 00:10:12.389
the internet and now runs ARTEX against its own
00:10:12.389 --> 00:10:15.850
systems. The sandboxes in the last story are
00:10:15.850 --> 00:10:18.649
about containing the agents you run. This is
00:10:18.649 --> 00:10:22.389
the other side. Agents someone else runs, on
00:10:22.389 --> 00:10:25.440
purpose or not, that never get tired of trying
00:10:25.440 --> 00:10:28.005
the dev credential you forgot about.
00:10:35.554 --> 00:10:39.850
Quick lightning round. First, Microsoft's November Patch Tuesday
00:10:39.850 --> 00:10:43.769
came with one bug already under attack. It's
00:10:43.769 --> 00:10:46.929
an Outlook spoofing flaw. Open a crafted Office
00:10:46.929 --> 00:10:51.830
file and classic Outlook leaks your NTLMv2 hash,
00:10:52.129 --> 00:10:56.409
which can be used to impersonate you. CISA's
00:10:56.409 --> 00:11:00.220
federal deadline is November 24th. And the fix
00:11:00.220 --> 00:11:03.360
ships with Office updates, not Windows Update.
00:11:03.600 --> 00:11:07.879
Azure DevOps Server also has a critical 9.9
00:11:07.879 --> 00:11:11.919
remote code execution bug. Second, N-able. On
00:11:11.919 --> 00:11:16.000
Sunday, a researcher published 33 N-central exploits
00:11:16.000 --> 00:11:20.220
he calls Able2Shell. N-able took hosted N-central
00:11:20.220 --> 00:11:25.559
offline on Monday and shipped 2026.4 Hotfix 2
00:11:25.559 --> 00:11:30.240
on Tuesday with fixes for seven CVEs. It reports
00:11:30.240 --> 00:11:33.940
scanning, but no successful exploitation. If
00:11:33.940 --> 00:11:37.179
you run it on-prem, keep it offline until it's
00:11:37.179 --> 00:11:41.039
upgraded. Then update agents and probes. Third,
00:11:41.279 --> 00:11:45.419
KubeVirt graduated at the CNCF. It runs virtual
00:11:45.419 --> 00:11:49.159
machines alongside containers on one Kubernetes
00:11:49.159 --> 00:11:53.360
control plane. And recent releases added confidential
00:11:53.360 --> 00:11:57.740
VMs and GPU claims through dynamic resource
00:11:57.740 --> 00:12:01.639
allocation. If you are planning a move off VMware
00:12:01.639 --> 00:12:05.480
onto Kubernetes, that's a maturity signal. And
00:12:05.480 --> 00:12:09.559
fourth, Red Hat made OpenShift 5 generally available
00:12:09.559 --> 00:12:14.159
on Kubernetes 1.36 as an in-place update from
00:12:14.159 --> 00:12:18.759
4.22. Its AI troubleshooting is GA, with the
00:12:18.759 --> 00:12:21.659
more autonomous agent features still in tech
00:12:21.659 --> 00:12:26.587
preview. Standard support goes from 18 to 24 months.
00:12:35.136 --> 00:12:37.210
The human closer this week comes from
00:12:37.210 --> 00:12:40.230
Brent Chapman in a piece featured in SRE Weekly
00:12:40.230 --> 00:12:44.009
called Why Incident Resolution Time Isn't Something
00:12:44.009 --> 00:12:47.269
You Should Target. His argument is that resolution
00:12:47.269 --> 00:12:50.830
time mostly depends on the incident, not on your
00:12:50.830 --> 00:12:54.529
process. A config rollback and a cascading failure
00:12:54.529 --> 00:12:58.190
at peak traffic are different problems. So one
00:12:58.190 --> 00:13:02.779
target is either too easy or impossible. Once
00:13:02.779 --> 00:13:06.080
it's a target, Goodhart's law
takes over. The
00:13:06.080 --> 00:13:09.179
number gets managed instead of the system. His
00:13:09.179 --> 00:13:12.000
example is fire departments. They set targets
00:13:12.000 --> 00:13:16.179
for dispatch, turnout, and travel time. But none
00:13:16.179 --> 00:13:19.600
for time on scene. Because a kitchen grease fire
00:13:19.600 --> 00:13:23.460
and a warehouse fire aren't the same job. So
00:13:23.460 --> 00:13:26.419
target the steps that look the same every time
00:13:26.419 --> 00:13:30.279
and that you control. How fast a page is acknowledged.
00:13:30.879 --> 00:13:33.779
How fast the incident channel is staffed. The
00:13:33.779 --> 00:13:37.299
first customer update. Track diagnosis and mitigation
00:13:37.299 --> 00:13:42.059
as trends, and dig into the outliers. His test:
00:13:42.059 --> 00:13:45.539
if you can write down two or three specific things
00:13:45.539 --> 00:13:48.860
you change to improve a step, it's controllable
00:13:48.860 --> 00:13:53.000
enough to target. That test fits this week. You
00:13:53.000 --> 00:13:55.019
don't control whether a vendor you connected
00:13:55.019 --> 00:13:58.279
years ago gets breached. You do control whether
00:13:58.279 --> 00:14:02.059
that role still exists and which addresses it
00:14:02.059 --> 00:14:05.460
accepts. You don't control what a coding tool
00:14:05.460 --> 00:14:09.659
writes to disk. You do control your .dockerignore
00:14:09.659 --> 00:14:14.159
and what your scanners check. And WorkOS couldn't
00:14:14.159 --> 00:14:18.240
control an agent showing up, but five days to notice
00:14:18.240 --> 00:14:21.460
is something a rate limit and a volume alert
00:14:21.460 --> 00:14:24.600
can change. The parts that are yours to change
00:14:24.600 --> 00:14:28.539
are usually smaller and more boring than a number
00:14:28.539 --> 00:14:31.970
on a dashboard. That's it for this week of Ship
00:14:31.970 --> 00:14:34.669
It Weekly. Follow or subscribe wherever you are
00:14:34.669 --> 00:14:37.210
watching or listening. You can find the weekly
00:14:37.210 --> 00:14:41.029
story list and source links at oncallbrief.com
00:14:41.029 --> 00:14:44.809
and past episodes and show notes at shipitweekly.fm.
00:14:44.809 --> 00:14:48.970
One quick note, Ship It Weekly is a finalist
00:14:48.970 --> 00:14:53.110
in this year's DevOps Dozen for Best DevOps Community
00:14:53.110 --> 00:14:57.129
Program or Event. And I'm a finalist for Enterprise
00:14:57.129 --> 00:15:00.190
DevOps Champion of the Year. The public vote
00:15:00.190 --> 00:15:03.870
counts, and it's open through December 31st.
00:15:03.870 --> 00:15:06.830
The link's in the show notes. Thank you. In this
00:15:06.830 --> 00:15:10.649
episode, we covered AWS roles that trusted Qubole,
00:15:10.750 --> 00:15:14.870
Claude Code's leaky settings file, KubeCon's agent
00:15:14.870 --> 00:15:19.090
sandboxes, and the WorkOS breach. I'm Brian Teller
00:15:19.090 --> 00:15:21.529
from Teller's Tech. Thanks for listening. And
00:15:21.529 --> 00:15:24.769
remember, target what you can change and track
00:15:24.769 --> 00:15:25.330
the rest.