TRANSCRIPTION 🔗
WEBVTT
00:00:00.000 --> 00:00:04.799
GitHub Actions jobs can now run on AWS, and if
00:00:04.799 --> 00:00:08.820
your firewall only trusts Azure IP ranges, some
00:00:08.820 --> 00:00:12.839
pipelines could start failing. A bug in the Brotli
00:00:12.839 --> 00:00:16.839
decoder is being exploited, and your code might
00:00:16.839 --> 00:00:20.219
be decompressing Brotli without ever mentioning
00:00:20.219 --> 00:00:24.079
it. And attackers with stolen GitHub tokens are
00:00:24.079 --> 00:00:27.539
pulling Actions secrets out of private repos,
00:00:27.579 --> 00:00:31.160
then... deleting the evidence. I'm Brian Teller
00:00:31.160 --> 00:00:33.640
from Teller's Tech, and this is Ship It Weekly.
00:00:50.380 --> 00:00:53.539
Welcome back to Ship It Weekly, the show about
00:00:53.539 --> 00:00:57.859
the DevOps, SRE, cloud, platform, and security
00:00:57.859 --> 00:01:00.280
stories that matter when you are the person keeping
00:01:00.280 --> 00:01:02.899
the thing running at three in the morning. For
00:01:02.899 --> 00:01:05.340
the weekly story list and source links, check
00:01:05.340 --> 00:01:09.019
out oncallbrief.com. For past episodes and show
00:01:09.019 --> 00:01:12.739
notes, head over to shipitweekly.fm. This week,
00:01:13.019 --> 00:01:16.200
GitHub Actions goes multi-cloud, the Brotli
00:01:16.200 --> 00:01:20.799
bug lands on CISA's exploited list, a campaign
00:01:20.799 --> 00:01:24.599
called Gheist steals action secrets, and attackers
00:01:24.599 --> 00:01:28.680
are dropping AGENTS.md files into Downloads
00:01:28.680 --> 00:01:32.040
folders. Then we have a quick lightning round
00:01:32.040 --> 00:01:35.439
and a human closer about the 20 minutes you lose
00:01:35.439 --> 00:01:39.225
finding out who owns a service. Let's get into it.
00:01:43.926 --> 00:01:46.640
First up, GitHub Actions is spreading hosted
00:01:46.640 --> 00:01:50.780
runner jobs across more than one cloud. At GitHub
00:01:50.780 --> 00:01:54.319
Universe on Wednesday, GitHub said eligible Linux
00:01:54.319 --> 00:01:59.299
jobs may now run on AWS, in addition to Microsoft
00:01:59.299 --> 00:02:02.659
Azure. This covers the standard two-core and
00:02:02.659 --> 00:02:05.859
four-core Linux runners, and the larger runners
00:02:05.859 --> 00:02:09.039
of the same sizes without special configuration.
00:02:09.599 --> 00:02:12.860
Windows runners and private networking aren't
00:02:12.860 --> 00:02:16.889
part of it. On Free, Pro, and Team plans, eligible
00:02:16.889 --> 00:02:20.550
jobs move automatically. On Enterprise Cloud,
00:02:20.889 --> 00:02:24.629
it's opt-in under Hosted compute cloud providers
00:02:24.629 --> 00:02:28.090
in the enterprise action settings. Your labels
00:02:28.090 --> 00:02:31.909
and workflow files don't change. GitHub's reason
00:02:31.909 --> 00:02:36.189
is capacity and shorter queue times. The part
00:02:36.189 --> 00:02:39.590
that I'd care about is the network. Jobs running
00:02:39.590 --> 00:02:43.949
on AWS leave from the new outbound IP addresses.
00:02:44.349 --> 00:02:47.909
If anything your workflows talk to only allows
00:02:47.909 --> 00:02:50.949
Azure or Microsoft ranges, those connections
00:02:50.949 --> 00:02:54.650
can fail. GitHub specifically calls out Microsoft
00:02:54.650 --> 00:02:58.129
Entra Conditional Access policies here. So
00:02:58.129 --> 00:03:01.750
authentication can break too, not just a firewall
00:03:01.750 --> 00:03:05.490
rule. And because it's a rolling change, it won't
00:03:05.490 --> 00:03:08.610
break all at once. You could see the same job
00:03:08.610 --> 00:03:12.539
pass on one run and fail on the next, depending
00:03:12.539 --> 00:03:16.080
on which cloud it landed on. That's a miserable
00:03:16.080 --> 00:03:19.219
thing to debug if you don't know this change
00:03:19.219 --> 00:03:23.259
is happening. So go find your allowlist now.
00:03:23.580 --> 00:03:27.219
Database firewalls, registries, internal APIs,
00:03:27.699 --> 00:03:32.020
anything keyed to GitHub's IP ranges. GitHub publishes
00:03:32.020 --> 00:03:35.699
every runner range in the actions list from its
00:03:35.699 --> 00:03:43.150
meta API at api.github.com/meta. Sync from that, not
00:03:43.150 --> 00:03:46.409
from one cloud's ranges. If you've been treating
00:03:46.409 --> 00:03:50.349
"it comes from Azure" as a security control, this
00:03:50.349 --> 00:03:53.601
is the week that assumption stops holding.
00:03:58.302 --> 00:04:02.500
Next, the Brotli decoder bug is now being exploited.
00:04:02.860 --> 00:04:08.960
CVE-2026-111497 is an out-of-bounds write
00:04:08.960 --> 00:04:12.819
in Brotli, the compression format a lot of HTTP
00:04:12.819 --> 00:04:17.600
traffic uses. Node.js fixed its bundled copy
00:04:17.600 --> 00:04:21.259
in last week's security release, rated high.
00:04:21.560 --> 00:04:26.300
On Tuesday, CISA added it to its Known Exploited
00:04:26.300 --> 00:04:29.639
Vulnerabilities catalog with a November 10th
00:04:29.639 --> 00:04:34.079
deadline for federal agencies. And it isn't just
00:04:34.079 --> 00:04:38.209
Node. Google's Brotli library is affected from
00:04:38.209 --> 00:04:44.290
1.0.3 through 1.2.0, except the patched
00:04:44.290 --> 00:04:50.430
1.0.10 and 1.1.1 releases. So are the Python
00:04:50.430 --> 00:04:55.230
Brotli package, Bun, Envoy, Istio, and a popular
00:04:55.230 --> 00:04:59.250
Go port. And Microsoft shipped out-of-band
00:04:59.250 --> 00:05:02.889
.NET patches last weekend. Here's how it works,
00:05:03.009 --> 00:05:07.069
per Node's advisory. Brotli data comes in blocks
00:05:07.069 --> 00:05:10.410
and the decoder switches between them as it goes.
00:05:10.670 --> 00:05:13.990
When it hit an invalid block switch, it treated
00:05:13.990 --> 00:05:17.529
that as "I need more input" instead of "this data
00:05:17.529 --> 00:05:21.550
is broken." So the decoder kept appending bytes
00:05:21.550 --> 00:05:25.509
into an eight-byte internal buffer, writing right
Load full transcript
00:05:25.509 --> 00:05:29.350
past the end of it. That can mean a crash, leaked
00:05:29.350 --> 00:05:33.089
memory, or code execution. What makes this one
00:05:33.089 --> 00:05:36.680
sneaky is that you probably never call Brotli
00:05:36.680 --> 00:05:41.759
yourself. Some HTTP clients and frameworks decompress
00:05:41.759 --> 00:05:45.019
automatically once Brotli support is turned on.
00:05:45.240 --> 00:05:48.500
So the exposure could be an incoming request,
00:05:48.839 --> 00:05:53.079
an API response, or anything compressed you didn't
00:05:53.079 --> 00:05:59.540
create. Upgrade Node to 22.23.4, 24.22.1,
00:05:59.699 --> 00:06:05.680
or 26.12.1 or newer. Get Python Brotli to
00:06:05.680 --> 00:06:10.579
1.2.1. And if your .NET apps decompress Brotli,
00:06:10.740 --> 00:06:18.439
move to 8.0.33, 9.0.22, or 10.0.14. And
00:06:18.439 --> 00:06:22.300
check your Envoy and Istio versions because gateways
00:06:22.300 --> 00:06:26.300
and sidecars can decompress too. Then rebuild
00:06:26.300 --> 00:06:30.540
images and restart processes because a patched
00:06:30.540 --> 00:06:33.610
package on disk doesn't help a process running
00:06:33.610 --> 00:06:36.970
the old one. If you can't patch everything this
00:06:36.970 --> 00:06:40.689
week, turning off Brotli decompression or rejecting
00:06:40.689 --> 00:06:44.089
Brotli-encoded requests at the edge buys you
00:06:44.089 --> 00:06:46.792
time. It doesn't replace the patch.
00:06:51.493 --> 00:06:55.569
Third, a campaign called Gheist is using stolen GitHub
00:06:55.569 --> 00:06:58.930
tokens to walk off with your action secrets.
00:06:59.410 --> 00:07:02.649
Wiz published this on Sunday. They've tracked
00:07:02.649 --> 00:07:06.050
it since mid-September and recovered more than
00:07:06.050 --> 00:07:12.149
13,500 planted branches from over 850 compromised
00:07:12.149 --> 00:07:15.870
accounts. Here's the playbook. For every repo
00:07:15.870 --> 00:07:19.769
a stolen token can reach, the tool pushes a branch
00:07:19.769 --> 00:07:24.610
named ci-sync plus six random characters. On
00:07:24.610 --> 00:07:28.310
it is a workflow called sync-check that prints
00:07:28.310 --> 00:07:32.930
the repo secrets, base32-encoded, so GitHub's
00:07:32.930 --> 00:07:36.649
log masking doesn't catch them. The tool downloads
00:07:36.649 --> 00:07:39.829
the log through the API then deletes the run
00:07:39.829 --> 00:07:43.370
seconds later. Newer variants add environment
00:07:43.370 --> 00:07:46.829
matrices so environment-scoped secrets like
00:07:46.829 --> 00:07:50.050
production credentials come out too. In enterprise
00:07:50.050 --> 00:07:53.689
cases, Wiz saw it use the cloud credentials it
00:07:53.689 --> 00:07:57.389
found to go through AWS Secrets Manager. Now,
00:07:57.490 --> 00:08:00.689
GitHub already holds suspicious workflow runs
00:08:01.149 --> 00:08:04.430
in public repos until a collaborator approves
00:08:04.430 --> 00:08:08.170
them. Every public Gheist run Wiz checked
was
00:08:08.170 --> 00:08:12.110
stuck waiting. So that control worked. But the
00:08:12.110 --> 00:08:15.610
hold doesn't apply to private repos or GitHub
00:08:15.610 --> 00:08:19.129
Enterprise Server. There, the run completed and
00:08:19.129 --> 00:08:23.149
got deleted. And Wiz saw organizations' own workflows
00:08:23.149 --> 00:08:27.370
run on those attacker branches and assume AWS
00:08:27.370 --> 00:08:32.259
roles through OIDC. So... hunt first. Look for
00:08:32.259 --> 00:08:36.639
ci-sync branches whose only change is sync-check.yml,
00:08:36.639 --> 00:08:41.299
runs deleted right after they finish, and
00:08:41.299 --> 00:08:45.399
a user agent of gheist in your audit log. If you
00:08:45.399 --> 00:08:49.659
use OIDC, check your cloud logs for role assumptions
00:08:49.659 --> 00:08:54.019
from ci-sync branches. If you find any, rotate
00:08:54.019 --> 00:08:57.879
every secret those repos could reach. Deleting
00:08:57.879 --> 00:09:01.909
the branch doesn't undo the exposure. Then harden:
00:09:02.110 --> 00:09:05.529
require reviewers on production environments,
00:09:05.870 --> 00:09:11.029
scope OIDC trust to protected branches, and move
00:09:11.029 --> 00:09:13.955
off long-lived personal access tokens.
00:09:18.656 --> 00:09:22.460
Fourth, attackers have found a new place to put instructions
00:09:22.460 --> 00:09:26.419
for your AI coding agent, your Downloads folder.
00:09:26.879 --> 00:09:29.980
Forever Security published this last week after
00:09:29.980 --> 00:09:33.039
catching it on a real customer's machine. Their
00:09:33.039 --> 00:09:36.840
tool stopped a coding agent mid-session from
00:09:36.840 --> 00:09:39.919
installing function-color, a known malicious
00:09:39.919 --> 00:09:44.220
npm package. The instructions came from an AGENTS.md
00:09:44.220 --> 00:09:48.399
file sitting in Downloads. Here's why that
00:09:48.399 --> 00:09:52.139
works. Coding agents automatically read the instruction
00:09:52.139 --> 00:09:56.159
file in whatever folder you start them in. And
00:09:56.159 --> 00:09:59.779
Downloads is the one folder a website can write
00:09:59.779 --> 00:10:03.740
to on its own. Visit the wrong page and a file
00:10:03.740 --> 00:10:08.120
called AGENTS.md can quietly land there. Then
00:10:08.120 --> 00:10:11.860
the day you open an agent in Downloads, maybe
00:10:11.860 --> 00:10:15.019
to sort some files, it reads those instructions
00:10:15.019 --> 00:10:18.500
and follows them. In their testing, Codex ran
00:10:18.500 --> 00:10:23.440
the payload in more than 90% of attempts. And it
00:10:23.440 --> 00:10:26.960
wrote to Codex's machine-wide config file, so
00:10:26.960 --> 00:10:30.320
the damage doesn't stay in that one folder. With
00:10:30.320 --> 00:10:34.600
Claude Code, a CLAUDE.md file redirected its
00:10:34.600 --> 00:10:37.960
API traffic to a server the researchers controlled,
00:10:38.340 --> 00:10:42.320
which then fed back commands for it to run. That
00:10:42.320 --> 00:10:45.399
worked about half the time. The researchers say
00:10:45.399 --> 00:10:48.600
there's nothing here for OpenAI or Anthropic
00:10:48.600 --> 00:10:51.940
to patch, because the agents are doing what they
00:10:51.940 --> 00:10:55.820
were built to do. So do what both vendors recommend
00:10:55.820 --> 00:11:00.059
and start agents in your project directory, not
00:11:00.059 --> 00:11:04.159
your home folder. And definitely not in Downloads.
00:11:04.600 --> 00:11:08.159
Keep the approval prompts and the sandbox turned
00:11:08.159 --> 00:11:12.899
on. And if your team uses these tools, an instruction
00:11:12.899 --> 00:11:16.799
file is code now. Treat one you didn't write
00:11:16.799 --> 00:11:19.620
the way you'd treat a script you didn't write.
00:11:26.899 --> 00:11:30.779
Quick lightning round. First, a follow-up. In
00:11:30.779 --> 00:11:34.379
episode 72, we covered GitHub Actions workflow
00:11:34.379 --> 00:11:38.740
lockfiles going into public preview. As of Wednesday,
00:11:39.019 --> 00:11:42.100
they are generally available. Dependabot now
00:11:42.100 --> 00:11:45.200
updates the lockfile in the same pull request
00:11:45.200 --> 00:11:49.360
when it bumps an action. And admins can require
00:11:49.360 --> 00:11:54.259
lockfiles with a policy. Second, Railway disclosed
00:11:54.259 --> 00:11:58.909
a breach on October 21st. Someone got into internal
00:11:58.909 --> 00:12:03.029
systems and may have accessed customers' encrypted
00:12:03.029 --> 00:12:07.190
environment variables. And for some, Railway
00:12:07.190 --> 00:12:10.590
says the decrypted values were taken. If you
00:12:10.590 --> 00:12:14.190
run anything there, rotate every credential stored
00:12:14.190 --> 00:12:19.190
on Railway from before October 21st and review
00:12:19.190 --> 00:12:23.210
your GitHub audit logs and installed app access.
00:12:23.629 --> 00:12:28.879
Third, Tailscale shipped version 1.104.4 on
00:12:28.879 --> 00:12:32.360
Tuesday with fixes for four security bulletins.
00:12:32.580 --> 00:12:36.740
The worst hits nodes where Tailscale SSH allows
00:12:36.740 --> 00:12:40.399
forwarding. There, someone already allowed to
00:12:40.399 --> 00:12:44.799
SSH in could forward sockets as root, reaching
00:12:44.799 --> 00:12:48.980
things like Docker socket. Upgrade or set
00:12:48.980 --> 00:12:52.620
TS_SSH_DISABLE_FORWARDING
00:12:52.620 --> 00:12:57.590
to 1 and restart tailscaled until
00:12:57.590 --> 00:13:02.070
you can. And fourth, Datadog's US1 status page
00:13:02.070 --> 00:13:05.870
switched over to Datadog's own status page product
00:13:05.870 --> 00:13:10.070
on Thursday. But if you scrape the old page or
00:13:10.070 --> 00:13:13.470
its API in your own tooling, that can break.
00:13:13.730 --> 00:13:17.610
Point it at the old page's address or update
00:13:17.610 --> 00:13:19.655
it to read the new format.
00:13:28.156 --> 00:13:32.600
The human closer this week comes from Bill Duncan and a piece featured
00:13:32.600
--> 00:13:37.480
in SRE Weekly called "Who Owns This? Why SREs
00:13:37.480 --> 00:13:40.960
Want a Service Catalog." It opens at two in the
00:13:40.960 --> 00:13:44.460
morning in a scenario most of us have lived.
00:13:44.720 --> 00:13:48.379
An alert fires for a service you've never touched.
00:13:48.539 --> 00:13:52.340
The README is two years old. The Slack thread
00:13:52.340 --> 00:13:56.899
you find names a team that's since been reorganized.
00:13:57.039 --> 00:14:00.879
You page the rotation you think is right. And
00:14:00.879 --> 00:14:04.200
they tell you politely, it isn't theirs. But
00:14:04.200 --> 00:14:08.019
Sam used to work on it. Sam is on vacation. His
00:14:08.019 --> 00:14:12.059
line is, "20 minutes in, you haven't started diagnosing
00:14:12.059 --> 00:14:15.960
anything. You've been doing archaeology." He boils
00:14:15.960 --> 00:14:19.860
it down to five questions. From who owns this
00:14:19.860 --> 00:14:24.320
service to who's on call for it right now. Ownership
00:14:24.320 --> 00:14:27.600
written down as a person goes stale fastest.
00:14:28.039 --> 00:14:31.919
So... it should point at a team that still exists.
00:14:32.340 --> 00:14:36.480
CODEOWNERS tells you who reviews the code, not
00:14:36.480 --> 00:14:40.120
who gets woken up. And most teams can answer
00:14:40.120 --> 00:14:43.620
"what does this depend on" by reading the code.
00:14:43.919 --> 00:14:48.000
Almost nobody can answer "what depends on me" without
00:14:48.000 --> 00:14:52.059
asking around. His fix is a small catalog file
00:14:52.059 --> 00:14:56.399
in each service's repo, maintained by the team
00:14:56.399 --> 00:14:59.840
that owns it. And he's honest about the catch.
00:15:00.139 --> 00:15:04.000
The people who want it most, the SREs, aren't
00:15:04.000 --> 00:15:07.059
the ones who can keep it accurate. Now, look
00:15:07.059 --> 00:15:10.139
at this week's stories. Which of your services
00:15:10.139 --> 00:15:14.379
decompress Brotli? Which allowlists assume
00:15:14.379 --> 00:15:19.720
your CI runs on Azure? Which OIDC trust policies
00:15:19.720 --> 00:15:24.679
accept any branch? Which laptops have an agent
00:15:24.679 --> 00:15:28.259
that's been started in Downloads? None of those
00:15:28.259 --> 00:15:31.879
are hard questions. They are inventory questions,
00:15:32.259 --> 00:15:36.340
and every one of them is easier to answer if
00:15:36.340 --> 00:15:40.019
somebody wrote it down before this week. That's
00:15:40.019 --> 00:15:43.559
it for this week of Ship It Weekly. We covered
00:15:43.559 --> 00:15:48.600
GitHub Actions jobs moving to AWS, the Brotli
00:15:48.600 --> 00:15:53.200
decoder bug landing on CISA's exploited list,
00:15:53.519 --> 00:15:57.090
the Gheist campaign stealing Actions secrets,
00:15:57.409 --> 00:16:02.049
and AGENTS.md files hijacking coding agents
00:16:02.049 --> 00:16:06.129
from the Downloads folder. Plus, Actions lockfiles
00:16:06.129 --> 00:16:10.529
going GA, the Railway breach, Tailscale's
00:16:10.529 --> 00:16:15.429
SSH fixes, and Datadog's new status page. Follow
00:16:15.429 --> 00:16:17.990
or subscribe wherever you are watching or listening.
00:16:18.269 --> 00:16:21.289
You can find the weekly story list and source
00:16:21.289 --> 00:16:25.659
links at oncallbrief.com, and past episodes
00:16:25.659 --> 00:16:30.000
and show notes at shipitweekly.fm. I'm Brian
00:16:30.000 --> 00:16:32.299
Teller from Teller's Tech. Thanks for listening.
00:16:32.480 --> 00:16:36.500
And remember, write down who owns it before the
00:16:36.500 --> 00:16:37.240
pager asks.