النص 🔗
WEBVTT
00:00:00.000 --> 00:00:04.259
A memory bug in Java's Base64 decoder, nicknamed
00:00:04.259 --> 00:00:08.759
Java4Shell got patched Tuesday, and your CPU
00:00:08.759 --> 00:00:12.580
decides whether you are exposed. Microsoft's
00:00:12.580 --> 00:00:16.739
Agent Framework package on PyPI shipped a credential
00:00:16.739 --> 00:00:20.699
stealer that ran every time Python started. And
00:00:20.699 --> 00:00:23.980
attackers went from an old Confluence SOAP library
00:00:23.980 --> 00:00:29.519
to Microsoft 365 in about 12 hours. I'm Brian
00:00:29.519 --> 00:00:32.020
Teller from Teller's Tech, and this is Ship It
00:00:32.020 --> 00:00:32.505
Weekly.
00:00:49.304 --> 00:00:51.850
Welcome back to Ship It Weekly, the show
00:00:51.850 --> 00:00:56.570
about the DevOps, SRE, cloud, platform, and security
00:00:56.570 --> 00:00:59.070
stories that matter when you are the person keeping
00:00:59.070 --> 00:01:01.890
the thing running at three in the morning. For
00:01:01.890 --> 00:01:04.609
the weekly story list and source links, check
00:01:04.609 --> 00:01:08.409
out oncallbrief.com. For past episodes and show
00:01:08.409 --> 00:01:11.989
notes, head over to shipitweekly.fm. This week,
00:01:12.090 --> 00:01:16.930
the Java4Shell OpenJDK flaw, Microsoft's hijacked
00:01:16.930 --> 00:01:20.409
agent framework package, a Confluence bug under
00:01:20.409 --> 00:01:24.489
active attack, and the router change behind the
00:01:24.489 --> 00:01:28.150
FAA's ground stop. Then we have a quick lightning
00:01:28.150 --> 00:01:31.689
round, and a human closer about why your root
00:01:31.689 --> 00:01:35.489
cause analysis might not be working. Let's get
00:01:35.489 --> 00:01:35.943
into it.
00:01:40.642 --> 00:01:46.489
First up, Java4Shell, CVE-2026-114938,
00:01:46.930 --> 00:01:51.870
a memory corruption bug in the OpenJDK HotSpot
00:01:51.870 --> 00:01:56.390
JVM. Oracle and the OpenJDK project fixed it
00:01:56.390 --> 00:02:00.250
on Tuesday, November 17th. It affects Java 17
00:02:00.250 --> 00:02:04.290
and newer, including Corretto, Zulu, and other
00:02:04.290 --> 00:02:07.989
downstream builds. Calif, which found it, rates
00:02:07.989 --> 00:02:12.889
it 9.2 critical. Oracle says 8.1 because it's
00:02:12.889 --> 00:02:16.439
hard to exploit. The mechanism is narrow. To
00:02:16.439 --> 00:02:20.539
decode Base64 quickly, the JVM swaps the normal
00:02:20.539 --> 00:02:24.699
Java code for handwritten assembly that uses
00:02:24.699 --> 00:02:29.919
AVX-512 instructions. That assembly miscounts
00:02:29.919 --> 00:02:34.000
the output size when the input ends with an unexpected
00:02:34.000 --> 00:02:38.000
number of equal signs. The length math underflows,
00:02:38.259 --> 00:02:42.419
and the JVM writes past the end of a buffer on
00:02:42.419 --> 00:02:46.629
the heap. Calif says 64 letter A's followed
00:02:46.629 --> 00:02:50.389
by one equal sign is enough to trigger it on
00:02:50.389 --> 00:02:54.469
newer JDKs. The vulnerable path only runs on
00:02:54.469 --> 00:03:02.150
x86 CPUs with the AVX-512 VBMI instructions. Intel
00:03:02.150 --> 00:03:07.229
Xeons from Ice Lake on and AMD Zen 4 and newer.
00:03:07.569 --> 00:03:12.169
ARM64 isn't affected, so your Graviton nodes
00:03:12.169 --> 00:03:17.930
aren't either. On Linux, look for the AVX-512 VBMI
00:03:17.930 --> 00:03:23.590
flag in /proc/cpuinfo. VulnCheck says crashing
00:03:23.590 --> 00:03:28.050
a service is easy. Code execution is much harder.
00:03:28.229 --> 00:03:31.969
It needs heap grooming plus a separate memory
00:03:31.969 --> 00:03:36.110
leak. Calif says it got code execution in apps
00:03:36.110 --> 00:03:39.930
like Jira and Jenkins, and it's holding those
00:03:39.930 --> 00:03:43.870
exploits back. As of Wednesday, GreyNoise had
00:03:43.870 --> 00:03:47.370
reported exploit attempts, but no successful
00:03:47.370 --> 00:03:51.789
break-ins. So patch the JDK on your normal schedule,
00:03:52.009 --> 00:03:57.050
but find all of it. Base images, runtimes, bundled
00:03:57.050 --> 00:04:00.990
in vendor appliances, that old build agent. If
00:04:00.990 --> 00:04:04.180
a host can't be patched this week, Wiz says
00:04:04.180 --> 00:04:09.560
a JVM flag that turns off the Base64 intrinsics
00:04:09.560 --> 00:04:13.479
closes the vulnerable path, at some cost in performance.
00:04:14.060 --> 00:04:17.560
The bug is in Java, but the inventory problem
00:04:17.560 --> 00:04:18.830
is yours.
00:04:23.679 --> 00:04:26.399
Next, Microsoft's Agent Framework package
00:04:26.399 --> 00:04:31.199
on PyPI shipped a credential stealer. On Tuesday
00:04:31.199 --> 00:04:36.389
evening, UTC, two malicious versions, 1.26.1
00:04:36.389 --> 00:04:42.889
and 1.26.2 went up 33 minutes apart. Both were
00:04:42.889 --> 00:04:46.790
off PyPI within an hour of the first one. The
00:04:46.790 --> 00:04:50.990
package gets about 190,000 downloads a month,
00:04:51.149 --> 00:04:56.949
and the last clean version is 1.26.0. How the
00:04:56.949 --> 00:05:00.370
attacker got publishing access hasn't been disclosed.
00:05:00.529 --> 00:05:03.850
What makes this one worth understanding is how
00:05:03.850 --> 00:05:07.279
it ran. Neither version needed you to import
00:05:07.279 --> 00:05:12.839
anything. Each one dropped a .pth file into site
00:05:12.839 --> 00:05:16.879
packages. Python reads those files every time
00:05:16.879 --> 00:05:20.740
the interpreter starts. And it runs any line
00:05:20.740 --> 00:05:24.319
in them that begins with the word import. So
00:05:24.319 --> 00:05:28.240
the stealer ran with your tests, your app, any
00:05:28.240 --> 00:05:32.480
script in that environment, even pip. It collected
00:05:32.480 --> 00:05:35.449
environment variables with names like token,
00:05:35.649 --> 00:05:39.730
secret, and key, plus credential files for the
00:05:39.730 --> 00:05:45.410
Azure CLI, npm, Docker, and Git. Then it asked
00:05:45.410 --> 00:05:49.709
Azure for fresh managed identity
Load full transcript
tokens for Key
00:05:49.709 --> 00:05:54.269
Vault, Storage, and Resource Manager, plus Azure
00:05:54.269 --> 00:05:58.529
DevOps OIDC tokens. The second version added
00:05:58.529 --> 00:06:02.949
GitHub Actions OIDC tokens in workflows allowed
00:06:02.949 --> 00:06:06.709
to request one. All of it went out in a single
00:06:06.709 --> 00:06:11.250
post to a webhook.site address. Those fresh
00:06:11.250 --> 00:06:14.810
tokens change the cleanup. Rotating your stored
00:06:14.810 --> 00:06:18.910
secrets doesn't touch a token the attacker requested
00:06:18.910 --> 00:06:24.209
as you. Wiz says those tokens last up to 24 hours.
00:06:24.490 --> 00:06:28.250
So check what your managed identities and federated
00:06:28.250 --> 00:06:31.980
credentials actually did. Search your lock files,
00:06:32.360 --> 00:06:39.980
CI logs, and images for 1.26.1 and 1.26.2
00:06:39.980 --> 00:06:47.240
and pin 1.26.0. Look for webhook.site in your
00:06:47.240 --> 00:06:51.899
proxy and DNS logs. If it ran, rotate everything
00:06:51.899 --> 00:06:55.540
that environment could reach and review sign-ins
00:06:55.540 --> 00:06:58.565
for every identity it could use.
00:07:03.414 --> 00:07:07.220
Third, attackers are exploiting Confluence Data Center
00:07:07.220 --> 00:07:15.160
through CVE-2026-90436, a 9.8 critical bug in
00:07:15.160 --> 00:07:20.199
Apache Axis 1. Axis is the old SOAP engine behind
00:07:20.199 --> 00:07:25.399
Confluence's remote API. The flaw is unsafe reflection.
00:07:25.399 --> 00:07:29.759
A request gets to choose which Java class is
00:07:29.759 --> 00:07:34.600
loaded, and no login is needed. Axis 1 is end
00:07:34.600 --> 00:07:38.279
of life, and Apache isn't planning a fixed release.
00:07:38.939 --> 00:07:43.060
Rapid7 saw it used against a customer last Friday,
00:07:43.259 --> 00:07:46.860
November 13th. Unauthenticated requests hit the
00:07:46.860 --> 00:07:51.000
SOAP endpoint, and within seconds, the Confluence
00:07:51.000 --> 00:07:55.220
Java process was starting bash and curl. About
00:07:55.220 --> 00:07:58.560
three minutes in, the attacker logged into MySQL
00:07:58.560 --> 00:08:02.480
with Confluence's own database credentials, read
00:08:02.480 --> 00:08:05.779
pages, and searched attachments for credentials,
00:08:06.100 --> 00:08:10.639
tokens, VPN configs, and private keys. About
00:08:10.639 --> 00:08:13.540
four hours in, they were querying a production
00:08:13.540 --> 00:08:17.120
data warehouse with a service account. About
00:08:17.120 --> 00:08:22.160
12 hours in, they signed into Microsoft 365 as
00:08:22.160 --> 00:08:25.600
an integration service account using legacy authentication.
00:08:26.180 --> 00:08:29.360
I think the interesting part is the setting that
00:08:29.360 --> 00:08:32.440
makes you vulnerable. The remote API has been
00:08:32.440 --> 00:08:36.659
off by default for new installs since Confluence
00:08:36.659 --> 00:08:42.019
3.0. So the exposed servers are the ones where
00:08:42.019 --> 00:08:45.139
someone switched it on for an integration, maybe
00:08:45.139 --> 00:08:49.039
a year ago, and nobody switched it off. If you
00:08:49.039 --> 00:08:52.820
run Confluence Data Center, go to General Configuration,
00:08:52.820 --> 00:08:57.779
then Further Configuration, and check Remote API.
00:08:58.139 --> 00:09:03.980
Atlassian fixed it in 9.2.28, and 10.x doesn't
00:09:03.980 --> 00:09:07.220
have the feature. Turning the setting off works
00:09:07.220 --> 00:09:10.600
without a restart, but anything that integrates
00:09:10.600 --> 00:09:14.259
through it will break. Atlassian Cloud isn't
00:09:14.259 --> 00:09:18.740
affected. N-able also says attackers on N-central
00:09:18.740 --> 00:09:22.980
strongly correlate with this flaw, and it's fixed
00:09:22.980 --> 00:09:29.149
in 2026.4 Hotfix 3. A wiki is only as safe as
00:09:29.149 --> 00:09:31.994
the credentials people have pasted into it.
00:09:36.693 --> 00:09:40.799
Fourth, the FAA's nationwide ground stop last Friday.
00:09:40.980 --> 00:09:45.220
At about 3:30 p.m. Eastern on November 13th,
00:09:45.240 --> 00:09:49.159
the FAA halted flights across the country. Traffic
00:09:49.159 --> 00:09:53.440
started moving again at 8:15 that evening. FedScoop
00:09:53.440 --> 00:09:56.820
says that's only the second time since September
00:09:56.820 --> 00:10:00.389
11th. Operators canceled about 12,000 flights
00:10:00.389 --> 00:10:03.190
that day, not all of them because of the stop.
00:10:03.450 --> 00:10:07.269
The FAA says there's no sign of a cyber attack.
00:10:07.669 --> 00:10:11.350
The preliminary story is a network change. The
00:10:11.350 --> 00:10:15.669
FAA's early findings point to two sites, one
00:10:15.669 --> 00:10:18.830
of them a radio site in El Dorado, Arkansas,
00:10:19.169 --> 00:10:23.549
where its network provider L3Harris was installing
00:10:23.549 --> 00:10:27.789
a new router as part of a move from copper to
00:10:27.789 --> 00:10:31.070
fiber. Officials say that router flooded the
00:10:31.070 --> 00:10:33.629
network with about a thousand connection attempts
00:10:33.629 --> 00:10:38.470
every four to five seconds. And the legacy TDM
00:10:38.470 --> 00:10:42.750
equipment behind it went unstable. It hit 17
00:10:42.750 --> 00:10:47.129
of the 22 centers that handle high altitude traffic.
00:10:47.610 --> 00:10:51.049
What I like about this story is what came next.
00:10:51.470 --> 00:10:55.850
FAA administrator Brian Bedford said they spent
00:10:55.850 --> 00:10:59.549
about 45 minutes trying to find the cause. That
00:10:59.549 --> 00:11:03.090
maintenance is logged by hand, so they were going
00:11:03.090 --> 00:11:05.990
through sheets of paper to find out what had
00:11:05.990 --> 00:11:10.690
changed. L3Harris blocked both sites, and the
00:11:10.690 --> 00:11:14.649
network settled. The FAA has since disabled more
00:11:14.649 --> 00:11:17.950
than 60 other Juniper routers as a precaution,
00:11:18.070 --> 00:11:21.730
and says it will rethink when this kind of work
00:11:21.730 --> 00:11:26.169
gets done. This one happened midday. The questions
00:11:26.169 --> 00:11:29.690
carry over. When something breaks, can you list
00:11:29.690 --> 00:11:32.830
every change from the last hour? Can you search
00:11:32.830 --> 00:11:37.090
that list? Or is it someone's notes? And does
00:11:37.090 --> 00:11:40.549
your maintenance window line up with your quietest
00:11:40.549 --> 00:11:44.110
traffic? Or with when the contractor is available?
00:11:51.570 --> 00:11:56.450
Quick lightning round. First, wfx-loot. Starting
00:11:56.450 --> 00:12:00.649
November 10th, attackers used stolen GitHub credentials
00:12:00.649 --> 00:12:09.610
from 4,837 accounts to probe 61,764 repositories
00:12:09.610 --> 00:12:13.730
for write access, according to Aikido. Where they
00:12:13.730 --> 00:12:16.909
had it, they pushed a credential stealer into
00:12:16.909 --> 00:12:21.789
a workflow or an npm install hook, then reverted
00:12:21.789 --> 00:12:24.990
it and deleted the run within about a minute.
00:12:25.470 --> 00:12:31.690
Look for a branch called wfx-docs or a gap in
00:12:31.690 --> 00:12:35.289
your workflow run numbers. And rotating secrets
00:12:35.289 --> 00:12:39.490
isn't enough. Where it could, it also rewrote
00:12:39.490 --> 00:12:43.889
AWS role trust policies to accept any account
00:12:43.889 --> 00:12:47.809
and added Azure federated credentials. Second,
00:12:48.029 --> 00:12:53.169
PHP is back. A month after PHP2Shell, PHP
00:12:53.169 --> 00:12:56.129
shipped another emergency release on Thursday,
00:12:56.350 --> 00:13:01.509
with seven critical fixes all rated 9.2. At
00:13:01.509 --> 00:13:04.909
least two of them get around October's PHP2Shell
00:13:04.909 --> 00:13:09.169
patches, so being patched last month isn't
00:13:09.169 --> 00:13:14.200
enough. One, nicknamed phpAfterShell, was found
00:13:14.200 --> 00:13:18.019
by depthfirst with help from its AI models. An
00:13:18.019 --> 00:13:21.360
interrupted session write can leave a file that's
00:13:21.360 --> 00:13:25.399
half old, half new, and PHP trusts it anyway.
00:13:25.799 --> 00:13:33.179
Update to 8.5.13, 8.4.28, 8.3.37, or 8.2.36.
00:13:33.179 --> 00:13:37.620
If you can't yet, turn off session upload
00:13:37.620 --> 00:13:41.659
progress. And if you are still on 8.1 or older,
00:13:41.779 --> 00:13:45.500
there's no official fix. And third, Kubernetes
00:13:45.500 --> 00:13:50.100
now has a date for removing kube-proxy's IPVS
00:13:50.100 --> 00:13:54.879
mode: version 1.40. Back in August, we talked
00:13:54.879 --> 00:13:58.580
about it starting to retire. On 1.36 or later,
00:13:58.779 --> 00:14:04.299
with a 5.13 or newer kernel, SIG Network recommends
00:14:04.299 --> 00:14:09.039
nftables, which is a config change and a kube-proxy
00:14:09.039 --> 00:14:10.339
restart.
00:14:18.838 --> 00:14:20.740
The human closer this week comes
00:14:20.740 --> 00:14:24.299
from Oakley Hall in a piece featured in SRE Weekly
00:14:24.299 --> 00:14:29.299
called Why Your RCAs Aren't Working. His point
00:14:29.299 --> 00:14:32.740
is that you can write a lot of root cause analysis
00:14:32.740 --> 00:14:35.779
documents without getting better at anything.
00:14:36.120 --> 00:14:39.840
He gives four reasons, and two of them fit this
00:14:39.840 --> 00:14:43.460
week really well. The first is that we collect
00:14:43.460 --> 00:14:47.220
documents, not data. There's no table that you
00:14:47.220 --> 00:14:51.000
can query across incidents, by change type, by
00:14:51.000 --> 00:14:55.240
platform, or by the hour a change went out. So
00:14:55.240 --> 00:14:58.940
spotting a pattern depends on whoever has read
00:14:58.940 --> 00:15:02.240
most of them and how much they remember. The
00:15:02.240 --> 00:15:05.399
second is that systematic problems get assigned
00:15:05.399 --> 00:15:08.879
to whichever team happened to hit them. That
00:15:08.879 --> 00:15:13.179
team can fix its own service. It usually can't
00:15:13.179 --> 00:15:16.379
change a shared platform, an unsafe default,
00:15:16.620 --> 00:15:20.659
or a dependency everybody uses. So it builds
00:15:20.659 --> 00:15:24.419
a local workaround. The RCA gets closed and the
00:15:24.419 --> 00:15:28.440
next team finds the same problem. This week fits
00:15:28.440 --> 00:15:33.100
both. The FAA spent 45 minutes finding a router
00:15:33.100 --> 00:15:36.360
change, partly because the maintenance record
00:15:36.360 --> 00:15:40.399
was on paper. That's documents instead of data,
00:15:40.519 --> 00:15:44.559
about as literal as it gets. And Java4Shell
00:15:44.559 --> 00:15:48.559
isn't a ticket for every app team. It's work
00:15:48.559 --> 00:15:52.860
for whoever owns the base images and the JDK
00:15:52.860 --> 00:15:55.870
fleet. Same with an end-of-life SOAP library
00:15:55.870 --> 00:16:00.850
inside Confluence, or a .pth file that runs before
00:16:00.850 --> 00:16:04.789
your code does. Neither belongs to the team that
00:16:04.789 --> 00:16:08.830
noticed it first. His fix is a formal way to
00:16:08.830 --> 00:16:12.669
escalate those actions into funded platform work,
00:16:12.889 --> 00:16:16.350
with an owner who has the authority to finish
00:16:16.350 --> 00:16:19.629
them. Otherwise, you get a name in the document
00:16:19.629 --> 00:16:24.100
and no path to a fix. That's it for this week
00:16:24.100 --> 00:16:26.879
of Ship It Weekly. Follow or subscribe wherever
00:16:26.879 --> 00:16:29.700
you are watching or listening. You can find the
00:16:29.700 --> 00:16:32.960
weekly story list and source links at oncallbrief.com
00:16:32.960 --> 00:16:37.759
and past episodes and show notes at shipitweekly.fm.
00:16:37.759 --> 00:16:41.299
Quick thing before I let you go. Today's
00:16:41.299 --> 00:16:45.120
November 20th, and Ship It Weekly is officially
00:16:45.120 --> 00:16:49.059
one year old. Episode one came out a year ago
00:16:49.059 --> 00:16:54.120
today, and this episode is episode 86. I started
00:16:54.120 --> 00:16:56.820
this because keeping up with DevOps is hard,
00:16:56.940 --> 00:17:00.320
and I wanted something practitioners could actually
00:17:00.320 --> 00:17:04.019
use. In year one, this show won a Davey Award
00:17:04.019 --> 00:17:09.400
gold, a dotCOMM gold, and a Viddy. And we're a DevOps
00:17:09.400 --> 00:17:13.900
Dozen finalist in two categories. The show for
00:17:13.900 --> 00:17:18.319
Best DevOps Community Program, and me for Enterprise
00:17:18.319 --> 00:17:21.900
DevOps Champion of the Year. If this show's helped
00:17:21.900 --> 00:17:24.880
you this year, I'd really appreciate your vote.
00:17:25.079 --> 00:17:28.140
It's open through December 31st, and the link's
00:17:28.140 --> 00:17:31.240
in the show notes. And thank you for listening,
00:17:31.460 --> 00:17:34.140
for sharing episodes, and for sending stories
00:17:34.140 --> 00:17:38.019
my way. None of this happens without you. Year
00:17:38.019 --> 00:17:41.460
two means more conversations, more where the
00:17:41.460 --> 00:17:44.740
industry's headed, and more community. In this
00:17:44.740 --> 00:17:48.559
episode, we covered the Java4Shell OpenJDK
00:17:48.559 --> 00:17:52.200
flaw, Microsoft's hijacked agent framework package,
00:17:52.460 --> 00:17:57.059
the Confluence Axis bug under attack, and the
00:17:57.059 --> 00:18:00.960
router behind the FAA's ground stop. I'm Brian
00:18:00.960 --> 00:18:03.420
Teller from Teller's Tech. Thanks for listening.
00:18:03.579 --> 00:18:07.400
And remember, a corrective action needs an owner
00:18:07.400 --> 00:18:09.480
who can actually finish it.