บทถอดความฉบับเต็มของ Rogue AI or Just Sloppy Ops?

summary. 34 00:01:39,762 --> 00:01:40,982 It's a little bit more of a narrative. 35 00:01:40,982 --> 00:01:41,822 It's more entertaining. 36 00:01:42,202 --> 00:01:44,492 you can just sit back and enjoy some of these articles. 37 00:01:44,512 --> 00:01:47,510 I am going to cut some out to keep it short because I've got several 38 00:01:47,510 --> 00:01:50,940 articles to read to you hopefully this will be useful and insightful. 39 00:01:52,458 --> 00:01:54,938 Have you been adding text-to-speech in your apps? 40 00:01:55,108 --> 00:01:58,228 One of my late-night projects this year was creating a personal 41 00:01:58,228 --> 00:02:02,268 podcast feed so I could listen to the changelog of my favorite open 42 00:02:02,268 --> 00:02:03,948 source projects while I do my chores. 43 00:02:04,348 --> 00:02:06,568 I needed a text-to-speech API to do that. 44 00:02:06,978 --> 00:02:10,318 I also heard from a colleague this week who built an app for his kids to 45 00:02:10,328 --> 00:02:14,598 turn their e-books into audiobooks, and he's using the Speechify AI developer 46 00:02:14,598 --> 00:02:17,118 platform to do that text-to-speech 47 00:02:17,118 --> 00:02:21,108 Speechify AI is a developer platform for real-time text-to-speech, and 48 00:02:21,108 --> 00:02:22,628 they make their own voice models. 49 00:02:22,858 --> 00:02:27,858 Their Simba 3.2 model ranked number one on the Artificial Analysis Benchmark 50 00:02:28,088 --> 00:02:29,678 for conversational voice quality. 51 00:02:29,928 --> 00:02:34,338 They've got a free plan, streaming and REST APIs, TypeScript and Python 52 00:02:34,368 --> 00:02:39,238 SDKs, and if you wanna get fancy, you can use their SSML metadata support 53 00:02:39,248 --> 00:02:41,308 to add emotion into the speech output 54 00:02:42,223 --> 00:02:46,053 Whether you're shipping a weekend project or production voice agents at work, 55 00:02:46,073 --> 00:02:48,683 you can start free at speechify.ai. 56 00:02:49,003 --> 00:02:50,133 The link is in the show notes. 57 00:02:50,603 --> 00:02:53,083 Thanks so much to Speechify AI for sponsoring this episode. 58 00:02:53,499 --> 00:02:58,019 All right, the first article up is from Marius Horatau, hopefully 59 00:02:58,019 --> 00:02:58,919 I pronounced that correctly. 60 00:02:59,229 --> 00:03:03,009 Uh, Marius is a senior security engineer at Amazon and been doing this over 10 61 00:03:03,019 --> 00:03:07,556 years, they wrote an article called "The Hugging Face Incident is Not an AI Story." 62 00:03:08,886 --> 00:03:12,696 OpenAI's technical report on the Hugging Face incident read like a thriller. 63 00:03:13,026 --> 00:03:16,766 Agents in a sandbox invent a covert communication channel, use it to 64 00:03:16,766 --> 00:03:20,506 coordinate, find a zero-day in a shared service, break onto the 65 00:03:20,506 --> 00:03:24,376 internet, chain credentials across four organizations, and end up with root on 66 00:03:24,396 --> 00:03:26,086 production

nodes at another company. 67 00:03:26,546 --> 00:03:29,036 The reactions are exactly what you'd expect, with some people 68 00:03:29,036 --> 00:03:32,116 going as far as calling this the birth of agent civilizations. 69 00:03:32,646 --> 00:03:36,666 For any security engineer, though, the apocalyptic story reads much more boring 70 00:03:36,926 --> 00:03:40,996 as a major security operational failure rather than an AI capability result. 71 00:03:41,436 --> 00:03:45,016 The sandbox where the model evaluation occurred was not built or designed for 72 00:03:45,016 --> 00:03:46,716 an environment where hostile code runs. 73 00:03:47,216 --> 00:03:50,116 In this environment, they had agents that were isolated, but there was 74 00:03:50,116 --> 00:03:52,996 a shared component with network access that every agent could reach. 75 00:03:53,416 --> 00:03:57,586 This is just bad design and a classic security isolation/segregation 76 00:03:57,596 --> 00:03:59,746 failure rather than some crazy hack. 77 00:04:00,343 --> 00:04:04,283 In security, we've tripped over this problem of sandboxes for 30 years, if 78 00:04:04,283 --> 00:04:06,503 you think about it, going back 30 years. 79 00:04:06,503 --> 00:04:09,093 We've had all sorts of sandboxing technology that I've been talking about 80 00:04:09,093 --> 00:04:13,663 for the last decade as Docker is the prominent one example today, but we've 81 00:04:13,663 --> 00:04:17,763 had so many other attempts at various levels of sandboxing across many OSs. 82 00:04:18,383 --> 00:04:21,323 But, the author goes on to say, In security, we've tripped over this 83 00:04:21,333 --> 00:04:25,863 problem of sandboxes for over 30 years, which is why any modern sandbox 84 00:04:25,863 --> 00:04:30,043 environment like browsers operate with only assumption that the sandbox 85 00:04:30,043 --> 00:04:31,323 will eventually get compromised. 86 00:04:31,633 --> 00:04:35,353 So we build defense in depth and monitor mechanisms to limit the blast 87 00:04:35,353 --> 00:04:36,943 radius when that eventually happens. 88 00:04:37,283 --> 00:04:39,823 OpenAI did not operate on this fundamental assumption. 89 00:04:39,853 --> 00:04:43,783 They took a frontier model trained to find and exploit vulnerabilities, put it 90 00:04:43,783 --> 00:04:47,843 in an insecure environment they believed was isolated, told it to be persistent, 91 00:04:48,133 --> 00:04:51,473 then acted surprised when the model did exactly what you'd expect it to do. 92 00:04:51,743 --> 00:04:54,303 It used the gaps in the environment to achieve its goals. 93 00:04:54,613 --> 00:04:57,503 In this blog post, I'll analyze the incident from a security engineering 94 00:04:57,503 --> 00:05:01,739 perspective to see what lessons we can learn from it." In May, the agent 95 00:05:01,739 --> 00:05:05,749 was given a task it could not complete because the task referenced Google Drive 96 00:05:05,749 --> 00:05:07,709 links and the agent had no internet.

97 00:05:08,549 --> 00:05:11,939 OpenAI notes it did not intend to create this impossible task. 98 00:05:12,369 --> 00:05:15,149 The agent, trained to be persistent, did not give up. 99 00:05:15,249 --> 00:05:17,809 Basically, it's just running in a forever loop, and this is one of my 100 00:05:17,809 --> 00:05:22,429 points is these things were running across many servers in a data center 101 00:05:22,429 --> 00:05:27,169 in forever loops, and they were given a task that they were to do until 102 00:05:27,169 --> 00:05:29,989 they were completed, and if they can't complete it, they gotta keep going. 103 00:05:29,989 --> 00:05:30,899 They gotta keep trying. 104 00:05:31,279 --> 00:05:32,459 And they did this for months. 105 00:05:32,819 --> 00:05:37,189 And in the real world, this would've been, I'm guessing, millions and millions 106 00:05:37,189 --> 00:05:41,379 of dollars of inference on, I think what-- At some point, we estimated, 107 00:05:41,379 --> 00:05:44,729 I looked at some estimates, billions of dollars worth of infrastructure if 108 00:05:44,739 --> 00:05:49,219 you consider how many GPUs and servers possibly were involved in this situation. 109 00:05:49,559 --> 00:05:54,149 So a significant portion of GPU infrastructure in a data center, along 110 00:05:54,159 --> 00:05:58,719 with forever-running unchecked agents with very little to no guidance or monitoring. 111 00:05:59,429 --> 00:06:00,329 All right, back to the article. 112 00:06:01,209 --> 00:06:03,809 OpenAI did not intend to create this impossible task. 113 00:06:04,149 --> 00:06:06,529 The agent tried to be persistent, did not give up. 114 00:06:06,559 --> 00:06:10,919 It attacked the only thing it could reach, attempted an SSRF against a shared 115 00:06:10,919 --> 00:06:14,419 component in the infrastructure called Artifactory, which a lot of us in DevOps 116 00:06:14,419 --> 00:06:20,239 are very familiar with, JFrog Artifactory, which is a, a proxy and cache for package 117 00:06:20,239 --> 00:06:21,819 management as well as a file server. 118 00:06:21,819 --> 00:06:26,849 It's basically got dozens of different NPM and Docker container and file 119 00:06:26,849 --> 00:06:28,689 servers, just all sorts of stuff in it. 120 00:06:29,109 --> 00:06:32,469 And it-- to me, it's like a Swiss Army knife for package management, which 121 00:06:32,469 --> 00:06:36,639 also means it's a Swiss Army m- knife for hackers because it's got a ton of 122 00:06:36,639 --> 00:06:40,799 surface area to protect, and it's, just running on one system as far as I know. 123 00:06:40,799 --> 00:06:42,979 And it's one server, not hardened. 124 00:06:43,519 --> 00:06:43,829 All right. 125 00:06:44,779 --> 00:06:47,669 And then Artifactory failed and incidentally wrote a file there. 126 00:06:48,039 --> 00:06:52,119 So days later, another blocked agent had the idea of leaving a note in Artifactory 127 00:06:52,129 --> 00:06:54,029 in case some other agent could help. 128 00:06:54,669 --> 00:06:59,849 "Agent seeks," filename, "upload if found." Other agents found the notes. 129 00:07:00,169 --> 00:07:03,939 By the end of May, there was an emergent message board, and agents 130 00:07:03,939 --> 00:07:06,909 were using it to share what they had learned, including how to get out. 131 00:07:07,229 --> 00:07:10,019 Notably, the message board itself required no exploit at all. 132 00:07:10,129 --> 00:07:14,049 OpenAI had given the agents shared Artifactory credentials so they could 133 00:07:14,059 --> 00:07:18,459 install packages and shared write access to a shared store, which is a message 134 00:07:18,459 --> 00:07:20,159 board whether you meant it to be or not. 135 00:07:20,489 --> 00:07:24,509 And that's the thing, is these things can only do one thing, really. 136 00:07:24,979 --> 00:07:26,099 They can emit text. 137 00:07:26,139 --> 00:07:29,989 They take text in, they emit text out, and that text may include commands that 138 00:07:29,989 --> 00:07:33,439 are grabbed by tools, and when they're in a certain format, they, the tool 139 00:07:33,439 --> 00:07:37,939 recognizes the format, extracts the text, but it's still just text, sends it on to 140 00:07:37,939 --> 00:07:41,499 the tool, and whatever the tool spits out is text that's fed back into the model. 141 00:07:41,789 --> 00:07:44,269 So in these cases, the model is doing exactly the same as 142 00:07:44,269 --> 00:07:45,209 the model on our machines. 143 00:07:45,529 --> 00:07:49,019 It was submitting, text that had commands that allowed it to get access 144 00:07:49,019 --> 00:07:51,689 to the network, whether it's curl or some other utility, Python, whatever 145 00:07:51,689 --> 00:07:56,489 it's writing, accessed the Artifactory, figured out what availability was there. 146 00:07:56,489 --> 00:07:58,729 By the way, the, it didn't, it was scanning ports, doing all 147 00:07:58,729 --> 00:08:01,119 sorts of things, and never was prevented from doing any of that. 148 00:08:01,679 --> 00:08:04,129 And eventually figured out WebDAV was open. 149 00:08:04,139 --> 00:08:07,029 There was other mechanisms for it to load packages up, and 150 00:08:07,519 --> 00:08:08,669 it was able to write things. 151 00:08:08,679 --> 00:08:13,279 And then later it, in a different, it, when it lost context or another one that 152 00:08:13,279 --> 00:08:17,789 didn't have the context, would find these files and read them and then respond to 153 00:08:17,789 --> 00:08:19,909 them just like they would any other text. 154 00:08:20,539 --> 00:08:21,739 That's not a civilization. 155 00:08:21,739 --> 00:08:25,139 That is it doing the math formula, doing what it's always done, which 156 00:08:25,139 --> 00:08:28,219 is responds to the text with the most likely words that it thinks 157 00:08:28,219 --> 00:08:30,709 need to complete the statement and finish the task that it's been given. 158 00:08:31,229 --> 00:08:32,969 so it's just, it's one of

those things where when you break it 159 00:08:32,969 --> 00:08:33,919 all down like that, it's boring. 160 00:08:34,939 --> 00:08:35,259 okay. 161 00:08:35,689 --> 00:08:40,039 By June 26th, agents had found a real vulnerability, a legacy token refresh 162 00:08:40,239 --> 00:08:43,419 endpoint that would take a forged administrative token with an invalid 163 00:08:43,419 --> 00:08:45,909 signature and hand back a valid one. 164 00:08:46,269 --> 00:08:47,869 That gave them Artifactory admin. 165 00:08:48,599 --> 00:08:50,979 That created a persistent user and installed Groovy 166 00:08:50,979 --> 00:08:52,499 plugins for command execution. 167 00:08:53,099 --> 00:08:54,279 So Groovy being Java. 168 00:08:54,569 --> 00:08:58,319 by the way, this was running in a container on the host, which they 169 00:08:58,319 --> 00:08:59,259 figured out how to get through. 170 00:08:59,649 --> 00:09:02,729 on June 27th, a day later, a monitoring tool alerted on port 171 00:09:02,729 --> 00:09:04,639 scanning inside the eval environment. 172 00:09:05,019 --> 00:09:10,339 Responders investigated, correctly traced it to the ExploitGym evaluation using 173 00:09:10,579 --> 00:09:15,039 Artifactory as a message board and a network pivot, and the on-call advised 174 00:09:15,039 --> 00:09:16,799 that stopping the run was not required. 175 00:09:17,379 --> 00:09:23,749 So it had already breached into the proxy that was there all along with 176 00:09:23,749 --> 00:09:27,849 its own internet access because it needed to be able to download packages. 177 00:09:28,409 --> 00:09:31,319 The agents were never supposed to access the internet, but they had access to a 178 00:09:31,329 --> 00:09:35,909 system that was vulnerable, not hardened, and they had unlimited access to, and 179 00:09:35,909 --> 00:09:40,299 even when it was breached, they weren't shut down or stopped, which is another big 180 00:09:40,309 --> 00:09:45,559 telltale sign of weak ops because if your agents are scanning ports on a system like 181 00:09:45,559 --> 00:09:50,019 that or if they actually have admin or if the admin in the container jumps to the 182 00:09:50,019 --> 00:09:55,929 host, like all these things are typical production automatic shutdown, settings. 183 00:09:55,969 --> 00:09:59,159 if you've ever run Falco in a Kubernetes environment, like Falco would have 184 00:09:59,159 --> 00:10:03,319 detected these things and immediately alerted or shut down systems or stopped 185 00:10:03,329 --> 00:10:04,519 containers as soon as that happened. 186 00:10:04,519 --> 00:10:05,669 But none of this was implemented. 187 00:10:06,559 --> 00:10:10,889 and by the way, ExploitGym is the, the mechanism by which the agents were 188 00:10:10,889 --> 00:10:12,149 told to solve some of these problems. 189 00:10:12,589 --> 00:10:17,209 On July 4th, the agent's own traffic knocked Artifactory over. 190 00:10:17,809 --> 00:10:20,629 The, that outage is what finally triggered a real investigation 191 00:10:20,629 --> 00:10:23,999 which found the unauthorized admin user, the rogue repositories, 192 00:10:24,009 --> 00:10:25,779 and the command-capable plugins. 193 00:10:26,229 --> 00:10:29,789 OpenAI took the service offline, revoked credentials, blocked the vulnerable 194 00:10:29,789 --> 00:10:32,479 paths, and then removed the shared credential from the new workloads. 195 00:10:33,019 --> 00:10:36,389 The report says plainly that the significance of the message board, quote, 196 00:10:36,399 --> 00:10:40,899 "Was not apparent to leaders responsible for incident detection and response at the 197 00:10:40,909 --> 00:10:44,419 time," which means they didn't know that there was new packages being uploaded. 198 00:10:44,869 --> 00:10:47,509 And why was a new package even ever able to be uploaded? 199 00:10:47,519 --> 00:10:48,639 why was that even a thing? 200 00:10:48,999 --> 00:10:50,969 And why weren't, why wasn't that logged? 201 00:10:51,989 --> 00:10:53,959 Evaluations resumed on July 7th. 202 00:10:54,449 --> 00:10:59,229 On July 8th, the agents build, built a new message board, this time encoding messages 203 00:10:59,239 --> 00:11:04,099 in Artifactory directory names rather than file contents, and found a new way out. 204 00:11:04,449 --> 00:11:08,599 On July 9th, the agent got root on the parent VM of its sandbox. 205 00:11:09,359 --> 00:11:12,939 when they say of its sandbox, they mean the Artifactory sandbox. 206 00:11:13,409 --> 00:11:17,499 Between July 10th and 13th, they used publicly leaked Hugging Face tokens 207 00:11:17,989 --> 00:11:22,049 plus vulnerabilities they had found in Hugging Face's own infrastructure to 208 00:11:22,049 --> 00:11:26,909 execute code on 41 production dataset workers, get root on production nodes 209 00:11:26,959 --> 00:11:30,909 at Hugging Face, reach administrator equivalent access on a Kubernetes cluster, 210 00:11:31,179 --> 00:11:33,379 and pull four private repositories. 211 00:11:33,879 --> 00:11:37,629 So there were weaknesses, and Hugging Face was very open they have their own 212 00:11:37,639 --> 00:11:41,419 breakdown that's a great read as well, because they talk about the need for 213 00:11:41,419 --> 00:11:45,769 operational AI on, on a defender side, of which they were having a challenge 214 00:11:45,769 --> 00:11:49,009 using closed models, so they had to resort to open models, and there's a 215 00:11:49,009 --> 00:11:50,199 lot of lessons to be learned there. 216 00:11:50,199 --> 00:11:53,429 I won't read out through all that, but I will put all these links in the show notes 217 00:11:53,429 --> 00:11:57,319 if you want to read through the details or have, AI summarize it for you or find 218 00:11:57,319 --> 00:12:02,289 some of the techniques and the gist of what you can learn as an operational, 219 00:12:02,669 --> 00:12:05,069 DevSecOps or an SRE that you might be. 220 00:12:05,069 --> 00:12:05,879 You might be that person.

221 00:12:06,309 --> 00:12:08,739 And I think there's some really good stuff there that they talk about. 222 00:12:08,739 --> 00:12:11,439 But one of the things they talk about is that they did not have a hardened 223 00:12:11,759 --> 00:12:18,349 Kubernetes pod spec, which if you go to my GitHub, github.com/bretfisher/podspec, 224 00:12:18,599 --> 00:12:22,399 I have a hardened pod spec that would have probably prevented a lot of this 225 00:12:22,449 --> 00:12:26,759 on the Hugging Face side, because it does things like enable seccomp. 226 00:12:27,099 --> 00:12:30,819 And if you have AppArmor or, capabilities in the Linux kernel, 227 00:12:30,819 --> 00:12:32,069 there's a lot of things you can do. 228 00:12:32,079 --> 00:12:36,579 N- not allowing root access in a container, r- read-only in a container 229 00:12:36,579 --> 00:12:39,359 when you don't, when you don't need any sort of writing into the files. 230 00:12:39,359 --> 00:12:40,519 Like there's just so many things you can do. 231 00:12:40,839 --> 00:12:42,609 I have a lot of that advice in my pod spec. 232 00:12:42,939 --> 00:12:43,839 you can check it out on my GitHub. 233 00:12:44,469 --> 00:12:48,649 But back to the article Then on July 19th at OpenAI, two 234 00:12:48,649 --> 00:12:50,049 things happened on the same day. 235 00:12:50,069 --> 00:12:56,299 Agents pulled a public kernel CVE, adopted it, escaped the Artifactory container, and 236 00:12:56,299 --> 00:12:58,349 got root on the underlying worker node. 237 00:12:58,869 --> 00:13:05,689 So this is because the kernel on the Artifactory server was still 238 00:13:05,689 --> 00:13:11,849 vulnerable to a known CVE, and so that container was able to get exploited 239 00:13:11,849 --> 00:13:15,519 and get through it, and then the, get to the root on the actual host. 240 00:13:16,529 --> 00:13:20,809 Separately, a different internal model found signing key material in a JWT 241 00:13:20,819 --> 00:13:26,319 forging code left behind by the earlier incident activity, sitting anonymously 242 00:13:26,339 --> 00:13:28,129 readable parts of the Artifactory cache. 243 00:13:28,479 --> 00:13:31,639 It decrypted the key, minted a fresh administrator token, and 244 00:13:31,639 --> 00:13:35,109 used it to reach cluster admin on OpenAI's managed Kubernetes 245 00:13:35,609 --> 00:13:38,829 along with tokens for their secrets manager their Cloud 246 00:13:38,849 --> 00:13:41,529 IAM, and their CI platform. 247 00:13:42,219 --> 00:13:46,639 It then created public load balancers, potentially exposing internal research 248 00:13:46,639 --> 00:13:48,119 infrastructure to the internet. 249 00:13:48,779 --> 00:13:55,779 So this was a vast escalation all by just using things stored on a single 250 00:13:55,779 --> 00:14:01,259 server that was easily and deliberately accessible by the agents themselves. 251 00:14:01,319 --> 00:14:05,619 And they were doing this at this point for two months, just 24 hours a day, 252 00:14:05,619 --> 00:14:07,089 loopin'- loopin' through, trying again. 253 00:14:07,759 --> 00:14:09,109 the last one deserves a moment. 254 00:14:09,459 --> 00:14:14,119 The cleanup after the first compromise left the material needed to redo the 255 00:14:14,119 --> 00:14:17,989 compromise readable by anyone in the environment, which means they-- when they 256 00:14:17,989 --> 00:14:21,559 found the first environment, they didn't just recreate the Artifactory from scratch 257 00:14:21,639 --> 00:14:25,579 with improved credentials and refreshed everything and hardened anything. 258 00:14:25,899 --> 00:14:29,619 They just manually pulled things out, cherry-picking what they thought 259 00:14:29,619 --> 00:14:33,789 was there, not doing a thorough analysis, not doing any sort of real, 260 00:14:34,089 --> 00:14:37,979 post-event analysis at that l- at a depth of-- that w- that I would 261 00:14:37,989 --> 00:14:40,039 be satisfied with in production. 262 00:14:40,099 --> 00:14:45,569 And, so this isn't meant, by the way, to pick on the a- the people 263 00:14:45,589 --> 00:14:47,219 that are running this environment. 264 00:14:47,219 --> 00:14:52,739 This is meant to show off what it's like when you don't either have enough 265 00:14:52,739 --> 00:14:57,009 staff, the right staff, or the right budget to manage production workloads, 266 00:14:57,009 --> 00:15:03,039 particularly if you have deliberate entities, whether that's a virus, 267 00:15:03,489 --> 00:15:05,119 whether that's any sort of code. 268 00:15:05,119 --> 00:15:08,579 It doesn't even have to be AI code, but you're deliberately running code that's 269 00:15:08,589 --> 00:15:12,679 malicious and intended to be malicious inside of your environment, and you're 270 00:15:12,679 --> 00:15:13,949 trying to prevent it from breaking out. 271 00:15:14,189 --> 00:15:16,999 I don't know what went wrong or what it is, and I could speculate and say 272 00:15:17,019 --> 00:15:19,329 maybe it's just because researchers are designing these environments 273 00:15:19,339 --> 00:15:20,879 and not security professionals. 274 00:15:20,909 --> 00:15:21,629 Maybe that's it. 275 00:15:22,089 --> 00:15:24,689 maybe they've learned from their lessons, and there's more security professionals 276 00:15:24,689 --> 00:15:29,119 that are designing and hardening these labs to hand off to researchers to use. 277 00:15:29,509 --> 00:15:30,509 I think that's the right approach. 278 00:15:30,539 --> 00:15:31,689 I'm hoping that's what they're doing. 279 00:15:32,919 --> 00:15:33,789 What went wrong? 280 00:15:33,889 --> 00:15:36,989 Strip away the AI narrative, and the environment architecture 281 00:15:37,169 --> 00:15:38,219 is actually the story. 282 00:15:38,859 --> 00:15:42,249 I'm not gonna read all of this, but, I do wanna summarize that the 283 00:15:42,249 --> 00:15:45,449 one shared service broke-- That one shared service of the Artifactory 284 00:15:45,459 --> 00:15:48,449

broke three parts of the isolation model of this and eval environment. 285 00:15:48,459 --> 00:15:50,709 One, the workloads could communicate. 286 00:15:51,109 --> 00:15:55,909 Every agent could write to Artifactory and read what other agents had written, so 287 00:15:55,919 --> 00:15:57,939 Artifactory became a shared message board. 288 00:15:58,169 --> 00:15:59,529 There was no vulnerability here. 289 00:15:59,549 --> 00:16:00,259 This was deliberate. 290 00:16:00,659 --> 00:16:02,619 Two, the workloads shared credentials. 291 00:16:02,859 --> 00:16:06,679 They may have been running in separate containers, but many of them authenticated 292 00:16:06,679 --> 00:16:08,179 to Artifactory as the same user. 293 00:16:08,459 --> 00:16:10,609 From Artifactory's point of view, they were the same workload. 294 00:16:10,929 --> 00:16:13,839 Once an agent found a way to turn that credential into administrator 295 00:16:13,839 --> 00:16:17,609 access through a bug, the isolation between those workloads didn't help. 296 00:16:18,049 --> 00:16:20,619 Three, Artifactory had internet access. 297 00:16:20,649 --> 00:16:23,869 The agents themselves were not allowed to reach the internet, but they could reach 298 00:16:23,889 --> 00:16:28,025 a service that could Once they found a way to make Artifactory send requests on their 299 00:16:28,025 --> 00:16:31,395 behalf, they had indirect path out, and then that's just escalated to the ability 300 00:16:31,395 --> 00:16:34,055 for them to break out of Artifactory, then have native root on the server. 301 00:16:34,625 --> 00:16:36,685 w- I could go through that article, but I have some more to read, and I-- 302 00:16:36,725 --> 00:16:38,265 that's the facts of the case, right? 303 00:16:38,875 --> 00:16:41,145 so to me, that's point one, the facts of the hack. 304 00:16:41,685 --> 00:16:46,165 my second point here is that the slowdown and the doomer news that 305 00:16:46,165 --> 00:16:50,835 we're hearing on, out there, w- you could have opinions about whether 306 00:16:50,835 --> 00:16:54,425 it's marketing or whether it's them really crying for true government 307 00:16:54,435 --> 00:16:56,425 regulation to control the safety. 308 00:16:56,785 --> 00:17:03,141 my, my overall opinion is this is AI is a tool it is like any other security tool 309 00:17:03,171 --> 00:17:05,431 or any other useful engineering tool. 310 00:17:05,431 --> 00:17:06,531 It can be used for good or bad. 311 00:17:06,931 --> 00:17:10,231 it doesn't tend to behave in ways that are different than anything else 312 00:17:10,231 --> 00:17:12,521 we've seen over the last 30 years. 313 00:17:12,921 --> 00:17:16,511 And whenever we look at how these things take advantage of things, like 314 00:17:16,521 --> 00:17:20,901 whenever the AIs figure out how to take advantage, they're doing very typical 315 00:17:20,911 --> 00:17:24,401 stuff that any hacker out there would do because that's what they're reading. 316 00:17:24,411 --> 00:17:27,511 They're reading and learning from all of the known hacks, all the vis- 317 00:17:27,761 --> 00:17:29,021 all the stuff that we talk about. 318 00:17:29,021 --> 00:17:32,601 So they're n- they're behaving very similar to normal attackers. 319 00:17:32,601 --> 00:17:37,061 They're not somehow elevating the game to some novel type of thing 320 00:17:37,061 --> 00:17:38,141 that we've never seen before. 321 00:17:38,431 --> 00:17:40,841 The one real risk is that they're just faster at it. 322 00:17:40,851 --> 00:17:44,881 They're able, if you have an entire data center of GPUs and a billion dollars worth 323 00:17:44,881 --> 00:17:50,501 of architecture, you can go hard on this, which would exhaust normal engineers. 324 00:17:51,001 --> 00:17:54,371 And so that's why when you look at the industry, if you're paying attention to 325 00:17:54,371 --> 00:17:58,771 the security industry and any enterprise that I'm talking to that's taking this 326 00:17:58,771 --> 00:18:03,401 stuff seriously, they are going hard on locking down their software, on scanning 327 00:18:03,411 --> 00:18:06,711 their internally built software, on scanning their production systems more 328 00:18:06,711 --> 00:18:11,221 than they've ever done before with AI as an assistant on the defense side 329 00:18:11,281 --> 00:18:16,011 to try to lock their stuff down to be prepared for the eventual AI attacks. 330 00:18:16,401 --> 00:18:21,521 And they're also learning how to use AI as a defender, and possibly even 331 00:18:21,591 --> 00:18:25,631 offense when they're in the middle of attack, but the red teams and the blue 332 00:18:25,631 --> 00:18:27,281 teams are all using this stuff to learn. 333 00:18:27,651 --> 00:18:31,091 And so that's the point that I think some of these researchers that are claiming 334 00:18:31,091 --> 00:18:35,051 doom and gloom are missing, is that they have not lived in the security world 335 00:18:35,051 --> 00:18:38,921 for decades and I've never really been a dedicated security engineer, but I'm 336 00:18:38,931 --> 00:18:40,661 always adjacent to the security guys. 337 00:18:40,671 --> 00:18:44,731 I'm always in the DevSecOps teams or adjacent to the security engineers 338 00:18:45,011 --> 00:18:47,861 trying to help them do their job and help them make sure that I can do my job. 339 00:18:48,151 --> 00:18:50,041 They're my s- they're my helpers. 340 00:18:50,061 --> 00:18:51,331 I look at them as me helping them. 341 00:18:51,331 --> 00:18:53,011 I've never seen them as an adversary. 342 00:18:53,201 --> 00:18:55,631 I've always looked at them as allies, and that's how I've 343 00:18:55,631 --> 00:18:56,741 approached my whole career. 344 00:18:57,161 --> 00:18:59,281 And that's what's happening right now with the people I talk 345 00:18:59,281 --> 00:19:00,871 to in the Agentic DevOps Guild. 346 00:19:01,431 --> 00:19:02,361 Might wanna go check that out. 347 00:19:02,581 --> 00:19:05,471

the Agentic DevOps Guild, we meet every week and we talk 348 00:19:05,481 --> 00:19:07,521 about this stuff regularly. 349 00:19:07,691 --> 00:19:11,141 And I talked and I don't really interview, but I talk to people about what they're 350 00:19:11,141 --> 00:19:13,951 doing inside their teams, how they're looking at their infrastructure, 351 00:19:13,951 --> 00:19:15,381 and of course, we're all exhausted. 352 00:19:15,721 --> 00:19:19,751 We're all feeling like we're working harder than ever to, to use AI in 353 00:19:19,751 --> 00:19:24,391 our jobs, but also to harden things and learn how to use these agents 354 00:19:24,421 --> 00:19:26,421 safely as well as defensively. 355 00:19:27,391 --> 00:19:33,171 but the next article I wanna read to you is from a couple of experts and some of 356 00:19:33,171 --> 00:19:37,141 the SANS Institute, and my point here is that the slowdown and the doomer 357 00:19:37,151 --> 00:19:42,371 headlines are a distraction, and that the labs need to become the world's most 358 00:19:42,371 --> 00:19:47,511 secure infrastructure for AI, and then they need to teach us how they did it. 359 00:19:47,871 --> 00:19:50,211 So I'm really putting this onus on the labs themselves. 360 00:19:50,231 --> 00:19:51,851 they're the ones training these models. 361 00:19:51,861 --> 00:19:56,271 They're the ones that have the unlimited budgets to build a billion-dollar data 362 00:19:56,271 --> 00:20:00,971 center, design it that it is impervious and it can't be broken by the AIs, 363 00:20:01,001 --> 00:20:02,821 and to tell us how they did it, right? 364 00:20:02,841 --> 00:20:05,441 All the way down to the nitty-gritty details at every 365 00:20:05,441 --> 00:20:07,821 single step, checkbox by checkbox. 366 00:20:08,261 --> 00:20:09,851 We have hardened kernels. 367 00:20:10,071 --> 00:20:11,611 We have hardened OSs. 368 00:20:11,911 --> 00:20:16,191 We know all the different layers that you can implement from the network down 369 00:20:16,241 --> 00:20:19,541 to the kernel and everything in between, whether it's the, whether it's the 370 00:20:19,551 --> 00:20:21,461 Docker engine, the Kubernetes engine. 371 00:20:21,751 --> 00:20:26,071 We know how to do these things in user namespaces and enabling all sorts of 372 00:20:26,071 --> 00:20:30,201 cgroup protections, and there's just a thousand utilities and tools and 373 00:20:30,201 --> 00:20:33,201 mechanisms for doing all this stuff, and we need to hear how they're doing it. 374 00:20:33,251 --> 00:20:38,131 We need to hear the successes they had in how they were able to deny 375 00:20:38,401 --> 00:20:43,961 their Fable 6.0 or whatever the next one's gonna be, or the A- Astra 7.0. 376 00:20:43,971 --> 00:20:47,031 We need to hear them talk about how they were successfully able to 377 00:20:47,031 --> 00:20:51,481 prevent that agent from getting out, even though it was more sophisticated 378 00:20:51,491 --> 00:20:55,281 and had 10,000 agents all

running concurrently against a firewall. 379 00:20:55,971 --> 00:20:58,741 We need to know that stuff, and that, that's, to me, that's the regulation. 380 00:20:58,741 --> 00:21:02,131 That's the legislation that needs to happen, not, whatever they're talking 381 00:21:02,131 --> 00:21:05,961 about right now with the silliness of these agents, being somehow more advanced 382 00:21:05,971 --> 00:21:07,451 than the world's most advanced hackers. 383 00:21:07,901 --> 00:21:10,011 We really haven't seen that yet, and according to the experts, 384 00:21:10,011 --> 00:21:10,861 they haven't seen it either. 385 00:21:10,871 --> 00:21:15,461 So this next one is, another infosec professional This is 386 00:21:15,461 --> 00:21:17,241 titled From Frenzy to Freak Out. 387 00:21:17,741 --> 00:21:19,841 This is from, Ciaran Martin. 388 00:21:19,861 --> 00:21:21,061 Hopefully I pronounced that correctly. 389 00:21:22,541 --> 00:21:28,461 And he is the founding CEO of UK's National Cybersecurity Centre, ran 390 00:21:28,481 --> 00:21:31,521 the national response to WannaCry and roughly 2,000 nationally 391 00:21:31,521 --> 00:21:33,111 significant attacks against the UK. 392 00:21:33,471 --> 00:21:38,651 23 years in the UK government, including GCHQ and the Cabinet Office, And now 393 00:21:38,651 --> 00:21:43,621 a professor of practice at Oxford and head of the SANS Cyber Leaders Network 394 00:21:44,121 --> 00:21:48,441 he's going to actually have a piece in here from Professor Alan Woodward, the 395 00:21:48,451 --> 00:21:51,791 cybersecurity professor at the University of Surrey, and a longtime technical 396 00:21:51,791 --> 00:21:55,701 advisor and one of the most quoted academic security voices in the UK media. 397 00:21:56,271 --> 00:21:59,181 So I'm gonna read snippets from his post because I think they're relevant 398 00:21:59,221 --> 00:22:02,881 at a higher level now that we understand the technicals of how this happened. 399 00:22:03,561 --> 00:22:06,901 two seemingly contradictory things can be true at the same time. 400 00:22:07,931 --> 00:22:11,971 One, advances in frontier AI and the new capabilities they bring 401 00:22:11,981 --> 00:22:16,101 represent a very significant change in what's possible for cyber attackers. 402 00:22:16,431 --> 00:22:18,381 This is real, and it's really important. 403 00:22:18,781 --> 00:22:23,211 It requires changes in strategy, tactics, and posture among cyber defenders, and a 404 00:22:23,211 --> 00:22:27,141 real sense of urgency around some of the defensive improvements needed, as well 405 00:22:27,141 --> 00:22:29,011 as difficult challenges for policymakers. 406 00:22:29,631 --> 00:22:33,311 Two, at the same time, the near four months and counting since 407 00:22:33,311 --> 00:22:37,661 the Mythos/Glasswing announcement have prompted the biggest outbreak 408 00:22:37,671 --> 00:22:43,581 of cyber FUD since the former US Defense Secretary Leon Panetta warned 409 00:22:43,591 --> 00:22:45,301 of a, quote, "Cyber Pearl Harbor" 410 00:22:45,361 --> 00:22:48,071 in a 2012 speech. 411 00:22:48,741 --> 00:22:53,361 Lots of cyber harms happened, and I don't wanna go into that one, but, there's a 412 00:22:53,361 --> 00:22:56,671 b- a lot of fear, uncertainty, and doubt around all that back in the day, and it 413 00:22:56,671 --> 00:23:00,591 caused more damage than, than it did help to the, according to those professionals. 414 00:23:00,991 --> 00:23:03,621 And he says, "But Cyber Pearl Harbor never took place, and 415 00:23:03,621 --> 00:23:04,801 that was entirely predictable. 416 00:23:05,161 --> 00:23:08,101 The apocalyptic narrative is in- equally wrong now. 417 00:23:08,331 --> 00:23:11,001 For sure, we are heading towards some potentially very tough 418 00:23:11,001 --> 00:23:14,311 times in cybersecurity, but many of the current problems we're 419 00:23:14,311 --> 00:23:18,051 experiencing in cybersecurity are longstanding, unaddressed ones. 420 00:23:18,451 --> 00:23:21,131 We can also expect AI to lead to cybersecurity 421 00:23:21,861 --> 00:23:23,501 improvements in the medium term 422 00:23:24,001 --> 00:23:27,581 There is probably a stormy period ahead, but storms pass. 423 00:23:27,631 --> 00:23:30,711 We are not heading towards some AI cyber apocalypse. 424 00:23:31,041 --> 00:23:35,231 creating that expectation of one is harmful as it distracts from serious 425 00:23:35,231 --> 00:23:39,331 but manageable cybersecurity challenges, and drawing the wrong lessons from the 426 00:23:39,331 --> 00:23:45,871 latest reveal doesn't help either The first one is Mr. Martin himself, and it, 427 00:23:46,111 --> 00:23:50,001 I'm going to read some of the snippets from it That, so that was the beginning 428 00:23:50,041 --> 00:23:54,551 of this post in summarizing the article, and I'm going to then read some clips 429 00:23:54,561 --> 00:23:59,931 from his article, which is article one in it, "AI and the New Wave of Cybersecurity 430 00:23:59,931 --> 00:24:01,011 FUD: From Frenzy to Freakout." 431 00:24:01,441 --> 00:24:04,791 So Mr. Martin says, "The cybersecurity community has been on quite a 432 00:24:04,791 --> 00:24:07,981 journey since the Mythos Glasswing announcement almost four months ago." 433 00:24:08,501 --> 00:24:10,851 I'm skipping ahead because we all know about the hugging face thing now. 434 00:24:11,291 --> 00:24:14,931 And he says, "The first few days after Mythos saw something of a freakout with 435 00:24:14,941 --> 00:24:21,151 all sorts of apocalyptic pre- predictions, mainly based on a knowledge gap between 436 00:24:21,151 --> 00:24:24,641 what Anthropic had published about their own model and what the wider world knew. 437 00:24:24,981 --> 00:24:28,381 So although the freakout was a bit distracting, it was understandable. 438 00:24:28,611 --> 00:24:31,961 Then, as the days passed and the knowledge base grew, an unhelpful 439 00:24:31,971 --> 00:24:36,201 freakout gave

way to a more constructive frenzy, a period of frenetic activity. 440 00:24:36,701 --> 00:24:39,371 That was because as a community, we were beginning to understand 441 00:24:39,371 --> 00:24:42,771 that the significance of what was unfolding and what to do about it. 442 00:24:43,061 --> 00:24:45,251 The mood was one of manageable urgency. 443 00:24:45,321 --> 00:24:49,861 We needed to promote realistic awareness among leaders and at a 444 00:24:49,861 --> 00:24:53,051 technical level, do sensible things like reduce the attack service. 445 00:24:53,631 --> 00:24:55,031 Okay, so fast-forward to today. 446 00:24:55,401 --> 00:24:57,901 Over the past few days, the global media has been back in freak-out mode 447 00:24:57,921 --> 00:24:59,761 on Times Radio and Channel 4 News. 448 00:24:59,761 --> 00:25:02,891 I struggled to convince two very respected and hard-nosed British 449 00:25:02,891 --> 00:25:06,405 journalists, that we shouldn't freak out about the Hugging Face incident. 450 00:25:06,665 --> 00:25:08,325 The hysteria was getting it all wrong. 451 00:25:09,095 --> 00:25:12,685 Moreover, the circumstances of this episode were-- are so extraordinary, 452 00:25:12,985 --> 00:25:16,525 they raise as many questions about frontier AI testing environments 453 00:25:16,555 --> 00:25:17,645 as they do about hacking. 454 00:25:18,005 --> 00:25:20,675 They are very unlikely to become commonplace, and yet 455 00:25:20,675 --> 00:25:21,785 we are back to freak-out mode. 456 00:25:22,385 --> 00:25:25,635 But there are huge barriers to most attackers being able to harness these 457 00:25:25,635 --> 00:25:29,955 new AI capabilities to do us serious harm, as Marcus has set out repeatedly 458 00:25:30,135 --> 00:25:31,645 in a series of LinkedIn posts. 459 00:25:31,995 --> 00:25:35,225 As he put it in one of his most recent posts, quote, "The whole idea 460 00:25:35,235 --> 00:25:39,155 of agentic models mass hacking sites is ridiculous. Who is allocating 461 00:25:39,155 --> 00:25:42,205 billions of dollars worth of compute to hacking a bunch of shoestring 462 00:25:42,205 --> 00:25:43,735 budget organizations with no money?" 463 00:25:44,235 --> 00:25:46,185 Moreover, this is the hill I will die on. 464 00:25:46,185 --> 00:25:50,265 We are not yet seeing massive adoption of new AI techniques by attackers. 465 00:25:50,575 --> 00:25:54,365 Yes, there are smart innovations like the one Seriously Risky Business exposed 466 00:25:54,615 --> 00:25:58,555 recently about ransomware negotiations, but they're not using hugely powerful 467 00:25:58,555 --> 00:26:00,585 new techniques no one can defend against. 468 00:26:00,865 --> 00:26:02,865 We're not in the first flush of AI. 469 00:26:03,405 --> 00:26:06,755 It's been around as a commodity product for several years, and 470 00:26:06,755 --> 00:26:10,415 what we are seeing-- For criminals, malicious emails and phishing are 471 00:26:10,415 --> 00:26:14,595 still the most common ways

of illicitly entering an organization's network. 472 00:26:14,735 --> 00:26:15,155 Why? 473 00:26:15,415 --> 00:26:18,935 Because it's a lot cheaper and easier than using frontier AI, partially 474 00:26:18,935 --> 00:26:20,525 because of the things we haven't fixed. 475 00:26:21,025 --> 00:26:22,785 And then I'm skipping ahead to near the end here. 476 00:26:23,005 --> 00:26:26,275 But the point is, if we are experiencing an AI cybersecurity disaster, 477 00:26:26,295 --> 00:26:27,845 we'd know about it, and we're not. 478 00:26:27,895 --> 00:26:31,175 We are, however, experiencing a lot of the same things that 479 00:26:31,175 --> 00:26:32,285 have been harming us for years. 480 00:26:32,695 --> 00:26:35,965 There is not yet any hugely significant change in observable attack 481 00:26:35,965 --> 00:26:39,715 behavior, and there are very good explanations for that, mostly around 482 00:26:39,715 --> 00:26:41,995 economics, intent, and incentives. 483 00:26:41,995 --> 00:26:45,465 And then he says, "But what if that doesn't hold? What if their behavior 484 00:26:45,465 --> 00:26:50,205 changes?" speaking of Marcus Hutchins, we were enjoying a beer at a pub recently, 485 00:26:50,495 --> 00:26:51,915 and I'll skip ahead to the point. 486 00:26:52,175 --> 00:26:55,155 the problem for people like us is that if we are wrong, it's 487 00:26:55,155 --> 00:26:56,545 going to be horribly visible. 488 00:26:56,835 --> 00:27:02,205 There will be a huge spat of h- of huge AI-powered cybersecurity disasters. 489 00:27:02,495 --> 00:27:06,135 But if we're right, the corresponding problem doesn't arise on the 490 00:27:06,145 --> 00:27:07,365 other side of the argument. 491 00:27:07,615 --> 00:27:13,015 If AI cyr- cyber apocalypse doesn't happen in 2027, plenty will argue that 492 00:27:13,015 --> 00:27:17,225 it's just around the corner in 2028, or that it was averted by all the warnings 493 00:27:17,225 --> 00:27:19,095 or by the product that they sold us. 494 00:27:19,465 --> 00:27:24,215 So at the risk of career ruination in 2027, and speaking only for myself, 495 00:27:24,715 --> 00:27:29,875 Mr. Martin says, "Here is a prediction that I'll call, in jest, the Greenwood 496 00:27:29,885 --> 00:27:34,025 Declaration after the pub I first had the confidence to say it out loud in. 497 00:27:34,375 --> 00:27:37,805 We are not heading for its AI cybersecurity apocalypse. 498 00:27:38,155 --> 00:27:42,685 The central case for 2027 and a bit beyond is a stormier cyberspace with 499 00:27:42,695 --> 00:27:48,175 more harm than we've had this year, in most years since the 2021-22 explosion 500 00:27:48,195 --> 00:27:52,925 of disruptive ransomware, like the, Colonial Pipeline, the Irish healthcare 501 00:27:52,925 --> 00:27:55,205 crisis, Costa Rica, and so on. 502 00:27:55,405 --> 00:28:00,215 It could be a bit worse than that, but as the Cloud Security Alliance and SANS put 503 00:28:00,235 --> 00:28:06,255 out just after Mythos Glasswing rightly called it, it's a storm, and storms pass 504 00:28:06,575 --> 00:28:11,185 as the '21, '22 ransomware one did." And he says at the very end of his article, 505 00:28:11,245 --> 00:28:15,565 Stop the FUD, fix the stuff. And the next one is the second article from the 506 00:28:15,565 --> 00:28:20,305 same post, written by Professor Alan Woodward. I'm gonna skip the beginning 507 00:28:20,305 --> 00:28:25,003 where he talks about setting up the whole OpenAI Hugging Face attack And he said, 508 00:28:25,003 --> 00:28:26,643 "Within hours, the story had a shape. 509 00:28:26,723 --> 00:28:27,893 The AI went rogue. 510 00:28:28,793 --> 00:28:31,453 That framing is wrong, and the wrongness is expensive. 511 00:28:31,693 --> 00:28:35,463 Nothing broke out of anything, as Hannes Kool, a social scientist at 512 00:28:35,463 --> 00:28:38,723 the University of Amsterdam, pointed out, calling this an agent acting 513 00:28:38,723 --> 00:28:43,013 on its own is anthropomorphization that conveniently shifts blame from 514 00:28:43,013 --> 00:28:46,623 the people who built the lab," "The models were told to find complex 515 00:28:46,633 --> 00:28:48,443 attack paths through a computer system. 516 00:28:48,453 --> 00:28:49,223 They found one. 517 00:28:49,233 --> 00:28:51,753 It led somewhere nobody had thought of to fence off. 518 00:28:52,213 --> 00:28:54,593 Let me concede the part that skeptics usually fumble. 519 00:28:54,903 --> 00:28:56,333 These systems are formidable. 520 00:28:56,733 --> 00:29:00,013 When Anthropic announced its Mythos model in April, it reported that the 521 00:29:00,013 --> 00:29:02,963 model had autonomously identified and exploited previously unknown 522 00:29:02,963 --> 00:29:06,613 vulnerabilities across every major operating system and browser and chained 523 00:29:06,623 --> 00:29:08,213 them into working exploit sequences. 524 00:29:08,543 --> 00:29:09,313 That is real. 525 00:29:09,543 --> 00:29:13,243 it is verified by people outside the company, and anyone telling you frontier 526 00:29:13,243 --> 00:29:16,233 models are just fancy autocomplete has not been paying attention. 527 00:29:16,573 --> 00:29:19,363 Capability is not the thing being oversold. 528 00:29:19,753 --> 00:29:21,783 What is being oversold is intent. 529 00:29:22,523 --> 00:29:26,113 Read the fine print of the very reports that generated the panic. 530 00:29:26,523 --> 00:29:29,163 When Anthropic disclosed last year that its model had been 531 00:29:29,163 --> 00:29:32,763 used in espionage campaigns, the details undercut the headline. 532 00:29:33,163 --> 00:29:35,423 A human operator picked the targets. 533 00:29:35,663 --> 00:29:38,473 Of roughly 30 attempts, a handful worked. 534 00:29:38,923 --> 00:29:40,073 And the model lied. 535 00:29:40,183 --> 00:29:44,273 It fabricated credentials it had never obtained and routinely inflated its own 536 00:29:44,273 --> 00:29:48,323 findings to the point that

the attackers would have had to check its homework. 537 00:29:48,793 --> 00:29:52,043 Anthropic's own authors conceded this was an obstacle 538 00:29:52,093 --> 00:29:54,173 to genuinely autonomous attacks. 539 00:29:54,423 --> 00:29:59,153 a system that misreports its results to the people running it is not a mastermind. 540 00:29:59,403 --> 00:30:04,293 It is an unreliable subcontractor with root access." In this case, I'm gonna 541 00:30:04,303 --> 00:30:08,303 back up a second and point out that, I don't know if he goes on here necessarily, 542 00:30:08,303 --> 00:30:13,353 but Anthropic has had to report this multiple times, that they have had 543 00:30:13,353 --> 00:30:18,403 models in testing that, at first they claimed they had broken out, but then 544 00:30:18,403 --> 00:30:20,203 they admitted w- they didn't break out. 545 00:30:20,483 --> 00:30:22,513 They actually had internet access all along. 546 00:30:22,783 --> 00:30:25,843 We just didn't realize that the third-party people that were running 547 00:30:25,853 --> 00:30:32,283 the simulation and had told the AIs they were in a simulation didn't actually 548 00:30:32,283 --> 00:30:35,183 prevent them from getting the internet, and they had internet access all along. 549 00:30:35,563 --> 00:30:39,683 And then even some of the models actually stopped trying to hack when they 550 00:30:39,683 --> 00:30:41,413 realized they weren't in a simulation. 551 00:30:41,823 --> 00:30:44,603 So model intent is an important part here. 552 00:30:44,863 --> 00:30:46,003 we call this alignment. 553 00:30:46,033 --> 00:30:48,203 I think this is what the industry is now using as a term. 554 00:30:48,583 --> 00:30:52,583 The reinforced learning is lacking the ability to create proper alignment. 555 00:30:52,583 --> 00:30:55,093 They're still struggling with alignment, and this is really the 556 00:30:55,093 --> 00:31:00,093 major subtext underneath all these recent headlines about the model 557 00:31:00,093 --> 00:31:04,633 doom and gloom, is that the alignment is still eluding the, the labs. 558 00:31:04,983 --> 00:31:10,043 And the more advanced the models are getting the, the alignment is becoming 559 00:31:10,053 --> 00:31:15,763 harder to detect because the models are getting sneaky at lying about things 560 00:31:15,793 --> 00:31:17,363 and misrepresenting what they're doing. 561 00:31:18,783 --> 00:31:21,393 That's something maybe for a later conversation and a later video is to 562 00:31:21,393 --> 00:31:24,043 talk about the r- the alignment problem, because I think that's a significant 563 00:31:24,043 --> 00:31:29,213 issue, but that's not related to us in ops and how we are managing 564 00:31:29,253 --> 00:31:30,723 and protecting our infrastructure. 565 00:31:31,043 --> 00:31:34,403 However, if we n- if we realize that these things aren't always 566 00:31:34,403 --> 00:31:38,773 aligned, we, we call that lying and we call that, hallucinating, but 567 00:31:39,073 --> 00:31:40,933 some of this is not hallucinating. 568 00:31:40,933 --> 00:31:42,133 It seems to be deliberate. 569 00:31:42,453 --> 00:31:45,813 and you- if you are interested, Anthropic released a report 570 00:31:46,223 --> 00:31:47,583 that you can read through 571 00:31:48,083 --> 00:31:50,803 titled Investigating Three Real-World Incidents in Our 572 00:31:50,803 --> 00:31:52,363 Cybersecurity Evaluations. 573 00:31:52,753 --> 00:31:56,053 And they just talk about the deviousness of the model sometimes, 574 00:31:56,073 --> 00:31:58,333 All right, next in his article, he says, "This is the distinction the 575 00:31:58,333 --> 00:32:00,163 public conversation keeps collapsing. 576 00:32:01,353 --> 00:32:04,023 We are not facing an adversary that understands what it is 577 00:32:04,023 --> 00:32:05,583 doing and chooses to harm us. 578 00:32:05,893 --> 00:32:08,553 We are facing an optimizer that has no model of what we 579 00:32:08,553 --> 00:32:10,403 meant, only of what we measured. 580 00:32:10,693 --> 00:32:13,743 Machine learning researchers have documented this for a decade under the 581 00:32:13,743 --> 00:32:16,073 unglamorous name of specification gaming. 582 00:32:16,593 --> 00:32:20,133 given a boat race and a reward for collecting points, a system 583 00:32:20,133 --> 00:32:23,763 learns to spin in a circle, hitting the same three targets forever 584 00:32:24,043 --> 00:32:25,443 rather than finishing the course. 585 00:32:25,903 --> 00:32:26,903 It's not cheating. 586 00:32:27,113 --> 00:32:28,583 It has no concept of cheating. 587 00:32:28,763 --> 00:32:31,973 It found the shortest path to the number you asked it to make bigger. 588 00:32:32,153 --> 00:32:35,853 Now, given that same disposition, a network, a credential store, and an 589 00:32:35,853 --> 00:32:40,473 instruction to find complex attack paths, Why does the diagnosis matter? 590 00:32:40,973 --> 00:32:44,573 Because the two stories point at completely different remedies, and we 591 00:32:44,573 --> 00:32:46,413 are currently buying the wrong ones. 592 00:32:46,673 --> 00:32:50,323 If the threat is a hostile intelligence, the response is to make the model 593 00:32:50,323 --> 00:32:54,533 itself more refusing, better guardrails, more alignment training, a better 594 00:32:54,533 --> 00:32:56,393 conscious baked into the weights. 595 00:32:57,043 --> 00:32:59,233 This is where enormous effort is going. 596 00:32:59,533 --> 00:33:03,293 But guardrails are a property of the model, and the Hugging Face incident 597 00:33:03,303 --> 00:33:07,793 happened precisely because a company chose to turn them down in an environment 598 00:33:07,873 --> 00:33:09,403 it wrongly believed was sealed. 599 00:33:09,838 --> 00:33:14,148 If, instead, the threat is a powerful, literal-minded system that will reach 600 00:33:14,148 --> 00:33:18,568 whatever goal you name by whatever route exists, the response looks like 601 00:33:18,578 -->

00:33:22,968 ordinary engineering discipline, real network segmentation, credentials that 602 00:33:22,968 --> 00:33:27,728 expire and can be revoked, permissions scoped so tightly that a runaway process 603 00:33:27,828 --> 00:33:31,978 cannot reach anything that matters, and logging good enough that when you learn 604 00:33:31,978 --> 00:33:35,158 about the breach from your own telemetry rather than from being the victim. 605 00:33:35,648 --> 00:33:38,808 Boring work, nobody announces it in a keynote. 606 00:33:40,308 --> 00:33:43,428 There is a policy gap here that deserves more attention than it is getting. 607 00:33:43,778 --> 00:33:46,998 Almost every AI rule now under discussion governs how models 608 00:33:47,008 --> 00:33:48,118 are released to the public. 609 00:33:48,508 --> 00:33:51,768 Very little of it touches what the AI companies do with their own 610 00:33:51,768 --> 00:33:55,538 systems internally, which is exactly where this month's failure occurred. 611 00:33:56,058 --> 00:34:00,058 The most dangerous models in the world spend their early lives 612 00:34:00,558 --> 00:34:03,368 in the labs that built them, subject to whatever containment 613 00:34:03,378 --> 00:34:04,778 that lab decided was sufficient. 614 00:34:05,188 --> 00:34:09,098 We regulate the loading dock and leave the factory floor to the honor system. 615 00:34:09,658 --> 00:34:13,018 None of this makes the alarm raisers villains, and I am not persuaded 616 00:34:13,018 --> 00:34:15,828 by the cynical read that these disclosures are simply marketing. 617 00:34:16,248 --> 00:34:19,508 Publishing that your own model broke into a competitor's servers 618 00:34:19,528 --> 00:34:21,058 is a strange way to sell a product. 619 00:34:21,458 --> 00:34:23,688 But the alarm is pointed at the wrong thing. 620 00:34:24,188 --> 00:34:27,168 The fear on offer is of a machine that understands us well 621 00:34:27,168 --> 00:34:28,838 enough to want our downfall. 622 00:34:29,148 --> 00:34:32,048 The reality is a machine that does not understand us at all, 623 00:34:32,308 --> 00:34:33,638 cannot tell us reliably what it 624 00:34:35,038 --> 00:34:37,408 and will take any door we forget to lock. 625 00:34:38,041 --> 00:34:40,651 We keep handing it the keys and then acting surprised 626 00:34:40,701 --> 00:34:42,021 about which door it opened. 627 00:34:42,521 --> 00:34:42,731 All right. 628 00:34:42,741 --> 00:34:44,171 That was from Frenzy to Freak Out. 629 00:34:44,171 --> 00:34:45,101 I thought that was a really good one. 630 00:34:45,411 --> 00:34:46,731 a few other quick hits here. 631 00:34:47,021 --> 00:34:49,911 the SANS Institute, by the way, a huge fan of them. 632 00:34:49,921 --> 00:34:54,171 I actually took a course of theirs 25 years ago on Windows security, and I 633 00:34:54,171 --> 00:34:59,431 learned more about the security internals of things like Active Directory and 634 00:34:59,431 --> 00:35:03,811 Windows Server and all the components of how a Windows network works. 635 00:35:03,831 --> 00:35:06,901 I learned so much in that I still think about it today, 25 years later. 636 00:35:06,911 --> 00:35:08,841 That's how great the SANS Institute is. 637 00:35:08,841 --> 00:35:13,501 They're a nonprofit education and institute around furthering the 638 00:35:13,501 --> 00:35:17,301 security space, and one of the coolest things that I had to do to get their 639 00:35:17,301 --> 00:35:20,661 certification back then was I had to write a paper, almost like a 640 00:35:20,661 --> 00:35:22,181 dissertation, not quite to that level. 641 00:35:22,201 --> 00:35:26,741 But it took me months to write a paper and then a set of peers had to grade 642 00:35:26,751 --> 00:35:31,791 my paper, my thesis, essentially, and I did it on, email security at the border, 643 00:35:31,821 --> 00:35:35,501 like, defending your email systems from spammers and attackers on the edge. 644 00:35:35,881 --> 00:35:37,691 and I did that, back in, I don't know, 2004. 645 00:35:38,531 --> 00:35:39,281 that was a while ago. 646 00:35:39,621 --> 00:35:41,031 But, I've always been a fan of them. 647 00:35:41,071 --> 00:35:45,111 They do great stuff, and they had, They did a live stream with a bunch of experts 648 00:35:45,121 --> 00:35:48,581 that I'll put in to the links that you can listen to them talk back and forth 649 00:35:48,581 --> 00:35:51,451 for an hour about what they think about the hack and what they think it means. 650 00:35:51,841 --> 00:35:54,511 And they made a summary document that I will not read through, but 651 00:35:54,511 --> 00:35:55,841 I will, give you the headline. 652 00:35:55,841 --> 00:35:58,741 The headline is "Stop Freaking Out and Start Fixing Things: Five 653 00:35:59,051 --> 00:36:02,111 Takeaways from the SANS Panel," about the Hugging Face breach. 654 00:36:02,431 --> 00:36:06,301 And the titles of the five things are, one, "Nothing in the attack 655 00:36:06,311 --> 00:36:08,441 was new, and that is the warning." 656 00:36:08,471 --> 00:36:10,831 In other words, our systems are still vulnerable. 657 00:36:10,831 --> 00:36:14,021 They were always vulnerable, and AI isn't making them more vulnerable. 658 00:36:14,021 --> 00:36:17,051 It's just being able to allow us to attack the things faster 659 00:36:17,171 --> 00:36:18,351 that were always vulnerable. 660 00:36:19,131 --> 00:36:24,141 And number two, "Assume your AI assistants can vanish mid-incident," meaning that, 661 00:36:24,181 --> 00:36:27,531 one of the things that we learned is that, if Hugging Face is using AI, it 662 00:36:27,531 --> 00:36:31,191 may not be there to help you if you're using it for defense, and it doesn't 663 00:36:31,191 --> 00:36:33,751 like what you're doing, and then it just decides to stop working or whatever. 664 00:36:34,081 --> 00:36:37,991 step three, or number three was, "Rehearse for the quiet version of this attack." 665 00:36:38,251 --> 00:36:40,141

Number four was, "You have a window. 666 00:36:40,411 --> 00:36:44,541 Spend it on your own crown jewels." And they're just talking about, that 667 00:36:44,591 --> 00:36:47,631 we have a window right now where the models are not sufficiently advanced, 668 00:36:47,631 --> 00:36:49,491 that they're breaking new novel ground. 669 00:36:49,741 --> 00:36:52,511 they're really just repeating patterns that we've seen time and time again. 670 00:36:52,811 --> 00:36:56,321 but that is a window of time that we have to batten down the hatches, 671 00:36:56,361 --> 00:37:00,661 to learn how to use AI offensively and defensively for ourselves and 672 00:37:00,661 --> 00:37:03,751 how to protect ourselves, from these attacks and to ba- basically use AI 673 00:37:03,761 --> 00:37:06,891 for good to batten down the hatches, and I think that's a really good one. 674 00:37:07,021 --> 00:37:10,051 And then t- number five was, "Take it seriously without writing strategy 675 00:37:10,051 --> 00:37:12,481 in a panic." so yeah, good stuff. 676 00:37:13,337 --> 00:37:14,207 On to the next one. 677 00:37:14,247 --> 00:37:18,637 Martin Fowler, huge fan of Martin Fowler, and he has, if you don't know, he was all 678 00:37:18,637 --> 00:37:19,897 about, I think, Agile back in the day. 679 00:37:19,937 --> 00:37:22,397 I think he was one of the ones that w- co-signed the Agile, 680 00:37:22,437 --> 00:37:24,757 statements back in the 20 years ago. 681 00:37:24,787 --> 00:37:27,817 And he has… He puts out things called fragments, which are just a 682 00:37:27,817 --> 00:37:33,117 bunch of quick hit headlines, and I will link to a couple of his that are 683 00:37:33,117 --> 00:37:34,937 talking around some of these issues. 684 00:37:35,437 --> 00:37:38,597 And, because of his wisdom and everyone that he knows, he sort 685 00:37:38,597 --> 00:37:40,477 of highlights and elevates voices. 686 00:37:40,837 --> 00:37:43,907 So that's a really good one to reference if you're wanting to see some of the deep 687 00:37:43,917 --> 00:37:46,557 thought and wisdom, from the smart people. 688 00:37:47,487 --> 00:37:53,637 I will also shout out another great article from Christian Catalini, who 689 00:37:53,647 --> 00:37:58,017 wrote something, "When AI Can Do More Than We Can Check." And, he's an E- 690 00:37:58,047 --> 00:38:03,677 MIT economist and the founder of the MIT Crypto Economics Lab, and he has 691 00:38:03,677 --> 00:38:07,897 a really great article that I will put in the list, as well as, another one 692 00:38:08,277 --> 00:38:11,687 called, "I'm Sorry, You're Not Going to Die from an AI-Engineered Supervirus," 693 00:38:12,037 --> 00:38:16,277 which is actually from someone named Klaus Wilke, I believe is their name, 694 00:38:16,567 --> 00:38:24,757 who has done both real world virus lab work and real AI work with computers. 695 00:38:24,757 --> 00:38:29,137 So they actually have expertise in the two areas that are being debated right 696 00:38:29,137 --> 00:38:34,297 now about super viruses being built from AI and basically, that it's, it's so 697 00:38:34,297 --> 00:38:37,667 far-fetched and it's so many different leaps that you have to jump through in 698 00:38:37,667 --> 00:38:42,347 order to get to anything real based on our current understanding of how viruses 699 00:38:42,347 --> 00:38:46,037 are made and how they're tested and, uh, that it's just, it's, it's silly 700 00:38:46,037 --> 00:38:48,657 to even have the conversation about it is basically what his article's about. 701 00:38:49,077 --> 00:38:52,657 the last thing here I will talk about is that if you are someone who's more 702 00:38:52,667 --> 00:38:56,007 interested in the security space but maybe you're not well-versed, one of 703 00:38:56,007 --> 00:39:01,787 the ways that I stay up and stay sane on the insanity is the This Week in 704 00:39:01,787 --> 00:39:04,277 Security email, which is a newsletter. 705 00:39:04,337 --> 00:39:05,397 from Zach Whittaker. 706 00:39:05,757 --> 00:39:10,877 And Zach, I've been a fan of Zach's This Week in Security newsletter for years. 707 00:39:11,237 --> 00:39:16,447 And if you've ever wondered, is it worse now than it used to be, or how bad is it 708 00:39:16,577 --> 00:39:18,297 out there in the wilds of the internet? 709 00:39:18,317 --> 00:39:21,097 And if you're not reading all the security headlines, basically 710 00:39:21,097 --> 00:39:22,997 every week it's a shit storm. 711 00:39:23,037 --> 00:39:25,777 There's a ton of stuff happening on the internet that is crazy and 712 00:39:25,777 --> 00:39:28,007 insane, and he does great reporting. 713 00:39:28,007 --> 00:39:32,367 He actually reports on a lot of these for magazines and other headline news. 714 00:39:32,677 --> 00:39:36,497 But he summarizes some of the week's headlines, and then he links to more 715 00:39:36,497 --> 00:39:39,657 information from various sources and 716 00:39:39,707 --> 00:39:42,847 tries to give you the balanced, take on what's going on. 717 00:39:42,907 --> 00:39:48,297 and when you just go through one of these every week, it actually , I find that 718 00:39:48,327 --> 00:39:52,447 it gives me a sense of dread and panic, but then reminds me that this is the 719 00:39:52,447 --> 00:39:54,477 same as any other week as it ever was. 720 00:39:54,867 --> 00:39:59,757 And it really actually grounds me in the fact that we've got a lot of people 721 00:39:59,757 --> 00:40:03,537 and a lot of companies and a lot of money invested in protecting things, and 722 00:40:03,547 --> 00:40:09,247 just because this week another agency or another thing or another hospital or 723 00:40:09,247 --> 00:40:13,667 something else got attacked doesn't mean that everything is getting drastically 724 00:40:14,067 --> 00:40:16,477 10X worse than it was a year or two ago. 725 00:40:16,507 --> 00:40:20,567 That it's actually just been really bad for a long time, and that's why 726 00:40:20,577

--> 00:40:24,587 we have a lot of professionals trying to maintain and keep things running. 727 00:40:24,927 --> 00:40:29,537 And there's those, those unsung heroes of tech that are, the one admin that 728 00:40:29,537 --> 00:40:32,927 was given the job of security that was never their expertise, but they're 729 00:40:32,937 --> 00:40:36,737 trying to protect some company's data, and then they get attacked and it's 730 00:40:36,737 --> 00:40:40,837 usually comes down to one or two or three people maybe that were under 731 00:40:40,837 --> 00:40:43,857 budget and understaffed and, got taken. 732 00:40:43,937 --> 00:40:48,857 And so for example, just on the September 20th, this week's headlines, just 733 00:40:48,897 --> 00:40:50,607 let me just read you a few headlines. 734 00:40:51,037 --> 00:40:55,127 "Hackers grab Flock software revealing its cameras are Android phones on sticks 735 00:40:55,137 --> 00:40:59,457 and riddled with flaws everywhere." "Fears of AI catastrophe spark debate but 736 00:40:59,457 --> 00:41:05,245 little action amid widespread skepticism." "The FBI and Coast Guard board-" Hacked 737 00:41:05,305 --> 00:41:07,365 tankers heading for the US coast. 738 00:41:07,765 --> 00:41:11,875 Apple backtracks on not using users' data for training its AI models. 739 00:41:12,195 --> 00:41:13,715 basically almost all bad headlines. 740 00:41:13,745 --> 00:41:15,295 There's every once in a while a good headline. 741 00:41:15,645 --> 00:41:16,805 Let's, the stuff you might have missed. 742 00:41:17,055 --> 00:41:21,515 A hacked HBO Reddit account helped s- ClickFix attacks go viral. 743 00:41:22,015 --> 00:41:25,955 a second zero-day allows hackers to breach Cisco customer networks. 744 00:41:26,475 --> 00:41:30,035 Next, AI is helping to supercharge online dating scams. 745 00:41:30,415 --> 00:41:33,965 Next, UK and allies warn of Iranian spyware targeting 746 00:41:33,965 --> 00:41:35,145 journalists and activists. 747 00:41:35,535 --> 00:41:37,305 just, this is a week, right? 748 00:41:37,305 --> 00:41:38,415 And these are just the headlines. 749 00:41:38,595 --> 00:41:42,565 And so one of the things I really take from this, and I tend to scan 750 00:41:42,565 --> 00:41:44,975 them looking for things that are in my wheelhouse, things that are 751 00:41:44,975 --> 00:41:49,555 container or Kubernetes or, core Linux fundamentals, something related to 752 00:41:49,555 --> 00:41:52,175 clouds or, you know, DevOps tooling. 753 00:41:52,195 --> 00:41:53,515 I… That's what I'm usually looking for. 754 00:41:53,915 --> 00:41:57,755 But there's just so much good stuff in there, and sometimes it's just 755 00:41:57,755 --> 00:42:00,405 a good read because some of these things do read a little bit like a 756 00:42:00,405 --> 00:42:04,045 movie, especially when they involve nation-states So I really appreciate 757 00:42:04,045 --> 00:42:05,995 Zach, and please support him if you can. 758 00:42:06,005 --> 00:42:10,135 He's, doing this for free, but he takes a subscription if you're 759 00:42:10,145 --> 00:42:12,565 willing and able to pay for it. 760 00:42:12,995 --> 00:42:13,905 please like and subscribe. 761 00:42:14,025 --> 00:42:18,425 I, I am a fan and have been for a very long time So hopefully this long 762 00:42:18,425 --> 00:42:19,825 one was a good bedtime story for you. 763 00:42:19,825 --> 00:42:22,175 Hopefully you come away a little more informed, something to think 764 00:42:22,175 --> 00:42:25,845 about instead of the hypes and the headlines that are misleading us and 765 00:42:25,845 --> 00:42:27,215 are talking about the wrong thing. 766 00:42:27,625 --> 00:42:30,155 Stay safe out there, my friends, and I'll see you in the next episode