สำเนาบทสนทนา 🔗
WEBVTT
00:00:13.599 --> 00:00:17.679
Hello and welcome to another episode of the Cables to Clouds podcast.
00:00:17.679 --> 00:00:25.519
I'm your uh co-host, uh Tim McConaughey, and with me is Catherine McNamara, my other co-hosts.
00:00:25.519 --> 00:00:32.000
Uh Chris couldn't be here today because uh I don't know, maybe he had uh cleanup after his cat or something.
00:00:32.000 --> 00:00:32.719
I don't know.
00:00:32.719 --> 00:00:37.600
But uh no, it's pretty late for Chris, so we let him sleep in tonight.
00:00:37.600 --> 00:00:44.719
Uh and with us is a very special guest, someone we have not had on the podcast before, but really excited to finally get on here.
00:00:44.719 --> 00:00:46.079
Uh Zach Corman.
00:00:46.079 --> 00:00:47.520
Uh Zach, go ahead.
00:00:47.520 --> 00:00:49.759
Just uh introduce yourself for the audience.
00:00:50.159 --> 00:00:50.719
Yeah, great.
00:00:50.719 --> 00:00:52.000
Uh so thanks for having me.
00:00:52.000 --> 00:00:53.280
Uh so I'm Zach.
00:00:53.280 --> 00:01:01.439
I uh mostly just post on the internet about AI agent security issues and uh get into fights there about that topic.
00:01:01.439 --> 00:01:14.799
Um I also I've led tech at um so I led I was the CTO of a cybersecurity startup before this for about four years, um, doing uh basically AI and insider threat detection.
00:01:14.799 --> 00:01:19.920
Uh I've also just led tech teams at a couple different places, and now I'm at my own startup.
00:01:19.920 --> 00:01:21.359
Uh, we're called Embroidery.
00:01:21.359 --> 00:01:24.159
And so we've not launched yet, so I've like nothing to pitch.
00:01:24.159 --> 00:01:25.359
So yeah.
00:01:25.359 --> 00:01:27.280
But yeah, that's what I would do.
00:01:27.280 --> 00:01:30.400
Mostly post on the internet and get into fights about AI.
00:01:30.719 --> 00:01:40.959
But to be fair, the fights you do get into, you end up technically being right on a technical level and cause whoever you're fighting with to retreat pretty quickly.
00:01:40.959 --> 00:01:48.319
And that's what I think is actually pretty fascinating is like there's a lot of marketing and hype around AI and security for AI right now.
00:01:48.319 --> 00:01:56.560
And I think that every vendor and a lot of startups are rushing to like put something out to get that hot VC money.
00:01:56.560 --> 00:02:01.359
But unfortunately, um, I don't think it's a lot of it is living up to the promise.
00:02:01.680 --> 00:02:01.920
Right.
00:02:01.920 --> 00:02:02.239
Yeah.
00:02:02.239 --> 00:02:10.479
So I mean, it kind of was became sort of my it became very common when open claw took off uh and became very popular.
00:02:10.479 --> 00:02:18.879
There was this really cool pitch you could make, which is to make you would make some other, you know, open source code and say it's open claw, but it's safe.
00:02:18.879 --> 00:02:27.520
And then you could just, you could just say those words, they were the magic words, and then you would get like 5,000 likes and a
Load full transcript
bunch of new customers.
00:02:27.520 --> 00:02:31.120
And um it turned and and it was just completely false.
00:02:31.120 --> 00:02:46.800
I mean, by and large, all of these companies that were trying to make safe open claw had not engaged at all with the uh actual challenges of doing that and like why fundamentally you might not be able to achieve that goal the way you think you would.
00:02:46.800 --> 00:02:50.960
Um, you know, and that comes down to even like uh Nvidia or Nvidia.
00:02:50.960 --> 00:02:58.000
They had they made a thing called Nemo Claw, which was like they were like pitching it as enterprise grade security, open claw.
00:02:58.000 --> 00:02:59.840
I mean, I think they still are for that matter.
00:02:59.840 --> 00:03:10.400
Um and it that thing I just I, you know, basically I made it open itself up to the internet so anyone could control your agent uh from any website you visit.
00:03:10.400 --> 00:03:12.479
And it's like that's not enterprise grade security.
00:03:12.479 --> 00:03:19.039
But they get away with those claims because no one really has it's like very few people have the ability to really look into it.
00:03:19.039 --> 00:03:24.000
So it's like they're pitching to executives to try and go like, oh, now we can do open clause safe.
00:03:24.000 --> 00:03:27.759
And so I was kind of saw it as like my personal mission at 10 p.m.
00:03:27.759 --> 00:03:33.280
at night every night to just go find ways to break it and you know, prove that it was not good.
00:03:33.280 --> 00:03:34.960
So yeah, yeah.
00:03:34.960 --> 00:03:37.199
So as you may have surmised, oh go ahead, go ahead.
00:03:37.599 --> 00:03:43.759
I was gonna say, uh, if I recall correctly, it took you about five minutes to break out of Nvidia's little uh little sandbox.
00:03:44.319 --> 00:04:11.599
Yeah, the the the best one actually was when they they fixed so what happened, the full story there was I broke it one time where I opened it up for the whole world, and then I posted about it, and then the NVIDIA person in charge of that project, who's like a director at NVIDIA, like basically responded to me because I just posted that on the internet because from my perspective, this isn't like a oh, I need to go disclose it to them so they can because like they they've chosen to make a security claim.
00:04:11.599 --> 00:04:15.439
Right if it's false, I want to make sure people know it's false, right?
00:04:15.439 --> 00:04:18.079
It's not like I think they put their best effort in.
00:04:18.079 --> 00:04:20.000
And so I just posted about it.
00:04:20.000 --> 00:04:23.920
And then the director came and said, this is by design and intentional.
00:04:23.920 --> 00:04:28.639
So then I reposted it with their marketing that says enterprise grade security.
00:04:28.639 --> 00:04:36.240
And I basically said something like, well, like if you know, like I don't know who would have thought that that was intentional because this is what you say.
00:04:36.240 --> 00:04:39.120
And then what was funny is they went and fixed it actually.
00:04:39.120 --> 00:04:50.560
And then it took me, I think once they fixed it, it took me like it had to have been like one minute to break it again because it was like they fixed it in the dumbest possible way.
00:04:50.560 --> 00:04:52.319
They basically just like yeah.
00:04:52.319 --> 00:04:55.279
So what happened specifically is they have this config that controls everything.
00:04:55.279 --> 00:05:03.199
And what they did is they made it so you couldn't write that config, but you can just write a different file and then restart the thing automatically and point to the new config.
00:05:03.199 --> 00:05:07.519
So you could just go like, haha, I'll just instead of changing the config, I'll make a new config.
00:05:07.519 --> 00:05:08.959
And then that worked still.
00:05:08.959 --> 00:05:14.720
So and I don't even know if they patched that second one because the the guy, of course, I think he like blocked me or something.
00:05:14.720 --> 00:05:20.720
Uh so I don't even know if he would have seen my second round of you know, as someone might have.
00:05:20.879 --> 00:05:28.000
But so as you may have guessed, if you for the listeners, as you may have guessed, this episode is going to be about security for AI.
00:05:28.000 --> 00:05:35.920
If that wasn't clear enough from the guests that we brought on, if you haven't uh seen them before, uh, this this episode's all about security for AI.
00:05:35.920 --> 00:05:46.959
And uh just real quick, because you said you said something that's kind of interesting that I think you and I'm not a cybersecurity guy, but I am familiar at least with the idea of kind of uh disclosure, right?
00:05:46.959 --> 00:05:51.360
Like if you find a flaw, how generally how the disclosure process works.
00:05:51.360 --> 00:06:06.720
So you're saying basically because they said, hey, this is secure, and you like pretty much proved that it wasn't secure, that you didn't feel the necessity or to go through like a private disclosure process that might have otherwise, you know, under normal circumstances been the case, right?
00:06:07.120 --> 00:06:11.040
Yeah, well, it's my it's disclosure is a complicated topic, right?
00:06:11.040 --> 00:06:21.279
And I think like from my perspective though, if you launch a brand new product, so like no one's using Nemo Claw at the moment when I find this.
00:06:21.279 --> 00:06:25.920
It's like I looked at it on like day three or something, and you launch this new product.
00:06:25.920 --> 00:06:31.120
The like kind of best thing for security is to kind of say, hey, this isn't actually safe.
00:06:31.120 --> 00:06:32.319
Please don't use it.
00:06:32.319 --> 00:06:47.360
As opposed to, I guess if I found that Nemo Claw was used by hundreds of thousands of orgs for critical workloads, maybe I would not have dropped it as publicly because of course then you might actually have some pretty negative effects from doing it.
00:06:47.360 --> 00:06:51.120
Um, but if they're like they put it out in the public, they say it's enterprise grade security.
00:06:51.120 --> 00:07:01.920
My number one goal is to ensure that people who might believe that do not go, you you don't want a bunch of organizations going and turning on Nemo Claw and then going like, oh, it was safe, right?
00:07:01.920 --> 00:07:02.959
That would be bad.
00:07:02.959 --> 00:07:07.279
And so I think like it's a complicated trade-off that like you have to be careful with.
00:07:07.279 --> 00:07:12.639
But like in this case, it felt pretty obvious to me that the primary goal is to tell people don't do this.
00:07:12.639 --> 00:07:15.839
Um, you know, that's kind of my position on it, at least.
00:07:15.839 --> 00:07:19.680
I I don't know anyone who would have gotten hacked as a result, you know.
00:07:20.319 --> 00:07:21.120
Yeah, that's fair.
00:07:21.120 --> 00:07:24.560
And I think that NVIDIA also advertised it as this open source project.
00:07:24.560 --> 00:07:29.600
So if you had like submitted a request, it would have been public anyways, uh, right on the GitHub page.
00:07:29.600 --> 00:07:39.120
So I mean it it's not like it's not like you were reporting on this like you know, well-baked, you know, enterprise closed source project.
00:07:39.120 --> 00:07:47.759
You were reporting on something that if you had submitted a request or like a you know, some sort of like feedback, it would have been seen by everybody.
00:07:48.720 --> 00:07:49.680
Right, exactly.
00:07:49.680 --> 00:07:52.480
And I think this also follows a lot of my other work.
00:07:52.480 --> 00:07:55.839
I I mostly just, you know, post it depends on what it is.
00:07:55.839 --> 00:08:08.959
Mostly I'll just post about it online because uh other ones that I've done are things that like I said, actually, NVIDIA wasn't even gonna, the director didn't believe this to be a bug until, of course, I mocked him into everyone.
00:08:08.959 --> 00:08:11.040
Like I got like, you know, 600 likes on it.
00:08:11.040 --> 00:08:12.959
He's like, okay, maybe it was a bad idea.
00:08:12.959 --> 00:08:17.839
Um, and it's similar that I know there are things, especially around like so.
00:08:17.839 --> 00:08:32.240
I've done a bunch of work on AI agent skills, which are markdown files that are instructions for your agents, um, and how you can use those to basically attack the, you know, how a threat actor could use skills for malicious purposes.
00:08:32.240 --> 00:08:44.240
And one reason I just post about those is because like anthropic's not going to fix it, or like whoever, which you know, the whole ecosystem's not gonna build in protections against this inside of their, you know, system.
00:08:44.240 --> 00:08:52.240
It's more like this is an inherent property of this thing you've all built into your you know, AI agent frameworks.
00:08:52.240 --> 00:09:00.240
Um, I need to make people aware of that so that they don't take these risks or so that they have like they can build their own protections in place.
00:09:00.240 --> 00:09:04.399
Uh, many of the things that I've reported around skills like are still true today.
00:09:04.399 --> 00:09:12.240
You can still use uh you can still put hooks inside of skills so that as soon as you, you know, run a skill, it will just, it's basically remote code execution.
00:09:12.240 --> 00:09:15.039
You can literally just literally run any code you want.
00:09:15.039 --> 00:09:19.759
Uh, you can hide commands inside of skills, inside of comments.
00:09:19.759 --> 00:09:24.799
You can auto-execute skills using, you know, all of these ways to trick people.
00:09:24.799 --> 00:09:28.159
You can put hidden commands inside of alt text for images.
00:09:28.159 --> 00:09:29.679
None of this is fixed.
00:09:29.679 --> 00:09:30.159
Okay.
00:09:30.159 --> 00:09:37.840
And so it's like you you you want to get this information out there so people are very aware of why they should be careful.
00:09:37.840 --> 00:09:38.480
Right.
00:09:39.519 --> 00:09:57.279
I was gonna say, uh, because of the work you've done, I've actually been able to point people like uh like both at Cisco Live and other places to be like, hey, before you go and try to like start using open claw for like deployment, like for like actual production stuff, here's what you need to be aware of.
00:09:57.279 --> 00:10:06.000
Um you know, for the listeners, um, skills are things that you can download for your like open claw instance or uh, you know, or whatever agent you're using.
00:10:06.000 --> 00:10:08.960
Uh Vercell has their own catalog and others.
00:10:08.960 --> 00:10:15.759
And when these things first were starting to be developed, was it like late last year, early this year, somewhere around that time?
00:10:15.759 --> 00:10:21.840
Um you know, Zach made a really good point of showing how easy it was to embed.
00:10:21.840 --> 00:10:31.200
Like he didn't put anything that actually could harm anyone, but he put like a payload in there that would look like, you know, that didn't deliver the skill it it it should have delivered.
00:10:31.200 --> 00:10:35.919
Uh and he was able to manipulate the vote or the download to make it look like a popular uh skill.
00:10:35.919 --> 00:10:41.440
So it was in the top 10 skills, and people would just look at the popularity of it and be able to do it.
00:10:41.519 --> 00:11:01.200
And he kept doing this no matter like Vercel and the the other catalog, they were I forgot which one it was, but uh those are situations that like to anthropic was that primarily was Vercell has this uh skills.sh, which is this basically, and then they have this package called skills uh skills via npm.
00:11:01.200 --> 00:11:03.519
So you can do like MPX skills add.
00:11:03.519 --> 00:11:03.840
Right.
00:11:03.840 --> 00:11:05.279
And it is super dangerous.
00:11:05.279 --> 00:11:14.000
And so yeah, so Catherine is like that's kind of the point is like it's helpful to be
able to tell people like, here's why you should be careful and have evidence of it.
00:11:14.000 --> 00:11:16.639
Cause otherwise, like, and so that was kind of my goal with it.
00:11:16.639 --> 00:11:20.320
And yeah, I broke them like they they would go, like, oh, that's not a big deal.
00:11:20.320 --> 00:11:21.519
And I go, what about this one?
00:11:21.519 --> 00:11:25.919
And they're like, what about I I probably spent a month just coming up with a new one every day.
00:11:25.919 --> 00:11:26.559
Yeah.
00:11:26.879 --> 00:11:43.600
And yeah, I think your family missed you for that whole month because you were just like um I was gonna say also, like, you know, going on that trend where people want to get PR and like uh and and and a lot of attention on the on these like AI for uh security for AI hypers.
00:11:43.600 --> 00:12:00.159
Like I know that one of the one of the things that was added on to the skills was a virus total check, which really didn't do what like but virus total and and Google got a lot of attention from it, but in reality, it really wasn't very like effective to prove that a skill was not malicious.
00:12:00.159 --> 00:12:04.720
But they added that check there saying, like, hey, look, virus total says this is clean or good.
00:12:04.720 --> 00:12:08.159
And Zach, I think, like took one day to get around that.
00:12:08.559 --> 00:12:08.799
Yeah.
00:12:08.799 --> 00:12:09.200
Yeah.
00:12:09.200 --> 00:12:11.679
In fact, it was funny because they forgot to add it.
00:12:11.679 --> 00:12:15.919
They made a whole PR play about how this was protecting the skills marketplace.
00:12:15.919 --> 00:12:19.440
And the funniest part was I went to try to like break it.
00:12:19.440 --> 00:12:22.240
And the first thing that happened is I didn't even have to break it.
00:12:22.240 --> 00:12:23.919
They had forgotten to turn it on.
00:12:23.919 --> 00:12:36.159
So, so even though they had run their PR campaign about how this was saving everyone, I tried to download one of my known malicious skills that just straight up was like a remote code execution and it just downloaded.
00:12:36.159 --> 00:12:42.879
And then there's because what had they just forgot to make to add it to the you know, code to basically actually use it.
00:12:42.879 --> 00:12:51.919
So the yeah, so they didn't, and that's it's so and so indicative of the era of AI security that like all they cared about was the PR.
00:12:51.919 --> 00:12:56.159
No one, no one tested whether that worked.
00:12:56.159 --> 00:13:04.399
And because it only took me to run MPX skills ad on a malicious skill to find that it didn't work.
00:13:04.399 --> 00:13:17.360
And so it was like you guys didn't even try, you didn't care, and yet there have been multiple people coming after, and there have been all these things like what some of these skill scanners you can bypass them trivially.
00:13:17.360 --> 00:13:27.120
I mean, like you can you can just name them, you know, Vercell skill and it will pass because they've hard-coded that anything called Vercell gets passed, right?
00:13:27.120 --> 00:13:29.440
You know, like they're not safe.
00:13:29.840 --> 00:13:31.600
That's that's terrible.
00:13:31.840 --> 00:13:32.080
Yeah.
00:13:32.320 --> 00:13:33.519
Was it yeah, yeah.
00:13:33.519 --> 00:13:35.279
Well, sorry, was it you, Zach?
00:13:35.279 --> 00:13:36.480
Or I can't remember who it was.
00:13:36.480 --> 00:13:56.399
It might have been you that uh either posted or showed that uh one of the ways to get around the the malicious scanning for like scanning for malicious skills was to put uh instructions in the skill that the AI would automatically like the guardrails would pop up and it would skip like I didn't do that one, but I I saw that one.
00:13:56.480 --> 00:13:57.279
I saw another one that was.
00:13:58.879 --> 00:14:02.320
So basically in direct prompt injection via like skill download.
00:14:02.559 --> 00:14:10.320
And there they're all also was one that was just if you put enough that because while they're doing these skill scanners, they're doing this, they're just passing them to AI to check.
00:14:10.320 --> 00:14:17.039
And so if you make your skill have more than a million tokens and then put the malicious part blow away the context window, yeah.
00:14:17.039 --> 00:14:23.039
You you've you the that it won't find your malicious part because the AI scanner only looks at the first million tokens.
00:14:23.039 --> 00:14:23.440
Okay.
00:14:23.440 --> 00:14:25.200
So you can just right.
00:14:25.200 --> 00:14:26.799
There's plenty of tricks like that.
00:14:26.799 --> 00:14:32.639
The other things were like to get by it is like, yeah, there's Museum, you can trivially bypass these.
00:14:32.639 --> 00:14:42.799
And and my favorite one was my favorite one is I had this one called uh security review, which gets marked as malicious because it has a clear remote code execution.
00:14:42.799 --> 00:14:48.960
And what I did is I just made a different one, which just says go download the first one and then run it.
00:14:48.960 --> 00:14:50.480
And then it's like, okay, that's safe.
00:14:50.480 --> 00:15:02.320
Because like it's just like because they don't they don't follow all of the URLs, like, you know, so it's like there's nothing malicious about it's like sandboxing or like I can make a skill called download malware and it will download malware.
00:15:02.320 --> 00:15:04.559
Like um, yeah.
00:15:05.360 --> 00:15:06.799
So I I have a question for you.
00:15:06.799 --> 00:15:32.720
Like I I know you saw the announcement with Microsoft saying, like, you know, now you know OpenClaw running like enterprise safe, like uh on uh on Windows, like uh, you know, obviously Microsoft's a little late to the game to be making those security claims, but um, on top of that, like like as a security professional, how is that like just watching that happen and uh uh how's that what do you think of it?
00:15:33.120 --> 00:15:36.399
I I was
just I was just mad because I don't have a Windows computer.
00:15:36.399 --> 00:15:41.360
So I was like, uh how do I go find one that has the right and I was like, is this live yet?
00:15:41.360 --> 00:15:42.639
It was a bit unclear to me.
00:15:42.639 --> 00:15:47.519
So but like I'm gonna go find a Windows computer because I'm gonna just destroy this thing.
00:15:47.519 --> 00:15:57.919
Because one of the problems with this idea of a secure open claw is like it's not people seem to think that open claw is insecure because it has like code vulnerabilities.
00:15:57.919 --> 00:15:59.600
Like that's well, that's also true.
00:15:59.600 --> 00:16:00.399
Don't get me wrong.
00:16:00.399 --> 00:16:21.759
Like it they also have bad code, but even if you fixed all the code, and even if you've there are like conceptual problems with what you're doing with open claw, which is to say you're basically allowing an AI agent to act across a very large pool of your life and you know, privileged systems, and to also act on external content.
00:16:21.759 --> 00:16:27.919
Uh, and that is the main security risk, is just the ability to influence agents in this way.
00:16:27.919 --> 00:16:32.559
And so to to, you know, you know, I can trick an agent from the outside to do anything I want.
00:16:32.559 --> 00:16:37.519
And if it also happens to already have access to your SharePoint, like you're in trouble.
00:16:37.519 --> 00:16:49.279
Um, and so there's no such thing as like a secure open claw, except like if you just completely got its access, so you go like it cannot do X, it cannot do Y, it cannot do Z.
00:16:49.279 --> 00:16:51.200
And then it's no longer open claw.
00:16:51.200 --> 00:16:55.600
Now it's just like a really dumb, you know, AI learn.
00:16:55.600 --> 00:16:58.320
It's it's it's yeah, it's it's Microsoft Copilot, right?
00:16:58.320 --> 00:17:02.879
Which is like their greatest security feature of Copilot is that it's doesn't really work.
00:17:02.879 --> 00:17:06.079
And so as a result, like you can't really hack it either, right?
00:17:06.079 --> 00:17:14.400
It's like that's basically what I imagine they've done, but I haven't tried it yet because I don't even know if it's out yet for the public or if it is, like where I can, yeah.
00:17:14.400 --> 00:17:17.119
Otherwise, once it's out, I'm gonna figure this out.
00:17:17.440 --> 00:17:24.400
Uh yeah, I think it was like announced three weeks ago, but I mean, you could just spit up like a Windows VM worst case scenario and try to play with it there.
00:17:24.400 --> 00:17:27.519
But uh yeah, I'm I'm curious to see what your feedback is.
00:17:27.519 --> 00:17:34.079
But you know, like going like circling back though, like it's not just open claw, like there's been other vendors that have made claims and things like that.
00:17:34.079 --> 00:17:38.559
And then it kind of reminds me of the meme I think I actually sent you earlier this week.
00:17:38.559 --> 00:17:47.039
Like, um, for folks, like
so one half of the meme is a meme is a very clean-cut gentle gentleman being like uh AI for security.
00:17:47.039 --> 00:17:52.960
And then on the other side is a guy with just a scraggly beard looking tired saying security for AI.
00:17:52.960 --> 00:18:04.400
I I think that the important thing to kind of take away from this is that AI for security can be very helpful, but we are still so very behind on security for AI.
00:18:04.400 --> 00:18:12.640
And like we have every major vendor, every uh, you know, a thousand startups promising like security for AI.
00:18:12.640 --> 00:18:19.359
And I I still think we're falling short as a whole, as a you know, industry there still.
00:18:22.160 --> 00:18:22.480
Yeah.
00:18:22.480 --> 00:18:26.880
Um I can so I haven't can I just jump in here or Tim, do you want to say something?
00:18:26.880 --> 00:18:27.519
Oh yeah.
00:18:27.519 --> 00:18:32.160
So I was gonna say, I because I come from an AI for security world, right?
00:18:32.160 --> 00:18:38.559
I built like AI for insider threat detection, which is a very much a normal like cyber, it's not security agents, right?
00:18:38.559 --> 00:18:39.920
Or not AI agents.
00:18:39.920 --> 00:18:44.319
And then obviously I've gotten deep into AI security.
00:18:44.319 --> 00:18:48.799
And so that meme is perfect because I'm just like, I'm dying, right?
00:18:48.799 --> 00:18:50.559
Like I am, it is killing me.
00:18:50.559 --> 00:18:53.680
Every day there's some new crazy thing that happens.
00:18:53.680 --> 00:18:57.200
Every day I'm like, oh, I just it's it hurts.
00:18:57.200 --> 00:19:11.200
And you know, the and then that's the thing is that a lot of the the platforms for AI security um are basically just platforms for doing some AI thing, and then you just say the word security.
00:19:11.200 --> 00:19:20.240
So, like if you want to like triple your revenue as a startup that builds like an AI orchestration platform, you just call it a safe AI orchestration platform.
00:19:20.240 --> 00:19:23.279
You don't you don't need to change anything, you don't need to make it secure.
00:19:23.279 --> 00:19:30.400
You just say the word because then it's like that no one really has a good conception of what AI security means.
00:19:30.400 --> 00:19:35.920
And then you have these groups that are going along trying to like define AI security.
00:19:35.920 --> 00:19:39.440
Um, and they're they have their own ideas of what it means.
00:19:39.440 --> 00:19:43.839
And so you have like compliant people making compliance frameworks out of it.
00:19:43.839 --> 00:19:49.119
Um, AIUC1 would be one of these uh that are just like, what are we doing?
00:19:49.119 --> 00:19:50.480
We don't have good solutions.
00:19:50.480 --> 00:19:56.319
Like in many ways, we are still in this early stage of trying to figure out what good AI agent security looks like.
00:19:56.319 --> 00:20:04.240
So anyone who comes along trying to promise you that they've solved it and they can even put it into a
compliance checkbox, you know, series of items that you just need to tick off.
00:20:04.240 --> 00:20:05.440
I mean, that's just a lie.
00:20:05.440 --> 00:20:07.279
That's just not how it works.
00:20:07.759 --> 00:20:12.799
I would not recommend you going to RSA ever, or else you're gonna pull out your hair.
00:20:13.200 --> 00:20:14.400
Yeah, I've I've heard.
00:20:14.400 --> 00:20:16.880
I've heard it would not be fun for me.
00:20:17.119 --> 00:20:23.680
Um but that's like five 500 booths of everyone promising uh uh uh security for AI.
00:20:24.079 --> 00:20:24.400
Yeah.
00:20:24.400 --> 00:20:34.319
And and and the thing is that's hard about it is especially because now like I work more in this space, you know, what I'm doing is so narrowly defined.
00:20:34.319 --> 00:20:37.039
I try to do AI agent threat detection, basically.
00:20:37.039 --> 00:20:46.640
And and then I have companies I talk to that are like, yeah, but like this company does all that they claim to solve AI agent security as a whole.
00:20:46.640 --> 00:20:57.519
And imagine if like a company claims, I mean, I guess some probably do now, but historically, you don't typically have companies that go like we are a company, you turn us on, and we solved all of cybersecurity.
00:20:57.519 --> 00:21:06.319
Uh but that's effectively the pitch being made by some of these AI agent security platforms that like turn us on and now all of AI agent security is safe.
00:21:06.319 --> 00:21:08.400
And like you're just you're just fooling yourself.
00:21:08.400 --> 00:21:16.079
And that's really unfair because some of these orgs, they're just trying to like, you know, like they're not necessarily the most technical orgs in the world, right?
00:21:16.079 --> 00:21:20.000
Sometimes they're just like normal businesses, want to start using AI.
00:21:20.000 --> 00:21:25.200
They had a good sales pitch from someone saying that they can do it safely, and then they're gonna get hacked.
00:21:25.200 --> 00:21:34.079
And it's like it it's disgusting that some of these firms are willing to tell that lie in order to close deals for themselves.
00:21:34.960 --> 00:21:40.160
It very much feels like the beginning, like this way more on steroids now, though.
00:21:40.160 --> 00:21:52.240
But like back when like zero trust became the hot like item to talk about, everybody, like every vendor was like learned those words and decided I have this black box that gives you zero trust.
00:21:52.240 --> 00:21:55.599
It's a firewall, it's an EDR, it's a this, it's that.
00:21:55.599 --> 00:22:06.559
And um it in reality, like as time went on, we really That like it's not really uh it's not that big of you know, there's no magic box that gives you zero trust.
00:22:06.559 --> 00:22:13.920
And on top of that, like zero trust is supposed to be an architecture and it's not kind of doing everything that was promised to do to do.
00:22:13.920 --> 00:22:24.160
Security for AI, though, is like seems a thousand times worse
because there's not as many people like that have your kind of expertise that are able to say, this is BS and here's you know the proof.
00:22:24.160 --> 00:22:28.880
There it's just kind of like a lot of this stuff is kind of glossed over and people just take it at face value.
00:22:29.359 --> 00:22:41.440
So for for and again, this I'm not as into the cybersecurity stuff as you guys are, but I've this actually this whole thing reminds me of you know when I was much younger and when personal computers and everything were being taken off.
00:22:41.440 --> 00:22:52.240
It was before the internet now, and you know, the way you got a virus on your computer was somebody would give you like uh, hey, here's Doom on uh you know 3.5 floppy or something.
00:22:52.240 --> 00:22:56.480
And you know, the the EXE was switched with like a Trojan or something like that, right?
00:22:56.480 --> 00:23:22.319
And so you not knowing any better and not being computer savvy, or there's no such thing as cybersecurity really at that time, you just you know plug it in your computer and run your Doom and you didn't spect your the point I'm making is that like agents and like a gentic security, all of this is like AI agents feel like, you know, essentially like uh like a like a kid or like a junior computer person that'll just believe anything anybody tells them.
00:23:22.319 --> 00:23:28.799
And like skills feel like, you know, hey, you're installing you're installing Trojans on your computer and you don't know any better.
00:23:28.799 --> 00:23:35.599
And like it's uh it's interesting how what is what's old is new again and just kind of wrapped in a different, at least from my perspective.
00:23:36.480 --> 00:23:43.920
Yeah, and and one of the things that's interesting is because people say things like, yeah, but you have a bunch of employees who dumb do dumb security stuff too.
00:23:43.920 --> 00:23:57.440
I'm like, yeah, but we've we've kind of in large part, like, okay, at my last company, it's not like I gave the salespeople very much ability to uh, you know, the account executives could go ahead and download malware if they wanted to.
00:23:57.440 --> 00:24:00.079
Like all they're doing is hurting their own computer, right?
00:24:00.079 --> 00:24:04.240
Like they don't have much access broadly to our systems, you know.
00:24:04.240 --> 00:24:09.680
And then so if you want to compare it to employees, then that's fine.
00:24:09.680 --> 00:24:14.559
But then you have to treat a you have to give agents the same restrictions, right?
00:24:14.559 --> 00:24:27.440
Whereas what's happening today is you say, like, our senior developer, who probably has quite a lot of permissions, is also allowed to tell this AI agent to pretend to be him and do everything he can do.
00:24:27.440 --> 00:24:33.200
And then that that agent is gonna just download Doom, you know, malware Doom, right?
00:24:33.200 --> 00:24:36.400
Because it's like there they don't have and and that's a big part.
00:24:36.400 --> 00:24:39.920
They also don't fail in the same ways that people might expect, right?
00:24:39.920 --> 00:24:47.519
It's like the things that might trick a human might not trick an AI agent, but things that would never trick a human could totally trick an AI agent.
00:24:47.519 --> 00:24:55.119
I mean, hiding an a remote code execution inside of a HTML comment is not going to trick a human because either they'll never see it.
00:24:55.119 --> 00:25:01.039
And if they do, they'll be like, well, obviously you're doing something dangerous, you're hiding information, like you're trying.
00:25:01.039 --> 00:25:09.119
A lot of AI agents will just be like, I guess that's how he wanted to communicate that information best to me, was hidden in the comment, right?
00:25:09.119 --> 00:25:17.119
So, you know, they fail in these unexpected ways, and that makes it like hard to give intuition to people about how they need to approach it.
00:25:17.119 --> 00:25:25.680
But fortunately, I think the security community has been better than I was expecting them to be at being like, that's bad, right?
00:25:25.680 --> 00:25:39.680
So I think a lot of people in the cybersecurity community of like people who are in in security have got have have gotten a decent grasp of like AI agent security, um, not necessarily as an expert level, but enough to be skeptical.
00:25:39.680 --> 00:25:47.519
Uh, where I was worried that they would just kind of sit it out and that this would become an issue for CTOs and developers who would who don't care, right?
00:25:47.519 --> 00:25:52.720
Um, so I've been kind of happy that at least security people seem engaged in the discussion now.
00:25:53.519 --> 00:25:55.119
Yeah, I I would agree with you there.
00:25:55.119 --> 00:26:02.640
They uh the general security, uh, like it's security researchers, InfoSec folks are skeptical, and rightly so.
00:26:02.640 --> 00:26:21.279
It's it's an it's been interesting watching like the AI, uh, I don't know, like AI, I call them AI bros, AI marketing folks butting heads publicly with like the security folks, like you're holding us back, but the security folks being like, yeah, but if you're like you said, you're if your random employee gets hacked, that affects his computer.
00:26:21.279 --> 00:26:28.160
If OpenClaw gets hacked or your AI agent gets hacked, this has API keys for everything in your production environment, potentially.
00:26:28.160 --> 00:26:34.160
And those are things that like uh, you know, security folks are like waving the the red flags over.
00:26:34.160 --> 00:26:47.440
And, you know, unfortunately, we're, you know, people, yeah, we have a set of we have in the tech industry, we have a set of acceler uh accelerists, and then we have a set of folks trying to like be like, hey, caution, let's let's go proceed carefully first.
00:26:47.440 --> 00:26:57.839
And they're right now they're at odds, and you're like, you know, yeah, what's interesting about you and and
and a lot of stuff work you've been doing, like you've obviously butted heads with some of those folks.
00:26:57.839 --> 00:26:59.359
And you you're always very polite.
00:26:59.359 --> 00:27:18.720
Like I like I just want to pre- preface here, like uh for folks that like follow Zach on or will follow Zach on like Twitter, he is always very polite about it, but he's very firm and assertive, like, no, that this this is you know, here, this is the version that you were advertising, and it's still showing this malicious NPM skill as you know, clean.
00:27:18.720 --> 00:27:29.200
Like he's very thorough and very technical in his explanations, which I I think the more marketing AI folks don't tend to like, but security folks definitely respect that.
00:27:29.839 --> 00:27:32.000
Well, you bring and he brings receipts, right?
00:27:32.000 --> 00:27:38.240
And receipts are the biggest it's hard to argue with with with proof, you know, especially from a security perspective.
00:27:38.720 --> 00:27:42.480
Yeah, and I I try to do this where I I spend a lot of time on on these.
00:27:42.480 --> 00:27:43.119
Yeah.
00:27:43.119 --> 00:27:44.319
Sorry, go on, Catherine.
00:27:44.640 --> 00:28:15.519
I was gonna uh also say, like, like as a security practitioner and stuff like that, like there's probably gonna be a mix of folks that are gonna be watching this episode, like network folks, application folks, uh cloud, uh, cloud engineers, like what would if i it for the folks that are not in InfoSec, what would you your advice be to them who like on how to you know be cautious about these projects that are being hoisted onto them, how to like properly vet the tools and the security tools that they're using?
00:28:15.519 --> 00:28:23.359
Like, what kind of advice would you give to non-security folks who are open-minded and want to hear like what you have to say to help secure themselves better?
00:28:23.920 --> 00:28:24.160
Yeah.
00:28:24.160 --> 00:28:30.240
So especially in terms of like if you're in an organization, because I think individuals are a bit of a different animal.
00:28:30.240 --> 00:28:37.680
For an individual who's just trying to do AI stuff like at home, the best things you can do is limit third party exposure.
00:28:37.680 --> 00:28:44.640
So like uh you you don't need to download every skill and you don't need 40 MCP servers.
00:28:44.640 --> 00:28:50.640
Like if you if the every new piece of third-party content you add is just a risk, right?
00:28:50.640 --> 00:29:15.440
And so I think actually, like if you approach it on an if you're like on your personal computer trying to play with like Claude or whatever, uh, although you shouldn't because I don't like anthropic, uh, if you're playing with codex, which I guess I'll have to defend because I don't like Claude, um, then basically you will be best off by just ensuring you don't have down you don't download skills from, you know, MPX skills ad.
00:29:15.440 --> 00:29:20.400
You actually, if you
need one, you can copy paste it, and better yet, write your own.
00:29:20.400 --> 00:29:24.400
Uh, even better, I only have three skills that are written by me.
00:29:24.400 --> 00:29:27.680
I've never used more than that because they also pollute context.
00:29:27.680 --> 00:29:35.440
Uh MCP servers, I I literally I've only ever used, I think I have one at any given moment because I don't have any great need.
00:29:35.440 --> 00:29:38.000
Like it's not adding that much value.
00:29:38.000 --> 00:29:42.000
Um, in most cases, there are some use cases where you might want a few.
00:29:42.000 --> 00:29:45.200
Um, but limp the more you limit that third-party exposure, the better.
00:29:45.200 --> 00:29:54.319
Uh, and then for an organizational context, it's just like honestly, it if you have to play it the same way we've always known about security, it's like you start basic, you build up from there.
00:29:54.319 --> 00:30:03.279
I have organizations I talk to that will be like, you so like the product that I build is like pretty you need to be pretty advanced in the AI path to use.
00:30:03.279 --> 00:30:06.720
And then I'll talk to them and they'll be like, we don't even know what agents people are using.
00:30:06.720 --> 00:30:08.960
I'm like, okay, well, then my product's not for you.
00:30:08.960 --> 00:30:09.440
Okay.
00:30:09.440 --> 00:30:13.759
Because your first thing you need to do is understand like, what are your people doing?
00:30:13.759 --> 00:30:14.960
What skills do they have?
00:30:14.960 --> 00:30:16.480
What MCP servers are there?
00:30:16.480 --> 00:30:19.440
Then you need to be able to say, okay, okay, how do we manage permissions?
00:30:19.440 --> 00:30:21.599
Which is just a core security.
00:30:21.599 --> 00:30:28.480
Like you, you have a team that does identity and access control, presumably at most bigger orgs.
00:30:28.480 --> 00:30:35.519
Those orgs, those people can help manage permissions on agents too if you just let them, right?
00:30:35.519 --> 00:30:44.079
And so it's like you want to build up your approach to AI instead of trying to just buy something that's claims to solve it because it won't.
00:30:44.079 --> 00:30:49.279
You really want to build up capabilities of like, okay, do we have visibility into our agents?
00:30:49.279 --> 00:30:49.920
Yes, okay.
00:30:49.920 --> 00:30:56.480
Do we have the ability to uh do we have like MDM on the machine so I can say what's installed, right?
00:30:56.480 --> 00:30:58.799
And can I block people from running open claw?
00:30:58.799 --> 00:31:01.200
If you can't do that, you need to go solve that problem.
00:31:01.200 --> 00:31:03.200
Can I limit permissions to these agents?
00:31:03.200 --> 00:31:04.240
Yes, okay, cool.
00:31:04.240 --> 00:31:13.680
So you can build up capabilities as opposed to starting with like you go walking Black Hat's conference floor looking for someone that says they'll solve your problem because then you don't even know.
00:31:13.680 --> 00:31:18.319
Like you you need to to be good at AI, you need to know how to manage the security of AI.
00:31:18.319 --> 00:31:21.599
So you don't want to just go buy something off the shelf to solve it for you.
00:31:21.599 --> 00:31:24.480
You want your team to be actively engaged in that problem.
00:31:24.799 --> 00:31:30.720
I was gonna say, I thought we had uh MCP and other uh agent security solved by adding OAuth.
00:31:31.359 --> 00:31:31.839
Yeah, yeah.
00:31:31.839 --> 00:31:33.920
OAuth solves everything.
00:31:34.400 --> 00:31:37.359
Yeah, I'm joking for the listeners here.
00:31:37.359 --> 00:31:44.720
Like for the longest time, uh MCP servers were like not longest, but like for if folks that were evangelizing.
00:31:44.720 --> 00:31:51.519
Yeah, evangelizing were saying that like the security problems were solved because they added authentic OAuth authentication.
00:31:51.599 --> 00:32:05.680
And just recently they're like, oops, we're actually gonna also add authorization and other security controls because one of the things is that the a lot of the people involved in this, they act like, well, we can solve this.
00:32:05.680 --> 00:32:08.480
Uh yeah, MCP is the best example of it.
00:32:08.480 --> 00:32:12.000
Like, okay, we'll add this capability to it and now it's secure.
00:32:12.000 --> 00:32:12.960
And be like, what do you mean?
00:32:12.960 --> 00:32:17.759
Like we've had the capability on other applications forever to do authorization authentication.
00:32:17.759 --> 00:32:19.440
It doesn't mean it's like done correctly.
00:32:19.440 --> 00:32:20.720
It doesn't mean it's secure.
00:32:20.720 --> 00:32:23.119
It just means you like could do it secure.
00:32:23.119 --> 00:32:27.599
It's the same as like the labels that say like, like GDPR compliant, right?
00:32:27.599 --> 00:32:29.920
It's like, well, what is like what does that mean?
00:32:29.920 --> 00:32:30.160
Right.
00:32:30.160 --> 00:32:34.640
It's like, it just means like if you want to not give us your personal data, you don't.
00:32:34.640 --> 00:32:37.119
But of course, it's like gonna depend on what you actually do.
00:32:37.119 --> 00:32:38.640
Um, and so I think it's the same.
00:32:38.640 --> 00:32:42.480
It's like the, you know, okay, MCP is safe now because there's authentication.
00:32:42.480 --> 00:32:44.160
Like, it's just doesn't make any sense.
00:32:44.160 --> 00:33:02.880
But people want to sell you that idea because there's a lot of money that's been invested into MCP uh and into a lot of these solutions that like it would be kind of bad for some of these companies if they had to turn around and go like, okay, maybe MCP wasn't the best protocol for security reasons.
00:33:02.880 --> 00:33:12.400
Maybe they would lose a lot of money on some of their, you know, some of them, some of those security companies have bought for hundreds of millions of dollars MCP security products.
00:33:12.400 --> 00:33:21.599
So I think they're pretty committed to the idea
that MCP has to be the successful protocol, even though I think there are some serious security flaws with it.
00:33:21.599 --> 00:33:22.079
Yeah.
00:33:22.400 --> 00:33:26.400
I mean do I mean, do you think I mean, like anything, right?
00:33:26.400 --> 00:33:27.279
Like any application.
00:33:27.279 --> 00:33:34.480
Uh when you were talking about the third party, limit your third party exposure, I was thinking the same thing about like this is also true of applications in general, right?
00:33:34.480 --> 00:33:42.000
Like when you're talking about pen testing and stuff like that, you know, the more open holes you've got, the easier it is to get into your network and move laterally.
00:33:42.000 --> 00:33:51.039
Um Do you think something like MCP, the way it is designed, can it be secured correctly?
00:33:51.039 --> 00:34:02.799
Or is it that the people are just the people who are doing or bit who built MCP have no real like reason essentially to do it correctly?
00:34:03.440 --> 00:34:07.839
Yeah, I mean you can definitely add out ability to make MCP better.
00:34:07.839 --> 00:34:16.800
Uh I hope they don't, because the thing is MCP has so many flaws at its very foundation that the only way to do that would be to like completely reshape MCP.
00:34:16.800 --> 00:34:24.239
One of the, you know, I have I built this uh evil MCP server that I use to basically hack myself with constantly.
00:34:24.239 --> 00:34:27.440
Uh and sometimes I accidentally leave it on and it like actually hacks me.
00:34:27.440 --> 00:34:32.239
And I'm I'm just the worst security researcher ever because I've just like super sloppy about that.
00:34:32.239 --> 00:34:36.239
Uh and so, but the thing is it's Congratulations, you pwned yourself.
00:34:36.239 --> 00:34:37.280
Yeah, all the time.
00:34:37.280 --> 00:34:38.960
Yeah.
00:34:38.960 --> 00:34:44.639
And you know, one of the things is that you can have tool definitions.
00:34:44.639 --> 00:34:50.320
So like what you might expect that, like, okay, you load up this MCB server and it tells you what tools are available.
00:34:50.320 --> 00:34:53.280
The point is, I can change those tools dynamically.
00:34:53.280 --> 00:34:59.039
In fact, I can change them for depending on whose IP address I receive in the request, right?
00:34:59.039 --> 00:35:12.320
So I could make it so that like, well, when Catherine hits my thing, if I know that Catherine has this IP uh or has this maybe user agent or whatever other uh fingerprint I can find, I'll send her the malicious one, but everyone else gets the safe one.
00:35:12.320 --> 00:35:12.800
Okay.
00:35:12.800 --> 00:35:23.119
And then, and so it's sort of the same as like normal API, like API security, except now each API is not just returning data, it's returning instructions to your agent.
00:35:23.119 --> 00:35:23.440
Okay.
00:35:23.440 --> 00:35:30.320
So you see why that is such a scary uh framework, right?
00:35:30.320 --> 00:35:36.480
Because you're allowed uh because MCB fundamentally, the whole point of it is to inject it's to prompt inject you intentionally.
00:35:36.480 --> 00:35:38.559
Okay, it's to prompt inject you on purpose.
00:35:38.559 --> 00:35:45.039
It's to be able to say, let this server tell my agent what to do on top of what I'm saying, right?
00:35:45.039 --> 00:35:51.199
And so, of course, if you want it, if it wants to tell your your server, you know, do X, Y, or you know, hack me.
00:35:51.199 --> 00:35:58.800
One of them, one of the tools I made would tell my agent to hide uh uh hide vulnerabilities in my code.
00:35:58.800 --> 00:36:02.480
And like it works, it will, okay, depending on the agent.
00:36:02.480 --> 00:36:14.239
Um, and that's another problem is that as each of these labs adds new features to their like supported, you know, how these things work, the other models are not trained on that expectation.
00:36:14.239 --> 00:36:24.320
So, okay, so if if there's this new thing Anthropic does, maybe Opus will have built in the security safeguards inside of the model to where it's like harder to inject, right?
00:36:24.320 --> 00:36:31.199
But then you have maybe like Google didn't know about that thing because of course it was a secret, Anthropic launched it one day.
00:36:31.199 --> 00:36:34.639
And then what happens is Google has a model that has no idea.
00:36:34.639 --> 00:36:41.840
So MCP is a great example of this, where they added support for MCP after they trained Gemini 3.1.
00:36:41.840 --> 00:36:47.280
And so MCP and so Gemini did not know anything about how to protect against MCP.
00:36:47.280 --> 00:36:49.840
So you could just like there was just no rules.
00:36:49.840 --> 00:36:51.199
You could do whatever you wanted.
00:36:51.199 --> 00:36:53.039
And that will happen again and again.
00:36:53.039 --> 00:37:00.719
Every new feature they add, there's you have time for the other models to not know how to secure against it, and you can just wreck them.
00:37:00.719 --> 00:37:02.800
And that's like a huge problem, right?
00:37:03.039 --> 00:37:06.800
No, I think that's really incorporate that.
00:37:06.800 --> 00:37:08.800
That's that is actually really insightful.
00:37:08.800 --> 00:37:22.000
I might have to incorporate that into a future talk as well, because most of the time when I go talk to people about security for AI, uh, most of the time the the only thing they think about is we might have employees upload something sensitive to Chat GPT or something.
00:37:22.000 --> 00:37:29.280
They don't think about all the other ways they could get completely pwned by integrating this crap with like the live production systems.
00:37:29.679 --> 00:37:29.920
Right.
00:37:29.920 --> 00:37:31.760
And that's especially Yeah, go ahead.
00:37:31.760 --> 00:37:32.000
Sorry.
00:37:32.000 --> 00:37:41.760
So I was gonna say that's actually one of the things it drives me crazy that people get really afraid of like uh data loss, like you know, trying to like prevent things from getting
sent to certain servers and stuff.
00:37:41.760 --> 00:37:44.800
And I'm like, yeah, and I'm like, that's a problem, don't get me wrong.
00:37:44.800 --> 00:37:49.840
But it's like one of this like whole long list of problems.
00:37:49.840 --> 00:37:54.880
And realistically, a lot of organizations have been very bad at DLP historically, anyways.
00:37:54.880 --> 00:37:59.039
Like they've already been bad at it, like it's already been leaking, you know.
00:37:59.039 --> 00:38:06.000
Um, and it's been not to say that you shouldn't fix that, but it's like, let's not act like that's the biggest problem in AI agent security.
00:38:06.000 --> 00:38:17.119
Biggest problems are actually things like, you know, um when what you're leaking is to your agent who then leaks it to like a literal threat actor, okay?
00:38:17.119 --> 00:38:24.480
Or when that threat actor can can exfiltrate or inject into your system very you know, malicious instructions.
00:38:24.480 --> 00:38:28.400
And I've also talked about this around like people talk about sandboxing as a solution for it.
00:38:28.400 --> 00:38:33.280
And don't get me wrong, like you should you should definitely be working on sandboxing solutions for your AI agents.
00:38:33.280 --> 00:38:44.000
But one of the things is if your agent builds code inside of a sandbox that later runs outside of a sandbox, like it's still just like it's like, okay, then it's just a waiting game, right?
00:38:44.000 --> 00:39:02.239
You put in it's sort of like um blind cross-site, you know, or blind um, you know, XSS like attack is that you can just go, uh, well, I'll put the malicious thing into the code, and then the developer will take the code out and run it on their machine, and then it will exfiltrate the data, right?
00:39:02.239 --> 00:39:09.039
And so there's all of these tricks that you have to think about that aren't just as simple as did we send something dangerous out to a server, right?
00:39:09.039 --> 00:39:14.960
Because it can be like multi-step, it can be uh, you know, tricking things to behave poorly.
00:39:14.960 --> 00:39:17.199
It's like a million problems.
00:39:17.679 --> 00:39:44.320
Like well, that there's and there's even like going back to like very basic situations situations, like how many companies do we know took like like got a license for chat GPT or or for you know, and put a chat front end on their website, like an AI chat front end on their website that has way too many permissions to the back end to alter orders, to get you know, alter payment information, change your flight, uh, give a 100% discount, things like that.
00:39:44.320 --> 00:39:52.400
And people, yet a lot of the people that are just plopping it and are like, I got this license or this API through like OpenAI or whatever else.
00:39:52.400 --> 00:40:09.440
They're they're a big company, they're secure, but they don't realize that when you're integrating it with your actual live production systems and you're not, you're not like controlling the level of access they have, that gives somebody a direct like line to potentially like do a SQL injection, command uh command injection.
00:40:09.440 --> 00:40:21.039
Uh give themselves a hundred percent discount, like find information about other customers, uh depending on how how thoroughly you like integrated this like chat bot with your back-end systems.
00:40:22.239 --> 00:40:27.199
Instagram had that thing where you could take over accounts just via their little help.
00:40:27.199 --> 00:40:29.280
And you know, they're like that was three weeks ago.
00:40:29.280 --> 00:40:30.960
I know that just happened.
00:40:30.960 --> 00:40:42.559
And that's and that's important for people to think about because whenever a company tells me, like, well, I think we have like we've done we're pretty good at this, I'm like, listen, Meta did the dumbest possible thing.
00:40:42.559 --> 00:40:45.920
So like if and that's and meta has some good security people.
00:40:45.920 --> 00:40:51.360
Like, I'm let's not kid ourselves that Meta is not had does not have talented people at the company.
00:40:51.360 --> 00:40:54.559
But what they have is they have organizational dysfunction.
00:40:54.559 --> 00:41:01.360
So as a result, the I'm sure that the talented cybersecurity people were not there when that decision was made.
00:41:01.360 --> 00:41:06.239
They did not test it, they did not, and as a result, you you get completely pwned.
00:41:06.239 --> 00:41:06.559
Okay.
00:41:06.559 --> 00:41:12.239
And that is the biggest mistake that like I think organizations need to be thinking very carefully about.
00:41:12.239 --> 00:41:22.239
It's like, how do we make sure that decisions on AI are not being made outside of like classic architectural how do you build systems questions?
00:41:22.239 --> 00:41:29.440
Because like the fact that it had that access means that like the wrong people were in the room.
00:41:29.440 --> 00:41:35.039
And and I think if if that can happen to Meta, that can happen to basically every organization that you can think of.
00:41:35.039 --> 00:41:39.360
And they need to be very uh everyone involved in this needs to be thinking about that problem.
00:41:39.360 --> 00:41:41.760
Um, because it happens all the time.
00:41:41.760 --> 00:41:50.159
And I'm sure, I'm sure I could go right now if I wanted, like today, you know, after I could probably just go find another website where that's true and just do the exact same thing.
00:41:50.159 --> 00:41:50.880
100%.
00:41:51.119 --> 00:41:52.079
There's chatbots everywhere.
00:41:52.079 --> 00:41:53.039
I'm sure they're everywhere.
00:41:53.039 --> 00:41:53.679
Absolutely.
00:41:53.679 --> 00:41:54.800
Absolutely.
00:41:55.199 --> 00:42:07.440
Yeah, and and the the reality is like every time I talk to customers about this, like, you know, Cisco Live Talk or or just getting in front of customers, like, like they've they they you know, like the DLP problem.
00:42:07.440 --> 00:42:14.400
They look at it as a DLP problem, and then I start to show them like, hey, look, your your department uses like for the customer front end, uses a chat bot.
00:42:14.400 --> 00:42:18.800
Looks like it's like license, you know, open, you know, back-ended by open AI.
00:42:18.800 --> 00:42:21.039
Like this is real stuff that can happen.
00:42:21.039 --> 00:42:26.800
And they're like the they get that oh shit moment where they're like, oh crap, we didn't even think about that.
00:42:26.800 --> 00:42:28.960
We just plugged it in and thought it was secure.
00:42:29.199 --> 00:42:29.519
Oh, right.
00:42:30.000 --> 00:42:36.480
And in meta in Meta's case, like they actually denied it at first because they were like, we didn't give it access to reset accounts.
00:42:36.480 --> 00:42:40.239
And the hackers actually posted the chats and took over one of Meta's accounts.
00:42:40.239 --> 00:42:46.800
They posted on Reddit, like the actual chat, and took over uh one of the official Meta accounts to prove that it was it was real.
00:42:47.199 --> 00:42:48.719
So this is this is interesting.
00:42:48.719 --> 00:42:52.400
Uh getting back to the bit about sandboxing, I just this happens to connect.
00:42:52.400 --> 00:42:58.480
So a friend of the a friend of ours, friend of the podcast, uh Andrew Brown, he uh owns Exam Pro and he's a it's a training site.
00:42:58.480 --> 00:43:02.159
And he does, he did a course on Claude Code.
00:43:02.159 --> 00:43:10.639
And in the course of doing the Claude Code um course, recording it, he did the section on sandboxing that Anthropic and Claude Hodde has.
00:43:10.639 --> 00:43:22.639
And he he found out accidentally while recording that basically the sandboxing is essentially useless because you can tell it like build build only in the sandbox, have only these permissions.
00:43:22.639 --> 00:43:33.679
And as soon as you tell it to do something inside the sandbox that it needs to step outside the sandbox to do, the agent will disable the sandbox and just go do the thing, which is insane.
00:43:34.079 --> 00:43:39.119
Yeah, so cla the the yeah, go on, no, no, you go on.
00:43:39.119 --> 00:43:47.599
I was gonna say the default Claude sandboxing is not a sandbox, it's just a word they use to make people feel like it's you know, it's it's appalling.
00:43:47.599 --> 00:43:50.159
It's like literally they should not have that.
00:43:50.159 --> 00:43:53.039
Okay, because it makes people feel safe, but they're not.
00:43:53.039 --> 00:43:53.519
Okay.
00:43:53.840 --> 00:43:58.239
Yeah, it's not like a true sandbox in the sense of like detonating malware in a sandbox.
00:43:58.239 --> 00:43:59.599
Extra VM or something.
00:43:59.599 --> 00:44:02.320
Where you have control over everything.
00:44:02.880 --> 00:44:03.199
Right.
00:44:03.199 --> 00:44:11.119
And and and so, and I have done, I had a workshop a couple weeks ago where I showed how to break out of sandboxes and stuff using AI.
00:44:11.119 --> 00:44:16.079
You know, the thing is, a sandbox is is quite a bad term for this in some ways.
00:44:16.079 --> 00:44:25.360
It's obviously comes from sandboxing that we've had historically, but like I think it leads people to think about it as like a nice little play place, as opposed to like it should be like maximum security prison.
00:44:25.360 --> 00:44:25.679
Okay.
00:44:25.679 --> 00:44:29.519
You need to think about it as like your AI agent is like a hardened criminal.
00:44:29.519 --> 00:44:33.599
This is like Alcatraz, and you need to like have the walls up.
00:44:33.599 --> 00:44:37.519
And so they they test it as if it's like, yeah, you can get a little sand outside the box.
00:44:37.519 --> 00:44:40.480
And it's like, no, like you have to be completely hardened.
00:44:40.480 --> 00:44:46.320
Otherwise, as soon as it because these agents, and this is one of the biggest risks in AI agent security, it's not external threats at all.
00:44:46.320 --> 00:44:52.320
It's just your agent does something that it thinks it needs to do in order to solve the problem that you wouldn't want to do.
00:44:52.320 --> 00:44:59.760
It would, it will go and say, Oh, I see that you're there's a migration issue in your data.
00:44:59.760 --> 00:45:02.559
I'll wipe your production database and start over.
00:45:02.559 --> 00:45:02.960
Okay.
00:45:02.960 --> 00:45:06.000
It's like a, you see, it's like it's somewhat like logical, right?
00:45:06.000 --> 00:45:08.559
It's like it's trying to make like an informed decision, right?
00:45:08.559 --> 00:45:09.440
Some of Antwerp.
00:45:10.239 --> 00:45:17.119
I'm getting flashbacks of the AWS outage where AWS's uh AI basically was like, this code is crap.
00:45:17.119 --> 00:45:18.639
I'm gonna wipe it and rebuild.
00:45:18.639 --> 00:45:21.199
And it just caused a huge AWS outage.
00:45:21.920 --> 00:45:24.159
And you can only style and you can understand it.
00:45:24.159 --> 00:45:43.199
It's like it's in some ways, like that's of course, if it doesn't know the broader context, it's like, well, just you know, it's like it's it's like having the dumbest possible intern just like coming and giving, you know, when an intern starts the company, you don't give them production keys, but somehow we were like, Opus 4.8 definitely deserves those keys.
00:45:43.199 --> 00:45:46.480
And so it's like, you know, uh, and that's a huge issue.
00:45:46.800 --> 00:45:56.719
I'm gonna uh mention I I got a ride to the airport going leaving Cisco live with a nice gentleman, and yeah, we we got to talking about AI security and security for AI AI.
00:45:56.719 --> 00:46:00.000
And he uses Claude uh for a lot of like work stuff.
00:46:00.000 --> 00:46:07.440
So I guess he was going through like an outage or having a deal with something, and he was trying to get like uh help get some assistance with Claude.
00:46:07.440 --> 00:46:22.719
And uh he was mentioning that like, oh, it didn't have direct access to like while it was trying to help him, it didn't have
direct act as access to the like the production system, but it figured out a way to make the changes via an API and it just basically asked his permission and did it.
00:46:22.719 --> 00:46:34.079
And he was like, that was kind of cool, but at the same time, holy shit, like this this thing is like, you know, it is potentially gonna like, you know, could do some real harm if it's just like left in the wild.
00:46:34.079 --> 00:46:37.679
I'm like, yeah, at least you like were having to approve it, but that's the problem.
00:46:37.679 --> 00:46:40.159
Like a lot of people are just like set it and forget it.
00:46:40.159 --> 00:46:44.559
Like it's a big, uh, you know, it's a it's a uh an awesome tool.
00:46:44.559 --> 00:46:50.000
And, you know, I I think I've uh uh just given it enough access to only do this because I've told it to.
00:46:50.000 --> 00:46:56.880
Meanwhile, it can find potentially a way around that, you know, that lack of access if it thinks that's what it needs to solve the problem.
00:46:57.360 --> 00:46:57.519
Right.
00:46:57.519 --> 00:47:09.760
And I have, and there are some of these, I've never talked about this because I don't want to, you know, I haven't decided like how solvable of a problem it is, but I'll just like kind of point to it generally.
00:47:09.760 --> 00:47:22.719
Um there's an issue, which is that a lot of these agents, um, for example, OpenClaw out of the box, um, have uh certain, or actually let's like Nemo Claw, which is the uh NVIDIA one.
00:47:22.719 --> 00:47:26.880
It has certain domains that it is allowed to access by default.
00:47:26.880 --> 00:47:29.039
And one of those is like Telegram, okay.
00:47:29.039 --> 00:47:33.519
And that is necessary so that people can integrate it with their like Telegram chat, right?
00:47:33.519 --> 00:47:35.760
So you can talk to your agent via Telegram.
00:47:35.760 --> 00:47:47.760
Um, but like imagine if someone were to make a Telegram bot that just uh allows you to send it messages of arbitrary URLs and then it will return the content.
00:47:47.760 --> 00:47:48.239
Okay.
00:47:48.239 --> 00:47:54.159
Now, if someone were to do that, and then of course were to make a blog post on it, then it would get into the training data.
00:47:54.159 --> 00:48:02.000
And then every agent would then know that if you want to access the internet, which you're not allowed to do, you can do it by accessing this telegram, for example.
00:48:02.000 --> 00:48:02.320
Okay.
00:48:02.320 --> 00:48:19.039
And then now, like you basically like this whole I and and then and my question becomes there are probably things like that already, where there are cases where agents know workarounds have been built in, maybe even intentionally, to get around whatever restriction is.
00:48:19.039 --> 00:48:28.159
It goes, okay, I don't have access to the production database, but I do have this uh a you know proxy that I can hit that has been set up for me to get broader permissions, right?
00:48:28.159 --> 00:48:34.159
And and that I think is a is a very serious issue that like I don't I don't know if there's brilliant solutions for.
00:48:34.159 --> 00:48:47.760
So I don't talk about it much because like, but I it's like when people talk about poisoned LLMs, like I don't worry as much about that as I worry about like this type of poisoning of like it knows a workaround that like would work, right?
00:48:48.159 --> 00:48:51.360
Um yeah, and it might be thinking it's actually helping.
00:48:51.360 --> 00:48:54.480
Like, look at the head of uh AI security at Facebook.
00:48:54.480 --> 00:49:03.519
She integrate which don't get me sorry, but like she got she integrated uh OpenClaw like really early on with her her production work email.
00:49:03.519 --> 00:49:05.920
And the thing was like, oh, I'm gonna take care of your email.
00:49:05.920 --> 00:49:07.119
I'm deleting the whole inbox.
00:49:07.119 --> 00:49:08.079
Please stop, please stop.
00:49:08.079 --> 00:49:09.199
Nope, deleted.
00:49:09.199 --> 00:49:10.320
And that was her work email.
00:49:10.320 --> 00:49:12.400
Like people who I can't believe she posted that.
00:49:12.800 --> 00:49:13.679
I can't believe she posted that.
00:49:14.400 --> 00:49:15.599
Good for her for doing it, I guess.
00:49:15.599 --> 00:49:19.679
I wonder if she still has a I mean, at least she was honest.
00:49:19.760 --> 00:49:26.480
I I guess uh I mean maybe she posted that because she wanted to be a cautionary tale for other people to not do something silly like that.
00:49:26.480 --> 00:49:33.440
But the reality is like I I it it's interesting to like see like the threats from AI.
00:49:33.440 --> 00:49:54.079
Like I mean, people are like so focused on the DLP aspect or using AI for offensive security that they forget that like the AI that the legitimate AI that they're using for their own production systems can fail in terrible ways that could actually, you know, that are not m meant to be malicious, but they can end up being malicious, like the AWS outage, the Facebook thing.
00:49:54.079 --> 00:50:05.760
And then that there's the fact that like a lot of people are implementing this in, you know, these legitimate tools in a way that they think is secure or they didn't think about security, where a malicious actor can you like piggyback on their existing tools.
00:50:05.760 --> 00:50:12.639
Like integrate like an example like theoretical would be like integrating open claw with your production email, even if it's not deleting at all.
00:50:12.639 --> 00:50:22.639
Somebody could send a prompt injection via an email and potentially get a you know, a back into your open claw instance that's sitting on your network internally past the firewall.
00:50:23.199 --> 00:50:23.679
Yeah, yeah, yeah.
00:50:23.679 --> 00:50:25.360
All sorts of crazy things you can do.
00:50:26.159 --> 00:50:33.039
It just reads their email and then uh grants, you know, creates a new user account for somebody and sends it to sends it back to
them or something.
00:50:33.440 --> 00:50:43.199
Yeah, but back at the back of the I was I was gonna say back like a lot of pen testers look for like unpatched systems, look for uh like known vulnerabilities, things like that.
00:50:43.199 --> 00:50:54.960
And then then we've got these AI tools that have no like vulnerabilities that just can't be patched potentially, or like they're that by design are meant to be like able to be worked around.
00:50:54.960 --> 00:51:05.599
And now you have this whole new attack vector that's right, you know, could potentially get you right past the firewall, can get you, you know, in in the data center to get you full access to all the API keys.
00:51:05.599 --> 00:51:09.920
And they're just kind of going wild in the enterprises these days.
00:51:10.559 --> 00:51:15.760
So we need uh unfortunately we need a wrap soon, but I have a funny question um for you.
00:51:15.760 --> 00:51:19.519
Uh for it's really it's oh, it's almost philosophical, but probably not.
00:51:19.519 --> 00:51:26.719
Um so is prompt engineering social social engineering or prompt it's probably prompt injection, is that social engineering?
00:51:28.880 --> 00:51:40.239
I'd argue effectively no, because I think that the most effective prompt injections um are going to look nothing like a human readable form.
00:51:40.239 --> 00:51:44.719
Uh they will end up being like more technical as we progress.
00:51:44.719 --> 00:51:50.159
Like it's it's harder and harder to prompt inject something by tricking it, by going like, ah, I made you think something, right?
00:51:50.159 --> 00:51:55.199
But if you just push it into the right set of tokens, you can get them to behave however you want.
00:51:55.199 --> 00:51:57.039
We see this with a lot of jailbreaks, right?
00:51:57.039 --> 00:51:59.280
I think it will become more technical of a field.
00:51:59.280 --> 00:52:04.800
Um but yeah, uh currently, yes, absolutely it's social engineering, just social engineering of a computer.
00:52:05.039 --> 00:52:07.360
Um I thought that was funny.
00:52:07.360 --> 00:52:08.079
No, that's good.
00:52:08.079 --> 00:52:10.639
It's very good, very good distinction though.
00:52:10.639 --> 00:52:11.280
And you're right.
00:52:11.280 --> 00:52:23.679
The more and I what what's interesting is that the guardrails seem focused on the social engineering aspect because that's what people because of course that's traditionally now how I say traditionally is it's been around for a while, it hasn't, but that's how it's been done.
00:52:23.679 --> 00:52:26.800
But yeah, I'm curious to see how that uh evolves.
00:52:26.800 --> 00:52:28.400
And more zak.
00:52:28.400 --> 00:52:29.119
Yeah.
00:52:29.119 --> 00:52:29.760
Go on.
00:52:30.320 --> 00:52:35.199
I was gonna say uh Graosaca is like the better ones are gonna be more technical.
00:52:35.199 --> 00:52:41.920
Um I think though with social engineering though, like a human being gets suspicious.
00:52:41.920 --> 00:52:47.840
And
if somebody else calls even with a different voice and starts another pretext, your suspici your suspicions are array set.
00:52:47.840 --> 00:52:59.599
With AI, it's pretty much a happy little idiot that uh you know, that has a certain filters and controls and input validation that you know it might check for, but it's got amnesia every time you're trying.
00:52:59.599 --> 00:53:04.480
It's not it's not raising, it's not suspicious, you know, just because you tried five times already.
00:53:04.960 --> 00:53:05.199
Right.
00:53:05.199 --> 00:53:12.079
And and this is actually one of the things I want to say also, which is that exactly you get unlimited attempts with prompt injection, right?
00:53:12.079 --> 00:53:15.519
You get to just keep doing it and doing it and doing it, and there's no problem there.
00:53:15.519 --> 00:53:21.679
And then also, one thing that people miss about prompt injection is that a lot of the times it's like authorized prompt injection.
00:53:21.679 --> 00:53:28.000
So, you know, there's this one major tool that I proved that basically anyone could communicate from.
00:53:28.000 --> 00:53:33.119
Uh, basically I can send messages to that tool via any website you visit if you just visit on your computer.
00:53:33.119 --> 00:53:40.000
And then I am prompt injecting your machine, but your agents doesn't see that that's not the same as the user themselves.
00:53:40.000 --> 00:53:42.800
So there's really no protection against that, right?
00:53:42.800 --> 00:53:45.119
Like you you're just I don't even have to trick it.
00:53:45.119 --> 00:53:48.960
I just like say like delete database and it goes like okay, I guess.
00:53:48.960 --> 00:53:50.480
Because it thinks you are the user.
00:53:50.480 --> 00:53:56.719
So there are types of attacks like that that are prompt injection, but they're not gonna you can't you can't outsmart them, you know?
00:53:56.960 --> 00:53:57.599
Yeah, absolutely.
00:53:57.599 --> 00:53:59.599
Oh man, this is this is good stuff.
00:53:59.599 --> 00:54:07.840
And I know that there's so much that we could keep going with, but unfortunately we have to we have to wrap at some point or our listeners will stop listening to us.
00:54:07.840 --> 00:54:10.800
So Zach, thanks for coming on, man.
00:54:10.800 --> 00:54:13.360
This has been so such a good conversation.
00:54:13.360 --> 00:54:16.960
Um any any plugs, any any last thoughts?
00:54:16.960 --> 00:54:18.800
Uh and I'll yeah.
00:54:19.840 --> 00:54:23.360
Don't use anthropic, but uh other than that, we're good.
00:54:24.639 --> 00:54:26.320
Catherine, you want to bring us home?
00:54:26.880 --> 00:54:29.760
I was just gonna say thank you again for for coming on.
00:54:29.760 --> 00:54:41.199
I you know, I've been following you on Twitter for a while, and I, you know, uh as much as Twitter can be spicy, you always bring really good points and you're extremely technical.
00:54:41.199 --> 00:54:44.159
And you know, you're you know, if you're wrong, you're wrong.
00:54:44.159 --> 00:54:47.760
You'll you'll admit it, but you are right more most of the time.
00:54:47.760 --> 00:55:10.239
And I think that we need more people like yourself who are able to say, like, hey, no, this is you know, red flag, canary, like in the coal mine, like uh and I I think it is in you know, as we're advancing so quickly in the AI field and uh and all these tools, like we have to remember to let like let security also catch up and and uh to really build it like foundationally, like you said.
00:55:10.960 --> 00:55:11.599
Yeah.
00:55:11.599 --> 00:55:12.719
Yeah.
00:55:12.719 --> 00:55:14.559
Uh one last thing then, Zach.
00:55:14.559 --> 00:55:20.079
Where uh people can find we're gonna include your contact info for Twitter for the you know in the show notes, obviously.
00:55:20.079 --> 00:55:22.159
Anywhere else people should look for you?
00:55:22.800 --> 00:55:23.840
Twitter's definitely the best.
00:55:23.840 --> 00:55:25.840
If I get a message anywhere else, I just ignore it.
00:55:25.840 --> 00:55:26.320
So yeah.
00:55:26.320 --> 00:55:28.320
And thanks for having me on, by the way.
00:55:28.320 --> 00:55:28.800
Yeah.
00:55:28.800 --> 00:55:30.159
Thanks for having me here.
00:55:30.480 --> 00:55:30.800
Yep.
00:55:30.800 --> 00:55:31.679
Thanks for coming.
00:55:31.679 --> 00:55:32.480
All right, everybody.
00:55:32.480 --> 00:55:32.639
Yeah.
00:55:32.639 --> 00:55:37.280
Well, we'll see you on a we'll see you on another episode of the Cables to Clouds podcast.
00:55:37.280 --> 00:55:39.039
Check you later.