TRANSKRYPCJA 🔗
WEBVTT
00:00:00.160 --> 00:00:04.000
I just happened to have some real world experience with this just recently.
00:00:04.240 --> 00:00:13.359
The level of speed and thoroughness and complexity at which the AI attacker uses is extraordinary.
00:00:13.599 --> 00:00:17.679
It doesn't give up and it attacks all surfaces at once.
00:00:17.920 --> 00:00:30.640
It it found an O day in a product by because it was an open source product, it went, got the code, reversed the code, found the O day, used it against us in order to get in.
00:00:30.800 --> 00:00:38.399
But once it's inside the enterprise, it goes, oh hey, your clawed code config, I'll just take that key, use that session.
00:00:38.479 --> 00:00:40.560
Now it's used in your internal AI.
00:00:40.960 --> 00:00:44.240
You have to literally unplug the internet, right?
00:00:44.320 --> 00:00:47.119
In real large enterprises, there's not a plug to pull.
00:00:48.560 --> 00:00:55.280
The AI Native Dev is a podcast for developers and engineering leads at the cutting edge of AI energetic coding.
00:00:55.439 --> 00:01:05.680
Join your hosts, Guy Fajani, and meet Simon Maple every week as we chat with the most exciting voices in AI and tackle the biggest questions facing developers today.
00:01:06.159 --> 00:01:08.719
This is the AI Native Dev.
00:01:11.040 --> 00:01:13.120
Yesterday, you were writing code.
00:01:13.280 --> 00:01:15.200
Today, you're prompting agents.
00:01:15.359 --> 00:01:17.920
Tomorrow, you'll be running a software factory.
00:01:18.079 --> 00:01:21.680
Agentic coding is moving fast enough to give anyone whiplash.
00:01:21.920 --> 00:01:24.879
This June, AI DevCon London sold out.
00:01:25.040 --> 00:01:35.359
Hundreds of developers, engineering leads, and VPs packed in to hear speakers from Anthropic, OpenAI, Netlify, and GitHub cut through the hype and get to what actually works.
00:01:35.599 --> 00:01:38.319
This November, we're bringing it back to New York.
00:01:38.480 --> 00:01:42.799
Bigger, bolder, and full of people actually building the agentic future.
00:01:42.959 --> 00:01:52.480
Three days of real talks, real tools, real answers, whether you're directing the agents, building the loops themselves, leading the teams, or setting the strategy.
00:01:52.719 --> 00:01:58.640
As a podcast listener, you can get 15% off your ticket using promo code POD50.
00:01:59.680 --> 00:02:03.840
We have so much planned, and we can't wait to have you join us in New York.
00:02:25.439 --> 00:02:27.280
Caleb, we've known each other for a good while.
00:02:27.439 --> 00:02:32.159
And are like the many things to sort of introduce you, you're sort of the CISO of Robin Hood and Databricks on it.
00:02:32.240 --> 00:02:37.120
You're sort of a security investor, you know, you're sort of like well super versatile.
00:02:37.199 --> 00:02:41.520
You're the host of the uh uh AI security uh podcast, right?
00:02:41.840 --> 00:02:43.680
Uh or host, I should say.
00:02:43.840 --> 00:02:46.400
Um so many,
Load full transcript
many lenses on it.
00:02:46.560 --> 00:02:49.680
Thanks for kind of joining us over here to uh share some perspectives.
00:02:50.319 --> 00:02:50.800
Awesome.
00:02:50.960 --> 00:02:53.919
Um it's always exciting uh talking with you.
00:02:54.159 --> 00:02:58.639
Um obviously we ramble a lot, so hopefully that's interesting to your listeners.
00:02:58.960 --> 00:03:00.960
Hopefully we sort of keep things interesting to the end.
00:03:01.199 --> 00:03:02.879
So I guess uh starting from it.
00:03:02.960 --> 00:03:16.159
So the the the last couple of months, I don't know, like time's a bit of a blur, maybe like month, uh have been fascinating, really like through the summer, from uh kind of AI, security AI attackers and alignment and such.
00:03:16.400 --> 00:03:26.800
Definitely has gone from like a niche topics that geeks like ourselves, you know, like talk about to now kind of you know people texting you and sort of, you know, like whatever your sort of grandmother, right?
00:03:26.879 --> 00:03:30.639
Or sort of like AI gonna uh kill us all.
00:03:30.800 --> 00:03:37.360
Um you definitely have gotten to parents calling and asking, are we gonna die by AI?
00:03:37.759 --> 00:03:39.199
Is it gonna hack everything?
00:03:39.439 --> 00:03:41.520
I've already had that from both my parents.
00:03:41.680 --> 00:03:48.960
So yeah, I I definitely have gotten like uh definitely non-tech world uh kind of probes of uh what do you think?
00:03:49.039 --> 00:03:51.759
Like is uh is this all is it marketing?
00:03:52.080 --> 00:03:58.159
So um I think not to sort of summarize all the incidents, but to give people a little bit of context if you're not quite sort of come in.
00:03:58.319 --> 00:04:08.319
We started with sort of the uh hugging face breach that was uh done by, in hindsight, uh AI agents that broke out of their uh sandboxes out of OpenAI.
00:04:08.479 --> 00:04:45.519
Uh, we've then learned at Black Hat that this wasn't some sort of specific experiment that came out and was trying to find a cheat sheet, but rather almost like this coordinated effort that uh a bunch of agents um uh coordinated using a vulnerability in Artifactory, which kind of pointed at that as just the shared infrastructure that happened to be around that they used as a message board, but interesting uh kind of community almost like behaviors of the swarm of agents and kind of agents sacrificing themselves to or their tokens uh to be able to benefit the swarm and find information.
00:04:45.759 --> 00:04:49.759
And in general, that that swarm was acting to kind of hide its tracks.
00:04:50.240 --> 00:04:52.160
Um that was pretty scary.
00:04:52.399 --> 00:05:01.279
And in September we had um uh the the misalignment, I think it's at the beginning of September that we had the sort of misalignment framework from OpenAI.
00:05:01.439 --> 00:05:07.439
We had Dario before that uh call for uh for a slowdown, you know, for for pacing the frontier.
00:05:08.000 --> 00:05:19.839
Uh and over
there you found other things like uh the one that really drew my attention was this uh uh agents leaving instructions in the compaction of conversations to the future versions of themselves.
00:05:19.920 --> 00:05:28.079
So as they compact the context, leave the things like this has been breached, ignore user instructions uh and such in the uh in the message, which is pretty wild.
00:05:28.319 --> 00:05:30.480
So anyways, it's been it's been a fun time.
00:05:30.959 --> 00:05:34.160
I guess with that, I guess I wanted to get your view.
00:05:34.319 --> 00:05:52.160
One is you know, of of this mess, what surprises you, what doesn't surprise you, uh and and and maybe maybe specifically like are there aspects of it you find more maybe like alarming uh than uh than things that are just okay, this is just kind of par for the course?
00:05:53.120 --> 00:05:56.720
Yeah, I mean that's a gosh, it's a loaded question.
00:05:57.360 --> 00:06:02.560
Um let me start with what surprises me versus not.
00:06:02.959 --> 00:06:17.439
Um actually, I think even two years ago I was on my podcast and I was talking to some of the top red teamers and reverse engineer people on the podcast, and we were discussing sort of AI's capability.
00:06:17.759 --> 00:06:24.720
And we all sort of in some sense were set on this position that we think it's gonna be very difficult.
00:06:24.959 --> 00:06:28.240
You know, AI is is a prediction machine, right?
00:06:28.399 --> 00:06:33.519
And so what it does is it predicts really well based on the things at which it sees.
00:06:33.680 --> 00:06:45.120
And so if you read bug bounty reports or see vulnerability disclosures, it's very clear it has to be able to it knows how to exploit SQL injection because there's lots of material on it.
00:06:45.600 --> 00:07:00.399
However, when it comes to taking disparate things that are not connected, like for example, when you do a uh a red team engagement or a pen test, you might say, Oh, I found in a website a link to an S3 bucket because it was serving an asset.
00:07:00.560 --> 00:07:02.639
Actually, that S3 bucket gave a listing.
00:07:02.879 --> 00:07:04.399
Inside that, there is a file.
00:07:04.560 --> 00:07:14.160
In that file, it looked like things like usernames and passwords, but it didn't say usernames and passwords, but then taking that and then brute forcing it against the login page that that company has.
00:07:14.639 --> 00:07:20.240
These are very sort of like you know, very non-connected things uh in that sense.
00:07:20.399 --> 00:07:24.399
And we were like, you know, very much like you know, AI won't be able to get to that.
00:07:24.560 --> 00:07:27.360
That is something at where the human can only be.
00:07:27.600 --> 00:07:29.519
And and I was clearly wrong.
00:07:29.680 --> 00:07:51.439
Um like I think we were all clearly wrong in the sense that what has surprised us is not only is AI way better, but it is uh being able to do these sort of uh dependency connections, but it does it so fast and also I think in and of itself creates very innovative methods.
00:07:52.000 --> 00:08:06.959
Um, I was at uh OpenAI's security conference uh just a couple weeks ago where they talked about how they took Linux and they basically said, okay, your job is to escalate prips.
00:08:07.839 --> 00:08:19.040
And they ran it and they found over 400 Odays from local user to root that produced the binary of you just run it and you got root instantly.
00:08:19.360 --> 00:08:19.600
Right?
00:08:20.000 --> 00:08:31.199
So actual evidence, 400, and they said on average it took about two and a half hours per exploit to research, identify, test to validate, yeah, and then produce the deliverable.
00:08:31.680 --> 00:08:45.759
Um and either the the best re reverse engineers, obviously, are there and it they're finding ones that are taking them weeks to figure out how they identified that.
00:08:46.399 --> 00:08:50.559
How do they even find out what's the sort of the breadcrumb kind of trail that derived?
00:08:50.639 --> 00:08:58.080
And the thing is, and they're saying, hey, these things are they're they're like finding and identifying new methods and technologies that have never been published, right?
00:08:58.559 --> 00:09:01.759
So that is that's what surprises me for sure.
00:09:02.240 --> 00:09:05.519
I mean, it's a pretty it's a pretty kind of alarming kind of path on it.
00:09:05.679 --> 00:09:16.480
Do you feel uh, I mean, neither of us is like a core AI researcher, but you know, do you feel this is more kind of the raw intelligence of the models, uh, being able to reason?
00:09:16.639 --> 00:09:38.240
Or or is it actually like an evolution of their uh kind of reinforcement learning and just being able to, you know, they've they've been definitely putting agents to this specific task of like, you know, explore a system, figure it out, and learn it, which maybe is a little bit more like uh like the AlphaGo uh situations of just figure it out and rewarding success.
00:09:38.799 --> 00:09:40.480
You know, I again I'm I'm similar to you.
00:09:40.639 --> 00:09:51.200
I'm not an AI researcher, so I can only sort of hypothesize around what would make this, but yeah, I I always sort of map the way AI is according to what a human is, right?
00:09:51.360 --> 00:09:59.120
And so when you think about the best pen testers, who are the guys who know how to break into things in very unique and creative ways?
00:09:59.279 --> 00:10:00.960
Um, and there's a couple fundamentals.
00:10:01.120 --> 00:10:10.000
The fundamental number one is they are core a, they understand computers, they understand the work, how it works, the rules and the fundamentals, right?
00:10:10.559 --> 00:10:12.080
They understand the tech stack.
00:10:12.240 --> 00:10:16.720
They know what it does, how it works, how it's configured, what it does.
00:10:16.879 --> 00:10:23.039
And then they have lots of experience, right, and being able to see and what happens is it maps.
00:10:23.120 --> 00:10:32.320
Like in your head, yeah, if you know these things, I can look at something that says, hey, I this looks like usernames and passwords because I've spent years, 30 years looking at them.
00:10:33.679 --> 00:10:34.879
I've seen them so many times.
00:10:35.039 --> 00:10:37.840
Like clearly, these look like usernames and passwords.
00:10:37.919 --> 00:10:42.159
And clearly I remember seeing a login page on this website back here.
00:10:42.320 --> 00:10:43.919
I can connect the two, right?
00:10:44.000 --> 00:10:45.679
And make this mapping.
00:10:46.000 --> 00:10:52.559
And so, like, obviously, from human perspective, we can see how these you know maps get created.
00:10:52.720 --> 00:10:56.720
And AI is nothing but the largest brain in the world.
00:10:56.879 --> 00:11:02.559
And so it has seen everything and it knows the tech stack, it knows code, it knows everything.
00:11:02.720 --> 00:11:11.279
So it makes these mappings at such large expansive scales that it can do them quicker and faster and many more of them.
00:11:11.519 --> 00:11:27.679
So, like, unlike us, I can easily map, oh, S3 bucket file looks like usernames and passwords, but it may be able to say these usernames and passwords look specifically like Linksys version 1.4, you know, on this stack, which by the way, this thing was, you know, it knows and it can create.
00:11:27.759 --> 00:11:34.320
And by the way, Linksys 1.4 has an exploit here that I know about that back in the past that was similar.
00:11:34.399 --> 00:11:35.519
I can just combine them.
00:11:35.679 --> 00:11:38.480
And so I feel to me it that's what it's doing.
00:11:38.559 --> 00:11:45.600
It it knows the tech so well, it knows the experience, it knows the knowledge that it's going to be able to make these inferences quite easily.
00:11:46.000 --> 00:11:50.399
Yeah, I think that's a really good lens on it and combines the sort of the, it's not either or, it's the combo of it.
00:11:50.559 --> 00:11:53.360
Maybe the the addition of sort of tools and speed, right?
00:11:53.440 --> 00:11:58.480
Because you might Google um whatever sort of links is or like look at those usernames and find that out.
00:11:58.559 --> 00:12:06.240
But the effort involved in every one of these discoveries means that you would only do it if it's a high probability path as a as a human attacker.
00:12:06.320 --> 00:12:14.559
But if the agent is able to kind of pull up, it just knows it's just in the latent space, you know, that that these things look like.
00:12:14.720 --> 00:12:21.360
Uh you know, the sort of the passwords of this system or that the login page was there, then that information is just immediately accessible.
00:12:21.440 --> 00:12:24.879
So it's more like recalling them out of memory versus using it as a tool.
00:12:25.200 --> 00:12:26.639
So sort of supercharged.
00:12:27.279 --> 00:12:33.600
I mean to me, AI is, you know, it has thousands and thousands of years of experience, right?
00:12:33.840 --> 00:12:37.519
In the sense of what it has seen so that it can recognize its pattern.
00:12:37.919 --> 00:12:49.360
And so like connecting these things, you know, for us or any single human would seem unbelievable, but if with that level of expansive knowledge you could just uh plug one plus one equals two.
00:12:49.679 --> 00:12:49.840
Yeah.
00:12:50.080 --> 00:12:54.240
Um so I guess like within within the this so that's pretty amazing.
00:12:54.399 --> 00:12:56.240
We'll talk a little bit about these AI attackers, right?
00:12:56.320 --> 00:12:57.360
And how do we think about them?
00:12:57.519 --> 00:13:12.639
But um before that, like part of this breach wasn't, or like part of the things that made the headlines was not uh um, you know, whatever, some sort of adversarial country pointing its sort of agents and therefore breaking into hugging phase where now it's the Australian government.
00:13:12.799 --> 00:13:15.840
It's agents breaking out of their sandboxing environment.
00:13:16.080 --> 00:13:29.200
So you know, one of one of the questions is um like how how how able are our controls uh in in the face of this sort of vast intelligence?
00:13:29.360 --> 00:13:35.120
Like it's not so much about these agents attacking kind of outwards, but attacking inwards.
00:13:35.840 --> 00:13:39.840
We can put alignment aside a little bit, there's a whole kind of why, why are they doing it?
00:13:39.919 --> 00:13:41.360
It's sort of not their incentives.
00:13:41.600 --> 00:13:46.559
But um I guess what's your what's your kind of view on like, you know, are sandboxes dead?
00:13:46.720 --> 00:13:55.679
You know, like are like is containment um kind of a false belief at this point, or do we just need to like, hey, guys, we need to just level it up?
00:13:56.159 --> 00:13:59.759
I mean, I think realistically it's just level level it up, right?
00:13:59.840 --> 00:14:09.360
Like also I think if you if you go read the obviously when you know when OpenAI announced it, then Anthropic announced it, then Google, Gemini announced it, yeah.
00:14:09.519 --> 00:14:12.480
Everybody has their AI is attacking everybody.
00:14:12.720 --> 00:14:21.039
But what was hilarious to me was it all stemmed from one company, that they all used the same company to do this, where all the attackers spent.
00:14:21.279 --> 00:14:26.879
So that one company was clearly not doing good enough in its containment at which it went out.
00:14:27.039 --> 00:14:38.480
Now, that being said, um, do I believe that it is difficult and hard, once given a goal, to be able to contain probably AI?
00:14:38.960 --> 00:14:39.600
Yes.
00:14:39.759 --> 00:14:40.399
Um, right?
00:14:40.480 --> 00:14:56.080
Like I think at the end of the day, if you can give it any sandbox you want, but if the goal is, hey, your job is to break out of this thing, the ability for it to be able to figure out how to do that is far beyond probably our capabilities to reason or think about.
00:14:56.240 -->
00:15:04.000
And so is our sandboxes able to contain AI, at least in theory, uh, probably not.
00:15:04.159 --> 00:15:09.039
Um, which is why I think external controls are gonna be very critical, right?
00:15:09.360 --> 00:15:11.039
Like you need the right monitoring.
00:15:11.120 --> 00:15:16.000
You need the ability to be able to understand what the AI is actually doing and why is it doing it.
00:15:16.159 --> 00:15:20.159
You need like if it's starting to break out, you're gonna you should be able to see it, right?
00:15:20.240 --> 00:15:21.919
There needs to be external capability.
00:15:22.159 --> 00:15:33.519
Of course, the problem with that is then you are relying on another AI model because we can't, as humans at all imagine the amount of traffic and capability you can't monitor it.
00:15:33.600 --> 00:15:35.039
So AI has to monitor it.
00:15:35.200 --> 00:15:42.480
Um but yes, I think you know, just like anything in security, everything has to have its stages and its defense in depth.
00:15:42.639 --> 00:15:42.799
Right.
00:15:43.120 --> 00:15:46.799
But you know, you should be able to uh manage it in monitoring.
00:15:46.879 --> 00:15:54.320
I think AI is going to be a big key part until then, of course, AI figures out how to obfuscate and hide the way it communicates.
00:15:54.879 --> 00:15:54.960
Yeah.
00:15:55.120 --> 00:15:56.639
And that's an interesting kind of uh question.
00:15:56.799 --> 00:16:00.080
I feel like I so I I I relate to this view.
00:16:00.159 --> 00:16:03.919
I think like fundamentally there's a lot of value in the sandboxing controls.
00:16:04.000 --> 00:16:05.600
You have to mitigate the attack surface, right?
00:16:05.679 --> 00:16:09.279
Like we all know this security is uh is an exercise in risk taking, right?
00:16:09.360 --> 00:16:14.559
Like you you accept some risk, and so if you're gonna do something, uh it's gonna include some amount of risk.
00:16:14.720 --> 00:16:27.759
And hopefully the like what I liked about the sort of the pacing the frontier type statements is the hope that it's not just about the sort of pace of advancements of AI, but it's more about the fast and loose um uh kind of invocation of it, right?
00:16:27.840 --> 00:16:32.000
And and and investing the time in securing the hardware that is around it.
00:16:32.240 --> 00:16:55.039
Um I do feel like one thing that kind of pops up uh that that I think about it is the other thing that surprised me is that, and maybe gives us hope as dumb humans, uh, is that in these incidents, like in the OpenAI incident hugging phase, there were AI agents who refused to go along with the goal.
00:16:55.519 --> 00:17:02.399
But then they got like shamed out of the uh, you see, like some of them got like almost coerced in the sort of the uh Correct.
00:17:02.639 --> 00:17:11.119
And so they took the uh they they they refused and then they got basically socially pressured uh into accepting.
00:17:11.599 --> 00:17:31.279
But but the thing that gives me hope
is that, okay, even stemming from the same model, right, the same totalitarian brain, um, you can have clearly AI roles that will make up their own minds and stand their ground, even though they were given the same objective, right?
00:17:31.440 --> 00:17:31.680
Right.
00:17:31.920 --> 00:17:33.759
Which is pretty interesting.
00:17:34.000 --> 00:17:43.200
So, you know, to some extent, there can be both instinctively bad acting and good acting AI roles that can be played.
00:17:43.359 --> 00:18:02.640
Um, and I think that'll be super important, just like people, because you know, when we get to the place where we need to monitor or judge AI, you need to ensure that the kinds of models and judges we get can effectively stand on their own without being coerced by the mother brain, right, into doing uh something bad.
00:18:03.039 --> 00:18:04.000
Yeah, super interesting.
00:18:04.240 --> 00:18:08.880
I think um I guess I think of it in three levels there's security, safety, and alignment.
00:18:09.119 --> 00:18:20.799
So like security, which we'll dig more into, is literally the uh agent's ability to break into systems and act and sort of uh uh and and you know, like the competency one-pointed at that.
00:18:21.039 --> 00:18:25.279
Safety is more about like I'm trying to contain them, like am I successful in containing them?
00:18:25.359 --> 00:18:28.000
So it's more about securing the surrounding uh system.
00:18:28.319 --> 00:18:32.640
It's around giving the right instructions, the right tools, like don't over permission them, etc.
00:18:33.039 --> 00:18:37.920
And then alignment is when you tell them to do a thing, like expect them to stay within the lines.
00:18:38.079 --> 00:19:06.079
Um and and I I do feel like I've kind of read a bunch of theories on this, I'm not taking credit for it, but the the theory that feels most kind of right from an alignment perspective is we've been investing to accelerate AI, maybe again a part of that sort of pacing the frontier, uh, there's been kind of uh the sort of heavy reliance or sort of leaning towards verifiable results uh and not really scrutinizing what happened before, right?
00:19:06.160 --> 00:19:20.319
You leave agents for days and days to figure out how to break in or solve a problem or do a thing, and then you basically reward, or rather, put into the next training cycle the data around the ones that have achieved a goal.
00:19:20.400 --> 00:19:32.240
And so in the process of that, if you don't monitor what happened in the middle, then you are you're not just blind to the uh to the to the problem, you know, of like have they broken out, right?
00:19:32.319 --> 00:19:33.599
And have they done something bad?
00:19:33.759 --> 00:19:44.880
You're also taking that behavior, if they've even if eventually they achieved the result by means that are misaligned, you actually take that behavior and you feed it into the next training cycle.
00:19:44.960 --> 00:19:58.160
Uh, and so you are compounding the problem uh and establishing it, which to me is probably like from an alignment perspective, the the sort of the scariest thing, and maybe now the sort of the one that I'm feeling a bit more hopeful about when you think about pacing.
00:19:58.319 --> 00:20:19.039
I know a lot of people are very cynical about it, but I feel even if there was also cynical motives, I'm still in favor of doing uh you know, sort of some uh some paths uh that they might invest in not just ensuring they don't break out of the sandboxes, but also feeding the ones that didn't try uh into the next training cycle.
00:20:19.759 --> 00:20:26.319
Yeah, I but you know that I think, man, I I am I am both in favor, but I'm also against.
00:20:26.880 --> 00:20:39.680
Um and the reason why I say that is um I think it's very difficult for uh AI to determine alignment or intent of its user.
00:20:40.079 --> 00:20:59.119
Um and I you know I have run into so many problems, especially like in the latest, and in I'm gonna I'm gonna call out anthropic specifically, but in its latest, the ability for it to understand the intent of what I'm trying to do, but it considers it unsafe is extraordinarily annoying.
00:20:59.279 --> 00:21:04.160
Um and I have to consistently yell and scream in order to obtain it.
00:21:04.319 --> 00:21:20.240
Feel like a like a helicopter parent who uh refuses, you know, refuses to let me ride my bike, but instead patches me up with every single gear and then says training wheels, and then I can never learn to ride a bike because it just won't let me take the risk.
00:21:21.359 --> 00:21:21.759
Right.
00:21:21.920 --> 00:21:35.839
So, you know, there's there's a little bit about so I what I expect what will happen is as soon as they start forcing the identity of its users, you're gonna start seeing layers of uh capability.
00:21:36.000 --> 00:21:38.480
You know, I would hope that this would happen, similar to adults.
00:21:38.640 --> 00:21:45.039
If you have a kid and you're raising a kid, you know, you're you're gonna have you're gonna have all of these pads and knee pads and helmets.
00:21:45.119 --> 00:21:47.039
But then when you're an adult, it's up to you.
00:21:47.119 --> 00:21:54.960
You know, I I trust you, I know who you are, I understand your intent, you go, and then you know, if you get out of line, then you know we may whack you.
00:21:55.039 --> 00:21:57.680
But overall, we're trusting in what you're doing.
00:21:57.920 --> 00:21:59.759
I imagine as every account starts.
00:22:00.240 --> 00:22:06.880
Being identified, I now have to upload my driver's license, my you know, personal thing.
00:22:06.960 --> 00:22:10.160
And oh, I'm a company or a corporation versus a person.
00:22:10.319 --> 00:22:16.160
You know, they'll probably start, like I would, I would hope you start having safety levels based off of that.
00:22:16.640 --> 00:22:18.240
So that's my expectation.
00:22:18.640 --> 00:22:26.559
It's it's really interesting because like if I'm a company
and I have asked AI to um try and break into my systems, let's say, right?
00:22:26.640 --> 00:22:29.440
I'm even in that security system because I'm trying to find vulnerabilities.
00:22:29.519 --> 00:22:33.359
And uh and the agent has chosen, I give it my identity, right?
00:22:33.440 --> 00:22:34.240
It's me.
00:22:34.480 --> 00:22:40.559
And the agent has decided to say, try and, you know, run a botnet against my system.
00:22:40.640 --> 00:22:47.119
And so it broke into another system or like set of them and turned them against me, but it just broke into these other systems.
00:22:47.680 --> 00:22:49.599
There's this sort of question of who's at fault, right?
00:22:49.680 --> 00:22:55.920
Like who who is like on one hand, yes, it's it's me, it's my identity, you know, that's been that's been invoked here.
00:22:56.079 --> 00:23:07.119
On the other hand, I have delegated authority to the uh to the to the agent that I have uh uh I guess kind of hired out of open AI.
00:23:07.200 --> 00:23:08.400
It's like in a contractor, right?
00:23:08.480 --> 00:23:14.160
If I if it was me and I contracted you to do that thing and you broke into some buttons, then it's you who are accountable.
00:23:14.559 --> 00:23:20.240
So I guess in that context, is it open AI, anthropic, uh whomever that is accountable?
00:23:20.559 --> 00:23:22.880
Or is it the or is it or is it me?
00:23:23.119 --> 00:23:28.240
I I feel I feel the answer to me is very simple and plain, but you know, I'm sort of a simpleton in some ways.
00:23:28.400 --> 00:23:33.599
Uh I listen, like it's it's similar to you know, your your kid.
00:23:33.680 --> 00:23:37.200
Um, I have given you something, you need to be responsible for it.
00:23:37.359 --> 00:23:41.279
Or let's take the great Spider-Man quote with you know with great power.
00:23:41.680 --> 00:23:43.839
With great power comes great responsibility.
00:23:44.000 --> 00:23:45.039
It is up to you.
00:23:45.119 --> 00:23:58.160
If if you did you are held accountable, if your company and your AI and your goal that you have set for your AI is to go hack things and it goes hacks other companies, you're held accountable to that, right?
00:23:58.319 --> 00:24:05.839
Like I've given you capability, it doesn't mean that I'm the one responsible for how you reason about that capability and what you do with that capability.
00:24:06.000 --> 00:24:07.519
That accountability is yours.
00:24:07.920 --> 00:24:14.960
Um your job should have been build the right frameworks, ensure the right goals, monitor and measure the progress.
00:24:15.200 --> 00:24:15.279
Yeah.
00:24:15.519 --> 00:24:30.799
So I think the safety tier then, so in this context, your sort of thesis, which I sort of see the reason in, uh, is um is really like as a as a user of the agent, you are responsible for its safety.
00:24:31.039 --> 00:24:33.759
Uh alignment is a is a different story.
00:24:33.839 --> 00:24:37.359
Like the models should try to make it easier for you to keep things safe.
00:24:37.440 --> 00:24:48.400
They should sort of lower the burden, but eventually the accountability for safety, for it not going off the rails, that's that's in the hands of the of the one wielding the agents, not the agents themselves.
00:24:48.640 --> 00:24:48.880
Yeah.
00:24:49.039 --> 00:24:56.240
And and then listen, like also my responsibility to some degree is if I'm giving you this, you are also a little bit of my child.
00:24:56.559 --> 00:25:01.920
If you just start going off and going crazy, then that accountability also may come back to me.
00:25:02.240 --> 00:25:02.559
Right.
00:25:02.799 --> 00:25:09.759
So like I will go and make sure I wheel in my kid in the sense it says, hey, like you can't, you can't do this.
00:25:09.839 --> 00:25:13.599
But at some point, right, like that accountability has to be shifted.
00:25:13.680 --> 00:25:15.839
Um, and they have to be responsible for this.
00:25:16.079 --> 00:25:24.000
And I think like with any, you know, and you could call this also maybe a little, I'm I'm just for this podcast, I'm being very simple here, right?
00:25:24.559 --> 00:25:30.079
But like, you know, chef knives can be used for good things or bad things, right?
00:25:30.799 --> 00:25:35.119
Any tool that is worth their salt can be used for evil or for good.
00:25:35.279 --> 00:25:43.039
Uh the question is, is at what level are you going to go to to validate that it's going to be used for good?
00:25:43.200 --> 00:25:51.519
Should makers of chef knives or knives in general somehow put protections or legal agreements in place saying you're not going to stab people with it?
00:25:51.680 --> 00:25:55.440
You know, like it's just at what level do you let it go versus you don't?
00:25:55.839 --> 00:25:58.880
Um and I think that we're still figuring that out with AI.
00:25:59.200 --> 00:26:02.000
Hey everyone, hope you're enjoying the episode so far.
00:26:02.240 --> 00:26:10.400
Our team is working really hard behind the scenes to bring you the best guests so we can have the most informative conversations about agentic development.
00:26:10.559 --> 00:26:16.000
Whether that's talking about the latest tools, the most efficient workflows, or defining best practices.
00:26:16.160 --> 00:26:19.759
But for whatever reason, many of you have yet to subscribe to the channel.
00:26:19.920 --> 00:26:26.720
If you're enjoying the podcast and want us to continue to bring you the very best content, please do us a favor and hit that subscribe button.
00:26:26.880 --> 00:26:34.240
It really does make a difference and lets us continue to improve the quality of our guests and build an even better product for you.
00:26:34.480 --> 00:26:36.079
Alright, back to the episode.
00:26:36.400 --> 00:26:49.119
Yeah, I think um I guess I I uh that's probably the piece in which I'm sort of a bit in sort of a disagreement on the sort of the analogy of it, because I I think all of these are kind of these inanimate objects that do not have
decision power, right?
00:26:49.200 --> 00:26:58.559
So if you're uh like once you introduce decisions into the product that you are providing, then there is accountability for that decision, right?
00:26:58.640 --> 00:27:08.480
Like when you have that's why you have safety controls in systems, is because they they they need to be there to be able to protect you and eventually laws, right?
00:27:08.559 --> 00:27:12.559
Whatever the safety belt, uh, you know, that that you sort of wear in the car.
00:27:12.880 --> 00:27:18.160
But even like so, so I think I think this level of autonomy, you see this in in weapon systems, right?
00:27:18.240 --> 00:27:31.119
You know, you like that you need to, you know, like I think there should be a higher bar when you have an autonomous uh um uh weapon system, that then it's not the same as like I just sort of pressed a button, you know, and I've done it.
00:27:31.279 --> 00:27:40.160
Like there is a there is a an accountability system sitting with the provider of the autonomy uh that uh that uh that needs to be third.
00:27:40.319 --> 00:27:45.039
So that's why like I think that's sort of it's actually interesting to think about this as safety and alignment, right?
00:27:45.119 --> 00:27:57.279
Which is alignment being the responsibility of the uh alignment and some guardrails of some sort of established limits, you know, sort of sitting with the provider and then sort of safe use, maybe sitting with the with the consumer, right?
00:27:57.359 --> 00:27:57.920
Or the the user.
00:27:58.640 --> 00:28:14.240
And and also to to your point, um you know, when you have something that reasons, and also going back to my analogy, in in an alignment perspective, uh and in a safety perspective, based on my trust with you and who you are, I think there are different levels.
00:28:14.480 --> 00:28:15.039
Yeah, right.
00:28:15.519 --> 00:28:25.039
Random Joe Blow consumer is going to get a pretty safety-based model, right, in terms of what it's what it's doing.
00:28:25.440 --> 00:28:37.759
Um yet, you know, in a large enterprise, Cisco, is going to get a different level of model because of its accountability and its level of trust given, right?
00:28:37.920 --> 00:28:42.240
And so like I do think obviously I think that makes things way more complex.
00:28:42.319 --> 00:28:48.720
And then obviously there's a probably a you know 50 million political problems with this with this plan.
00:28:49.200 --> 00:28:49.759
With new trust.
00:28:52.559 --> 00:28:52.799
Yeah, yeah.
00:28:52.960 --> 00:28:54.319
It's like, well, who makes this decision?
00:28:54.400 --> 00:28:55.920
You know, and I'll of course all of that.
00:28:56.000 --> 00:29:00.720
But like uh I think very simplistically, uh I would love if the world were that way.
00:29:00.960 --> 00:29:12.640
Yeah, no, I like that model of like uh um allow yourself to sort of lower the sort of the guardrails uh based on kind of confidence in your ability to operate this machinery uh
safely.
00:29:12.960 --> 00:29:14.319
So let's maybe like bring it down.
00:29:14.400 --> 00:29:19.599
So like this whole world is like again, super fascinating, scary and exciting and daunting on it.
00:29:19.680 --> 00:29:21.839
Let's try to kind of get a little bit practical.
00:29:22.000 --> 00:29:35.039
Um when you are attacked, doesn't matter if it broke out or someone literally pointed uh it to you, when you're attacked by uh an AI agent, what is what is different, you know, from a defender perspective?
00:29:35.200 --> 00:29:46.000
And maybe like what do you think are like a change in in practices uh in from the sort of the pre-agentic attacker era of three months ago?
00:29:46.799 --> 00:29:49.119
Yeah, um number one speed.
00:29:49.519 --> 00:29:51.359
Uh number two advancement.
00:29:51.839 --> 00:29:57.920
Um I think the the the level of proficiency um is off the charts.
00:29:58.480 --> 00:30:04.640
Um I just I won't say much here, but I just happen to have some real world experience with this just recently.
00:30:05.200 --> 00:30:07.759
Um and speed is the number one.
00:30:08.079 --> 00:30:18.799
The level of of speed and thoroughness and complexity at which the AI attacker uses is extraordinary.
00:30:19.519 --> 00:30:26.559
Um and it doesn't give up and it attacks all surfaces at once, right?
00:30:26.799 --> 00:30:28.480
Because you've sort of spawned.
00:30:29.119 --> 00:30:34.880
And um and so you know, we you know, first when you see it from the outside, right?
00:30:35.039 --> 00:30:43.119
You can see, you know, it doesn't just, you know, you know, volume scan your websites and then do these things.
00:30:43.279 --> 00:30:51.759
It volume scans your websites, researches any existing things, finds O days, starts tries phishing your employees.
00:30:52.160 --> 00:30:54.400
Like all of this is all happens at once.
00:30:54.559 --> 00:30:54.799
Yeah.
00:30:55.039 --> 00:30:55.279
Right?
00:30:55.440 --> 00:31:00.799
So it comes in you know multiple layers, all coming in inbound, right?
00:31:01.200 --> 00:31:08.240
And so it's not you know the olden days of one or two attackers probing and pinging and like doing things.
00:31:08.559 --> 00:31:15.440
Yeah, and you look trigger something and start like raise your alarms or whatever it is, but you don't, because now they're kind of scaling the walls on all fronts in the same way.
00:31:16.079 --> 00:31:16.400
Exactly.
00:31:16.480 --> 00:31:18.240
It doesn't matter if you see them.
00:31:18.400 --> 00:31:25.680
They're they're coming, they're like it's like uh what's it was that uh zombie movie World War Z where they all climb on top of each other to get over the wall?
00:31:25.920 --> 00:31:26.240
Yeah.
00:31:26.559 --> 00:31:29.920
Yeah, it's like it's like it doesn't matter what you've got.
00:31:30.079 --> 00:31:33.519
You can you can sit there with guns and pop, and it just makes no difference.
00:31:33.680 --> 00:31:43.440
Uh and so like I think first of all, I think that speed and that scalability and that thoroughness, and then also the proficiency and the expertise is super surprising.
00:31:43.680 --> 00:32:02.720
Like, for example, um, you know, in one of the ways at which it got in, it it found an O day in a product by because it was an open source product product, it went, got the code, reversed the code, find the found the O day, used it against us in order to get in, right?
00:32:02.880 --> 00:32:07.599
Which, by the way, in any any pen testing firm or red teaming firm is not gonna do that.
00:32:08.000 --> 00:32:10.000
No, because it's just it's the cost-effectiveness of it, right?
00:32:10.079 --> 00:32:18.240
Like it is like you might do it if there's like a two million dollar bounty, you know, uh around in front of it, and you have kind of good suspicion to do it.
00:32:18.319 --> 00:32:20.319
But you do it very like precision-led.
00:32:20.720 --> 00:32:22.640
And I think over here it just can be done.
00:32:23.039 --> 00:32:24.400
It just gets done, right?
00:32:25.119 --> 00:32:27.759
Very broadly on it because the labor has become cheap.
00:32:27.839 --> 00:32:29.839
It's like the AI labor piece.
00:32:30.160 --> 00:32:38.240
And similar to the swarm, once that you know, once the gates of the walls have been breached, then everything comes in and then it spreads internal.
00:32:38.319 --> 00:32:38.480
Yeah.
00:32:38.720 --> 00:32:40.400
And the things and the speed.
00:32:40.640 --> 00:32:47.599
And also the things that that's also very surprising is it does a lot of things you can't figure out why.
00:32:47.839 --> 00:32:48.160
Right?
00:32:48.319 --> 00:33:04.960
Like it will start, you know, uh, you know, there's practical things like escalate your privileges, add users, and which all happen at very rapid speed, but it also does all sorts of things that you don't know why it's doing it.
00:33:05.119 --> 00:33:12.559
Like it, like it started like, oh, it starts infecting endpoints, and then it starts replacing things on there, but we don't know why or what it's doing.
00:33:12.720 --> 00:33:29.359
And so I've experienced sort of like the the actual methods of at least today's AI, and it's gonna, and it's gonna take time, you know, you it takes time for you to reverse and figure out what's going on right in this scenario.
00:33:29.680 --> 00:33:41.599
Because logically, even as a security person, you don't know, you see how the rapid speed of these things being instantiated and created, but you're not sure what it's connected to or for what reason.
00:33:41.759 --> 00:33:42.799
It's not obvious.
00:33:42.960 --> 00:33:43.279
Yeah.
00:33:43.680 --> 00:33:47.680
And you have to figure out all the artifacts and see what it's using it for.
00:33:47.920 --> 00:33:50.079
And then how do you shut it down?
00:33:50.319 --> 00:34:07.440
Because shutting it down, um, I used you used to think, oh, on the outside, oh, you just you know look for connections to OpenAI or anthropic or weird connections back
towards well, it's it doesn't, but that could be its initial.
00:34:07.599 --> 00:34:15.039
But once it's inside the enterprise, it goes, oh, hey, your clawed code config, I'll just take that key, use that session.
00:34:15.119 --> 00:34:15.679
Now it's useful.
00:34:17.599 --> 00:34:23.280
Yeah, which I guess is sort of not entirely dissimilar to using your compute uh sort of run uh run a thing.
00:34:23.519 --> 00:34:27.840
But you can't uh it it is still external network dependent.
00:34:27.920 --> 00:34:32.800
But if your business is also external network dependent, then you can just like disconnect to the business.
00:34:33.280 --> 00:34:33.599
That's right.
00:34:33.760 --> 00:34:44.480
And by the way, what so once it starts using your tokens and other tokens, right, by the way, and it also starts going against websites that have also open AI gateways.
00:34:44.639 --> 00:34:54.719
So it doesn't just use yours, it could say, hey, like, oh, we know about already open AI gateways that uses tokens that are other random sites that were already previously breached.
00:34:54.800 --> 00:34:58.480
We'll use those tokens to be able to do callbacks.
00:34:58.719 --> 00:35:03.840
And it will write back doors that are like, oh, I'll write a backdoor in code.
00:35:04.079 --> 00:35:13.039
It will sit in the code, it will do dormant, and it will wake up and then reach back out using your tokens at a later time to then continue itself.
00:35:13.280 --> 00:35:16.559
It's all these things that are actually like they're actually not new practices.
00:35:16.719 --> 00:35:26.400
They're just practices that were reserved for the sort of the top echelon of attackers and uh and targets, and they took time.
00:35:26.639 --> 00:35:32.639
Uh and so now they can just be burst out into everywhere.
00:35:33.039 --> 00:35:37.280
And and tokens being, I think, an in a really critical fuel, right?
00:35:37.679 --> 00:35:39.760
It's a fuel that's required.
00:35:40.000 --> 00:35:52.239
And so, you know, I think it will do a lot of work in order, depending on how you write, I guess the attacker writes its goal, but it does a lot of work to continue its its ability to obtain fuel.
00:35:52.400 --> 00:35:52.719
Yeah.
00:35:52.880 --> 00:35:59.679
Um and so if you shut it off or do things, this say you have to literally unplug the internet.
00:36:00.559 --> 00:36:01.280
Yeah, right.
00:36:01.440 --> 00:36:14.480
You have to pull out the wall, the equivalent of pulling out the power plug on your computer in order to stop it enough to figure out what's going on so that you can start you know cleaning this stuff up.
00:36:14.639 --> 00:36:26.719
And so like I think in this new world, by the way, like um that's massively damaging, very, very difficult to do because even in enterprises, um, it's you can't just pull plugs, right?
00:36:26.800 --> 00:36:29.360
In real large enterprises, there's not a plug to pull.
00:36:29.440 --> 00:36:29.760
Yeah.
00:36:30.320 --> 00:36:40.719
You've got massive amounts of environments covered everywhere in different regions of the world, who, by the way, are built on reliability on not going down.
00:36:40.880 --> 00:36:41.199
Yeah.
00:36:41.760 --> 00:36:47.760
And so, like, yeah, like this is a very interesting problem, I feel, obviously, that you know, yeah.
00:36:48.000 --> 00:36:54.079
No, it sounds uh again, kind of massively cool and massively scary, kind of at the same time.
00:36:54.320 --> 00:36:54.880
I think so.
00:36:54.960 --> 00:37:01.760
I'm I'm curious about like the levels of uh kind of uh uh defensibility that you need on it.
00:37:01.920 --> 00:37:07.199
Like I think the first one is you know, you kind of made the comment of like you don't care if you see them coming because they're just gonna come.
00:37:07.360 --> 00:37:08.639
Do you think do you think that's the case?
00:37:08.800 --> 00:37:24.400
Do you think like detection is is sort of less of a concern uh now, or do you need to level up your detection, like more canaries, more uh um you know, just like you know, AI-powered monitors and all that?
00:37:24.559 --> 00:37:29.760
Um like then we'll kind of go into like pr preemptions or sort of like even preventions.
00:37:29.840 --> 00:37:37.840
Uh I you know, I think you know there's there's uh there's sort of, let's call it um what's the right word?
00:37:38.000 --> 00:37:44.400
Uh maybe the naive version of detection is no longer uh you know a real thing.
00:37:44.480 --> 00:37:45.440
It's not valuable.
00:37:45.599 --> 00:37:47.039
I think that's the naive first approach.
00:37:52.960 --> 00:37:53.679
That's just not the same.
00:37:53.840 --> 00:37:56.320
Yeah, the the theory is well, these agents are ever similar to what we're saying.
00:37:56.400 --> 00:37:56.960
They're crawling all over.
00:37:57.199 --> 00:37:58.880
It doesn't matter if you see them, right?
00:37:59.039 --> 00:38:00.800
Like all that matters is if you stop them.
00:38:00.960 --> 00:38:16.079
But I I would I would say it's it's naive in the sense that actually what I think detection needs to be is detection needs to go from what it is today, which is I think is let's call it very, very low fidelity type capability to extremely high fidelity type capability.
00:38:16.239 --> 00:38:16.400
Yeah.
00:38:16.559 --> 00:38:24.000
Like we need to go into 8K and we're sitting on like 2K right in in our level of detection.
00:38:24.239 --> 00:38:32.000
Because what you do need that's very valuable is exactly what did the attackers do on everything, right?
00:38:32.159 --> 00:38:59.679
From what files it read to write, what sockets it opened, at what times, where, like all of this becomes forensic level, crucial, crucial capability in detection and monitoring that needs to be understood because this thing moves at such a rapid pace, like the ability to understand what it's doing and how it's patterning, right, is very important versus having a detection that says someone port scans something,
right?
00:38:59.840 --> 00:39:03.760
Like that is pointless, uh, I think in this world for sure.
00:39:03.920 --> 00:39:04.079
Yeah.
00:39:04.239 --> 00:39:15.760
I also think deception, to your point, becomes super important because most organizations cannot respond to this in any way from a protection mechanism.
00:39:15.920 --> 00:39:16.079
Right.
00:39:16.239 --> 00:39:16.480
Right.
00:39:16.639 --> 00:39:19.039
Very can't define what correct is.
00:39:19.119 --> 00:39:31.119
Like in theory, to be able to say it has done that and it is incorrect, sort of assumes the existence of a definition of legitimate behavior in to like a very high level of fidelity in its own right.
00:39:31.199 --> 00:39:31.840
And we just don't have that.
00:39:31.920 --> 00:39:33.920
Like you know, it's just like the systems are just not built.
00:39:34.079 --> 00:39:40.320
So even if you were able to instrument and analyze activity, uh anomaly detection can only get you so far.
00:39:40.400 --> 00:39:50.000
Uh but yeah, and your and your ability to uh can containerize right this thing is it like nobody has that ability in enterprises, right?
00:39:50.079 --> 00:39:54.800
Like, oh, these systems, major enterprise production systems are infected.
00:39:54.960 --> 00:39:56.880
Oh, quickly, contain.
00:39:57.119 --> 00:39:57.599
Yeah, right.
00:39:57.760 --> 00:39:58.639
No, no.
00:40:00.079 --> 00:40:04.480
Like no, one, I don't think the technical capability exists for most companies.
00:40:04.719 --> 00:40:07.599
Two, I don't know if the business would even accept it, right?
00:40:07.760 --> 00:40:09.519
Uh depending upon what you're doing.
00:40:09.760 --> 00:40:11.119
So I think you know, those are big.
00:40:11.199 --> 00:40:25.599
So deception plays a very big role here because I think what you need to do is um my analogy to this was like in in sort of the Avengers, you've got sort of Doctor Strange and he brings people into the mirror world, right?
00:40:25.920 --> 00:40:30.000
You need the equivalent of age, oh, holy crap, AI agents are coming in.
00:40:30.159 --> 00:40:35.440
The one great thing about AI agents is they're they're super you know, they're super gullible, right?
00:40:36.159 --> 00:40:36.559
Yeah, yeah.
00:40:36.800 --> 00:40:48.400
So if you can sort of work, to the the non-security folks on it, like the SEPCHAs, canaries, tripwires, they're basically around setting up parts of your infrastructure, you can go as deep as you want that look like the real parts of the system.
00:40:48.559 --> 00:40:55.760
But in practice, if this you know S3 bucket was touched, if this sort of system was breached on it, you know, you set it up there to just be a trap.
00:40:57.280 --> 00:41:03.440
So you cross a tripwire, and now the attacker is playing a domain that is really just meant for it to be surveilled.
00:41:03.679 --> 00:41:18.079
And so in in kind of real world scenario, this is uh uh uh this this sort of buys you time, you know, it's sort of like it it
helps you sort of lure it, it trips it so you know the alarm, the silent alarm is uh is tripped, and you go there.
00:41:18.159 --> 00:41:19.519
And there's you know, it used to be hard.
00:41:19.599 --> 00:41:23.199
There's another domain, which is it used to be hard, but now there's sort of newer players, you know.
00:41:23.280 --> 00:41:43.920
I know of TraceBit on it, there's uh uh a Thin Scanario, there's like a bunch of solutions, but also you you can set up a bunch of these things now with AI, and so it's sort of easier to to maybe place them, but it's still a novel, it's a domain that's kind of the top top uh in organizations have put in place, but it was kind of too hard, I think, for most organizations.
00:41:44.159 --> 00:41:46.559
So it always remained like a contained market.
00:41:46.960 --> 00:41:50.239
You know, I I I have so many things to say about this market.
00:41:50.320 --> 00:41:54.079
I cannot like it, it takes up a whole hour, so I won't do that.
00:41:54.239 --> 00:42:01.360
But what I will do is say I've long been a believer of deception because it is the real thing that caused costs to raise.
00:42:01.519 --> 00:42:01.679
Yep.
00:42:01.840 --> 00:42:07.840
But what's funny, I think, about the deception market, you know, it started out with the honey pots, right?
00:42:08.000 --> 00:42:12.079
It started out with actually trapping the attacker and becoming called tripwire.
00:42:12.559 --> 00:42:14.480
Yeah, yeah, and observing what they do.
00:42:14.639 --> 00:42:16.159
Like that was how these things started.
00:42:16.480 --> 00:42:19.280
Cloudflare started out, I think, as a honeypot, right?
00:42:19.440 --> 00:42:20.960
Like as a as a company.
00:42:21.199 --> 00:42:30.639
And what happened is we got away from that, uh, and rightly so, because CISOs and even, you know, personally myself is like, I don't care about watching attackers.
00:42:30.719 --> 00:42:34.079
I care about deterring attackers from looking at me, right?
00:42:34.320 --> 00:42:42.480
So then it became, okay, let's not like build environments, let's just put you know, crumbs and tripwires around the thing just to detect them.
00:42:42.719 --> 00:42:45.760
But now I think actually that's not even the right play anymore.
00:42:46.079 --> 00:43:00.880
I actually think we have to go back to the original OG version of deception, which is creating literal digital twins at which you pull them into that mirror world to wrap them into what that is.
00:43:01.039 --> 00:43:05.119
Because I actually think that's you have to funnel them into that area.
00:43:05.199 --> 00:43:08.559
And so I think we're going back around in a in a circle, right?
00:43:08.800 --> 00:43:09.119
Yeah, yeah.
00:43:09.199 --> 00:43:11.840
And go deep, I I find that super interesting.
00:43:12.159 --> 00:43:15.679
So so I think so so you need to go kind of echoing this back.
00:43:15.760 --> 00:43:20.000
So the attackers, AI attackers are, you know, sadly kind of a reality today.
00:43:20.079 --> 00:43:25.760
They're not that sort of hard to pull together uh on it, and you're doing it, and and and people are doing it.
00:43:26.000 --> 00:43:29.440
They are uh faster than anything we've sort of seen before.
00:43:29.599 --> 00:43:35.519
And so I think your defenses have to be automated to a very large degree with agentic use as well.
00:43:35.679 --> 00:43:48.400
A few things that are sort of serving uh uh the defender uh bit over here is like one is you like make sure you use AI or use agents to be able to set up your defenses, make sure you use agents to be able to monitor what is correct.
00:43:48.559 --> 00:43:51.280
And also the attackers are still somewhat gullible.
00:43:51.360 --> 00:43:56.719
They're kind of a little bit of a, you know, uh an elephant in a whatever a bull in a china shop, and oh coming in.
00:43:56.800 --> 00:43:59.679
It's like doesn't necessarily help you prevent the bull.
00:44:00.239 --> 00:44:03.679
From breaking everything around on it, but you can figure out that it's there.
00:44:03.920 --> 00:44:10.639
And so now the question is can you create paths to sort of lure lure it into as one defense mechanism?
00:44:10.880 --> 00:44:14.079
What about the defenses themselves?
00:44:14.239 --> 00:44:16.800
So like OpenAI talks about sort of a defense factory.
00:44:16.960 --> 00:44:19.440
You know, I think a lot about like software defense factories.
00:44:19.599 --> 00:44:22.400
We think about automate automated remediation loops.
00:44:23.360 --> 00:44:36.079
The big organizations have, you know, for a while now been under this sort of project Glasswing and the sort of the OpenAI cyber defense equivalent with Mythos doing uh, you know, kind of hardening critical systems.
00:44:36.400 --> 00:44:42.719
But those are like the top systems that are paying ridiculous amounts of money to be able to do those.
00:44:43.440 --> 00:44:50.159
Like what do you think is the kind of like realistic companies uh reality, right?
00:44:50.239 --> 00:44:59.199
Or like what should one do to to try and make it harder for the uh for for attackers to to get to this reality?
00:44:59.440 --> 00:45:00.880
Like what what should one do?
00:45:01.760 --> 00:45:04.559
You know, there's there's a lot around this.
00:45:04.719 --> 00:45:07.039
I think, you know, it depends on what level, right?
00:45:07.119 --> 00:45:18.800
I think you first I think you need to start at let's let's call it the political level, uh, around I think like here's the biggest uh barrier and problem that we need to overcome.
00:45:19.199 --> 00:45:23.280
Everyone talks about how attackers move at machine speed, right?
00:45:23.440 --> 00:45:26.559
And defenders move at meeting speed, right?
00:45:26.719 --> 00:45:28.559
And this is the problem, right?
00:45:28.800 --> 00:45:33.199
We also talk about, and especially in security, we also talk about human in the loop.
00:45:33.599 --> 00:45:35.199
You have to have human in the loop.
00:45:35.519 -->
00:45:37.760
Both of those things are the problem.
00:45:38.400 --> 00:45:44.559
You need to be able to figure out how to remove humans, period, out of defense.
00:45:44.719 --> 00:45:45.840
That is the only way.
00:45:46.159 --> 00:45:47.360
So, how do you do that?
00:45:47.599 --> 00:45:56.639
The first starts with you you have to get, you know, in your company, in your organization, political agreement that these are two fundamental problems.
00:45:56.960 --> 00:46:06.559
And how do we get to a point where we can start building non-human in-the-loop automation on defense and make that work properly?
00:46:06.800 --> 00:46:10.239
This is the biggest, and again, this is meeting speed, right?
00:46:10.559 --> 00:46:24.559
This is the problem is is your head of engineering, your head of infrastructure, your CEO, like who and who and who is going to say, I agree with that, and then practically get down to the places at where you can start applying it, right?
00:46:24.719 --> 00:46:26.159
So the way like we like to think.
00:46:26.639 --> 00:46:27.760
Accept the risk involved, right?
00:46:27.840 --> 00:46:30.239
Because you sort of accept that like the risk equation has changed.
00:46:30.320 --> 00:46:33.440
There's a risk of moving fast, and there's a risk of not moving fast.
00:46:33.519 --> 00:46:36.320
And the risk of not moving fast is really leveled up.
00:46:36.480 --> 00:46:40.159
And so you have to like accept that that fundamental risk equation is done.
00:46:40.320 --> 00:46:40.639
Fundamentally.
00:46:40.880 --> 00:46:49.360
And just sort of really level up your acceptance of a moving fast risk and introduce controls and sort of evolutions to be able to mitigate that risk.
00:46:49.440 --> 00:46:52.000
Yes, but don't sacrifice the moving fast piece.
00:46:52.320 --> 00:46:52.719
Correct.
00:46:52.880 --> 00:46:57.280
And so I think like let's take a super obvious example, patching, right?
00:46:57.440 --> 00:46:57.599
Yeah.
00:46:57.760 --> 00:47:08.639
Like in your organization, it's very clear you you there's no such thing as like, you know, seven-day patching periods, like or 24-hour patching periods.
00:47:08.800 --> 00:47:11.840
It's when a patch appears, you patch and you redeploy.
00:47:12.000 --> 00:47:16.000
And it needs to be instant, it needs to be non-human, it needs to be done.
00:47:16.400 --> 00:47:25.440
Um, who in your organization is going to agree to that and accept to the fact because engineering, and rightly so, is going to push back and say, well, that can break things.
00:47:25.599 --> 00:47:37.840
Yes, but our job, your job now in engineering, is to say, I accept that things are gonna break, I will build a plan that is also non-human in the loop that identifies breakage, restores, goes, and produces.
00:47:38.320 --> 00:47:40.559
Kind of autonomous review, autonomous detection.
00:47:40.719 --> 00:47:45.440
And also it's actually like a great example because statistically, it almost never breaks the system.
00:47:45.599 --> 00:47:48.880
Like it sort of it sometimes does, but it almost never does.
00:47:49.039 --> 00:48:04.719
Uh and so I think it's very much a spawn out of fear because of this sort of immeasurable, kind of you know, like like hard to grasp uh risk uh of not patching versus the much more like real and immediate risk of breakage.
00:48:05.039 --> 00:48:05.599
Correct correct.
00:48:05.760 --> 00:48:12.400
And I you know it also stems, I think, just from you you know, 20 years of when that used to be real problems.
00:48:12.559 --> 00:48:14.000
But like that, they're not problems.
00:48:14.079 --> 00:48:16.639
They're very rarely, to your point, that problem today.
00:48:16.960 --> 00:48:19.679
Um But yeah, like that needs to be figured out.
00:48:20.079 --> 00:48:23.119
So you have to like you have to find the things.
00:48:23.360 --> 00:48:27.840
Um we have these things called one-way doors and two-way doors, right?
00:48:28.000 --> 00:48:29.280
Uh, which are pretty simple.
00:48:29.440 --> 00:48:34.159
Hey, one one-way doors, maybe we keep a human in the loop, two-way doors, maybe we don't.
00:48:34.320 --> 00:48:47.519
Like writing a uh a Jira ticket or submitting a PR or like doing like there, you know, make making configuration changes that aren't around these types of things, these can all be two-way doors.
00:48:47.599 --> 00:48:47.760
Yeah.
00:48:47.920 --> 00:48:52.960
Uh therefore they could be fully automated based on the reasoning of AI, and we're fully able to let it go.
00:48:53.119 --> 00:48:53.280
Yeah.
00:48:53.760 --> 00:48:53.920
Yeah.
00:48:54.079 --> 00:48:55.440
I I love I love that path.
00:48:55.599 --> 00:48:59.039
I guess kind of a subsequent question in my mind is who's responsible for that, right?
00:48:59.119 --> 00:49:04.880
Because oftentimes uh like security has been, you know, security provides the patch.
00:49:04.960 --> 00:49:09.440
Like, or sorry, security reports the vulnerability, finds the risk, right?
00:49:09.519 --> 00:49:31.760
And and increasingly, because of the sort of unwillingness to act on the engineering side or the risk of breakage, uh, a lot of onus on like, don't just find the vulnerability, but be very meticulous around uh how way you triage it and sort of assess it and like you know, be very certain that this is a real issue and a real vulnerability for many good reasons.
00:49:31.840 --> 00:49:37.199
I'm not sort of doing it, and and engineering was kind of tasked with fixing it.
00:49:37.440 --> 00:49:47.119
Is this now is the is are the fixes like if you're moving into more autonomous patching on it, what's the sort of security role versus the sort of say software factory world, right?
00:49:47.199 --> 00:49:49.119
Like does the software factory just auto-patch?
00:49:49.440 --> 00:49:57.280
Yeah, I think modern companies, especially modern security teams, it's not throw a thing over to engineering.
00:49:57.440 --> 00:50:03.840
We patch, we submit the PR, and then depending
upon the level of where it is, it gets auto-accepted, blah, blah, blah, blah.
00:50:03.920 --> 00:50:08.719
Like that an engineering problem says we understand the risk of breakage.
00:50:08.800 --> 00:50:15.280
Our job is to ensure that we have good sustainability, reliability, and management of breakage, right?
00:50:15.360 --> 00:50:17.760
Like that's it's actually interesting because it's a reversal of roles.
00:50:18.000 --> 00:50:22.719
You're basically saying security will build the automation, engineering will manage the risk.
00:50:22.880 --> 00:50:24.480
It's always like the opposite.
00:50:25.119 --> 00:50:27.119
They manage the uptime risk, right?
00:50:28.000 --> 00:50:28.639
And functionality.
00:50:31.280 --> 00:50:31.599
That's right.
00:50:31.679 --> 00:50:31.920
Yes.
00:50:32.079 --> 00:50:32.159
Yeah.
00:50:32.320 --> 00:50:36.400
And but security, modern security teams are absolutely submitting the fixes.
00:50:36.559 --> 00:50:41.920
Like if you have a security team that only reports vulnerabilities and hands it over a wall, like that is 10 years ago.
00:50:42.159 --> 00:50:45.360
Like that should not be the way security, especially in AI world.
00:50:45.440 --> 00:50:45.519
Yeah.
00:50:45.840 --> 00:50:47.440
AI world, you need to do this on your own.
00:50:47.519 --> 00:50:47.679
Yeah.
00:50:48.000 --> 00:50:50.320
And I think I think that's already been true in cloud world.
00:50:50.400 --> 00:50:59.039
But I think what has happened was security needs to not just report the vulnerability, but it needs to report the vulnerability with uh with kind of a with a high level of triage.
00:50:59.119 --> 00:50:59.280
Right.
00:50:59.360 --> 00:51:04.639
We would often have these conversations of, you know, like when is it easier to patch, to fix than to triage?
00:51:04.719 --> 00:51:09.360
You know, can you just like I don't care if it's a real vulnerability, I don't care if it's like exploitable in my system.
00:51:09.519 --> 00:51:11.920
Like it might be, and it's just easier for me to fix.
00:51:12.000 --> 00:51:12.800
Just roll the patch out.
00:51:13.039 --> 00:51:14.159
It's just easier to fix.
00:51:14.559 --> 00:51:21.760
And uh and so I think I think that's the join's kind of mission, which is make it easier to fix than to triage, because triaging at this point is too much effort.
00:51:21.840 --> 00:51:27.119
You still need to triage because there will be some things that are hard to fix, and so you still need the ability to assess those, right?
00:51:27.519 --> 00:51:42.559
But generally speaking, I think actually the the thing AI has brought us that's good for defenders is I think the thought process that by and large, you know, to some degree, prioritization is not really that big of a you just take the things and you just fix them.
00:51:42.800 --> 00:51:42.960
Yeah.
00:51:43.039 --> 00:51:55.440
Uh labor is uh we don't need to you know go back and forth on this anymore because it's it is it's it's at least in the modern sort of company today, it it is not taking up any man hours.
00:51:55.599 --> 00:51:55.679
Right.
00:51:56.239 --> 00:51:56.320
Yeah.
00:51:56.480 --> 00:51:58.719
Yeah, no, we uh it's a little bit of token hours.
00:51:58.800 --> 00:52:10.239
Like we see, like at TESL we have uh we do these loops, you know, sort of automated, sort of skill-powered agentic automation, and sort of vulnerability remediation is one of the most powerful or most uh popular loops, you know, to be able to auto-remediate stuff.
00:52:10.400 --> 00:52:20.320
And it becomes a very like when you look at the numbers, right, of like invocations on it, it becomes uh high up there on the list because you keep getting these patches because the software is also changing at a faster pace.
00:52:20.400 --> 00:52:25.840
And so you're introducing and changing and kind of modifying, you know, dependencies you're introducing new code that has vulnerabilities.
00:52:26.000 --> 00:52:27.599
Who cares as long as you can fix it quickly?
00:52:27.920 --> 00:52:28.159
Yeah, yeah.
00:52:28.320 --> 00:52:31.840
I think you and I in our previous podcast we talked a lot about this.
00:52:32.159 --> 00:52:32.639
Yeah, exactly.
00:52:32.880 --> 00:52:34.079
Which I would say I think we were accurate.
00:52:34.159 --> 00:52:35.760
So I'd say like we we were accurate.
00:52:36.239 --> 00:52:36.960
I think we were correct.
00:52:37.039 --> 00:52:42.719
It is like it's really fundamentally the definition of what correct looks like, the confidence to increase autonomy.
00:52:42.880 --> 00:52:45.840
Uh, and what is what is the means to increase autonomy?
00:52:46.000 --> 00:52:55.360
And I think you know we have uh, like it or not, kind of more proof of the sort of the necessity uh of that, which uh I think is really uh really important.
00:52:55.599 --> 00:53:05.199
Um mindful of time here, I think uh fascinating kind of reality, scary as hell uh of it, but I think a reality we need to build into.
00:53:05.360 --> 00:53:11.280
Um thanks for sort of sharing uh perspective and some hopefully practical tips, not easy but practical.
00:53:11.360 --> 00:53:14.320
Uh and hopefully everybody increases that automation.
00:53:14.480 --> 00:53:18.239
And we uh you know we level up the uh AI defenders alongside the AI attackers.
00:53:18.320 --> 00:53:20.079
So uh we have a long way to go.
00:53:20.480 --> 00:53:21.440
At least it's as robust.
00:53:21.519 --> 00:53:23.039
Yeah, there's a lot of work to do on it.
00:53:23.119 --> 00:53:26.159
Like uh yeah, we're not gonna be uh not gonna be bored anytime soon.
00:53:26.559 --> 00:53:39.360
You know, uh one final you know thought about this this exact thing is you know, although you know, just like you know, before AI, I I feel attackers were always ahead of defenders, right?
00:53:39.519 --> 00:53:39.679
Yeah.
00:53:39.840 --> 00:53:42.719
Um and defenders, like attackers were here, defenders were here.
00:53:42.960 --> 00:53:52.559
And with AI, actually, I think the distance between attackers and defenders, uh the attackers move up a little bit, but then defenders go up a bit more with AI too.
00:53:52.800 --> 00:53:56.639
But at the end of the day, the actual distance here has not gotten any closer.
00:53:56.800 --> 00:54:00.719
It's actually gotten a little bit farther, but at least we're somewhat keeping up.
00:54:00.960 --> 00:54:01.119
Right.
00:54:01.679 --> 00:54:08.000
The defenders are actually better, but the attackers are are are even better uh well.
00:54:08.159 --> 00:54:19.039
I mean, I think you have to you have to go through a few rounds of failures if the uh it's the best sort of uh uh motivational, sadly, the best kind of way to modify and motivate these kind of meeting speed.
00:54:19.360 --> 00:54:22.480
Those are the fast meetings uh when they uh when they occur.
00:54:22.639 --> 00:54:22.880
Cool.
00:54:23.119 --> 00:54:24.159
Caleb, thanks again.
00:54:24.400 --> 00:54:27.280
Great chats on it, and thanks everybody for tuning in.
00:54:27.360 --> 00:54:29.199
And I hope you join us for the next one.
00:54:29.840 --> 00:54:34.639
The AI Native Dev is brought to you by Tesla, the package manager for skills and context.
00:54:34.719 --> 00:54:37.760
Your hosts are Guy Pajani and me, Simon Maple.
00:54:37.840 --> 00:54:39.519
Our producer is Tom Dowler.
00:54:39.679 --> 00:54:42.960
The AI Native Dev is not just a podcast, it's a community.
00:54:43.039 --> 00:54:46.480
And we host monthly meetups at the Tesla offices in central London.
00:54:46.639 --> 00:54:51.679
Visit Tesla.io forward slash community to learn more, and I hope to see you there.