WEBVTT
00:00:00.160 -->
00:00:07.919The dirty secret of the entire security industry is there's been way more vulnerabilities that have gone unexploited than have been exploited.
00:00:08.080 -->
00:00:14.960And so most organizations, targets of opportunity that have not actually been targeted, it means they can just be relentless.
00:00:15.119 -->
00:00:19.679They can scale the monetization of those attacks so there's just no place to hide anymore.
00:00:19.839 -->
00:00:28.079AI-driven attackers, they're going to be relentless in finding that one misconfigured system, otherwise wonderfully configured cyber hygiene.
00:00:29.600 -->
00:00:32.799If AI has ever made you stop and think, wait, what is happening?
00:00:32.960 -->
00:00:34.079You're not alone.
00:00:34.320 -->
00:00:37.679I'm Mo, and I'm a security researcher asking the same questions.
00:00:37.920 -->
00:00:51.920On Curiouser and Curiouser, we're having open conversations with experts, researchers, and leaders working at the edge of this space, talking through how AI is taking shape, what's shifting, and how people inside the work are thinking about it as it happens.
00:00:52.159 -->
00:00:55.520So join us and listen in as the conversation takes shape.
00:00:56.719 -->
00:01:00.640Hello, hello, and welcome back to Curiouser and Curiouser.
00:01:00.719 -->
00:01:03.039Uh, really excited for today's guest.
00:01:03.200 -->
00:01:09.599Uh he's got a really cool background, um, but as usual, I do not want to skewer it, so I'll let him introduce himself.
00:01:09.680 -->
00:01:14.959But today we've got Phil Venables, who is a partner at Ballistic Ventures.
00:01:15.120 -->
00:01:17.760So, Phil, thank you so much for being on today.
00:01:18.159 -->
00:01:19.200Yeah, pleasure to be here.
00:01:19.359 -->
00:01:21.280So, yeah, just a bit of background about me.
00:01:21.359 -->
00:01:24.560So I've um uh been doing cybersecurity for a long time.
00:01:24.719 -->
00:01:30.480I initially started as a software engineer many decades ago, but got into doing security in various forms.
00:01:30.640 -->
00:01:38.239Um was a longtime chief information security officer at Goldman Sachs, then uh chief operational risk officer and a board director.
00:01:38.480 -->
00:01:47.359Spent the past five years at Google as the first CISO for Google Cloud and uh and ran security engineering for Google's technical infrastructure.
00:01:47.439 -->
00:01:51.200And yeah, now as you said, I'm a partner here at Ballistic Ventures.
00:01:51.280 -->
00:01:57.359So we uh uh we invest in Seed and Series A and other early stage cybersecurity companies.
00:01:57.519 -->
00:02:01.920We've got a great portfolio across uh across many different segments of cybersecurity.
00:02:02.000 -->
00:02:03.920So looking forward to the discussion today.
00:02:04.159 -->
00:02:07.120Yeah, no, I mean you guys definitely have a very cool portfolio.
00:02:07.200 -->
00:02:13.520I mean, I've been on one of your podcasts, and again, all the topics that uh y'all cover at ballistics are super cool.
00:02:13.680 -->
00:02:20.159Um, and I think you're probably one of the best people suited to like, I guess, talk about this space as well.
00:02:20.319 -->
00:02:29.680When we think about AI and like everything that AI is doing, especially in the security space, uh, there's just like so much noise, and it's really hard to just cut through it.
00:02:29.840 -->
00:02:40.000So that's like kind of how I want to start because you've seen this from a lot of places and angles where most people haven't, um, from Google infrastructure, um, even like some of the advisory boards you've been on.
00:02:40.240 -->
00:02:51.439So, what's kind of like the vantage point that you're seeing for um AI security from like what is all the noise that people are really just like getting bogged down by um versus what is actually helpful?
00:02:51.759 -->
00:03:03.360Yeah, so there's as with any massive technology shift, just like we've seen in prior shifts of internet, mobile, cloud, AI is probably an even more pervasive shift.
00:03:03.439 -->
00:03:07.599And when you look at how it impacts security, you've got to unpick it a little bit.
00:03:07.759 -->
00:03:31.520So there's the security of AI, and so that's all of the things that security and risk teams are doing to make sure that their organizations, when they deploy AI for business purposes, that they're doing that in safe, secure, regulatory, compliant, managed ways that that manage all the risks, not just the security risks of those AI AI deployments.
00:03:31.680 -->
00:03:46.319Um, and then you've got the you know security uh as delivered by AI, so AI for security, and this you see across everything from software security, security operations, a whole range of different things.
00:03:46.560 -->
00:03:53.520And then finally, you've got this broader impact of how AI is affecting the entire security landscape.
00:03:53.599 -->
00:04:00.719And I think, as you know, that's what's dominated the headlines for the past few months with the so-called mythos moment.
00:04:00.800 -->
00:04:21.040Although that's a little bit of a false moment in time because you know, the quarters and year before that, everybody in the security community was seeing and realizing the profound effect that AI models, particularly models trained for coding, could have on finding vulnerabilities and chaining vulnerabilities together for a tech.
00:04:21.360 -->
00:04:30.800So, you know, one element of this is we've got this tidal wave of vulnerabilities that is hitting us because the models are so good at finding vulnerabilities.
00:04:30.959 -->
00:04:33.759Um, but that's kind of a short-term thing and a long-term thing.
00:04:33.920 -->
00:04:36.000So the short-term thing is quite worrying.
00:04:36.240 -->
00:04:43.920Long-term, though, everybody is applying those models to their own code code base to find and fix things at rates that we've not seen before.
00:04:44.000 -->
00:04:46.720So I think ultimately that could be a good outcome.
00:04:46.959 -->
00:04:58.560Uh secondly, though, which I think is largely going underreported, but for me is more worrying is the extent to which attackers are now using AI to industrialize what they're doing.
00:04:58.800 -->
00:05:06.639So they're using attackers, they have always been resource constrained, and so there's always been more targets that they've not exploited than they have exploited.
00:05:06.800 -->
00:05:14.879Now in AI, just like everybody else, they can industrialize and scale and 10x or 100x of volumes of attacks they can put together.
00:05:15.040 -->
00:05:17.040So that's going to be the really worrying thing.
00:05:17.199 -->
00:05:23.199There's kind of no room and no place for organizations with weak security to hide anymore.
00:05:23.360 -->
00:05:28.560And then finally, you've got what you might describe as a quest for authenticity.
00:05:28.639 -->
00:05:42.160So you've got um fakes, fake workers, fake content, fake brands, all of this other stuff that's driving us as consumers and businesses and governments to want to know what is authentic versus not.
00:05:42.319 -->
00:05:45.199So that's kind of a kind of the grand tour of everything's going on.
00:05:45.279 -->
00:05:49.920So you can see why everything seems to be changing all at once, because it because it actually is.
00:05:50.079 -->
00:05:50.399Yeah.
00:05:50.560 -->
00:05:59.360And you touched on a lot of like really, really big things too, from like the attackers being able to catch up in these capabilities really fast.
00:05:59.519 -->
00:06:01.439I mean, if you just look very recently, right?
00:06:01.680 -->
00:06:11.199Fable five came out, and the only difference between Fable V and the most recent Mythos release is a couple of safeguards and and uh you know protections, right?
00:06:11.439 -->
00:06:21.600So it's like the the very thin line in the sand between uh an attacker's capabilities and what is actually frontier uh continues to get smaller and smaller.
00:06:21.680 -->
00:06:28.639Um, even as we continue to see like open source obliterated models um start to get used across open source attack platforms.
00:06:28.959 -->
00:06:38.720So really like just the ability and the scale that uh attackers are able to just go and productionize is really, really extreme.
00:06:38.959 -->
00:06:45.920On the other side, well, and and and as well, you know, when you look at you know, mythos is not the only model with these capabilities.
00:06:46.000 -->
00:06:49.279You've got Codix, you've got Gemini, you've got others.
00:06:49.439 -->
00:06:55.279And as you point out, over time, more and more of the open models are going to have more of these capabilities.
00:06:55.519 -->
00:07:06.319Certainly, when you look at many of the mythos discovered vulnerabilities, it's been clear after the fact that other lower-end models can and did also discover those when you apply them.
00:07:06.399 -->
00:07:23.040And so I I think anybody that's basing a sense of security on the restrictions in advanced models or the restricted availability of the advanced model is going to be disappointed because ultimately the cat's out of the bag, everybody's gonna have this capability, to your point.
00:07:23.279 -->
00:07:23.600Yeah.
00:07:23.839 -->
00:07:34.639Um, there was like early research done last year, and it continues to be done every time a new model is released, that um researchers continuously show that with open source models and the right type of reasoning, right?
00:07:34.800 -->
00:07:49.279Being able to like actually walk a model through how to go and do this, um, you eventually train these open source models to go and perform these mythos type um, you know, mythos level uh exploits and these chain of thought and chain of reasoning.
00:07:49.439 -->
00:07:57.040So really it seems like the reasoning layer is the big differentiator that is continuously getting smaller and smaller as these models just get better.
00:07:57.199 -->
00:08:01.759And the cost of that is really, again, it's becoming lower and lower.
00:08:01.920 -->
00:08:08.639Um, when I looked over the weekend and I saw like Fable was out, my Opus too um limits were now doubled, right?
00:08:08.800 -->
00:08:11.600Because they were like, oh, well, Opus is now gonna get cheaper.
00:08:11.680 -->
00:08:15.040Um, but that's because Fable's now out and that's the more expensive one.
00:08:15.120 -->
00:08:22.399Um but a couple months ago we saw that again, you know, Sonnet is now like the cheapest one, where at one time it was the most expensive thing to run.
00:08:22.639 -->
00:08:30.560So these costs, um especially from producing code, um, doing reasoning on these really complex problems, it's getting a lot cheaper and easier.
00:08:31.040 -->
00:08:33.919Um, but the vulnerability density isn't really going with it.
00:08:34.159 -->
00:08:41.360So um, we're not only just seeing like more code, but we're seeing way more surface area that attackers have to like kind of pick at.
00:08:41.519 -->
00:08:45.039And it's just like there's so much to attack now.
00:08:45.120 -->
00:08:50.399Uh, I'm wondering like, uh like our practices, they already don't cover most of these things.
00:08:50.559 -->
00:09:06.399Like, um, I've always said that AI is challenging the foundations of the organization uh security posture, where if your organization did not have that great of a security posture or was not ready in most cases, this is really going to test a lot of those foundational practices.
00:09:06.720 -->
00:09:16.799So I guess does that concern you at all with how fast the threat landscape is growing, how cheap it is to perform these attacks, and um how easy it is to scale them?
00:09:16.960 -->
00:09:20.399So, on the final point you made, so I I agree with that.
00:09:20.480 -->
00:09:30.320So, Google's developer operations research analysis, uh, this thing called Dora, not to be confused with the European Digital Operational Resiliency Act.
00:09:30.480 -->
00:09:33.519So the Google's DORA team did a lot of research on this.
00:09:33.600 -->
00:09:49.919And and the conclusion is kind of obvious in hindsight, but it's good for it to be founded on research that if you take AI-driven software production into an organization that's got quite unmanaged and chaotic software development lifecycles, you're going to get chaos amplified.
00:09:50.159 -->
00:10:02.399If you deploy AI-driven or agency-driven software production into an organization that's got quite a well-controlled software production pipeline and controlled build and testing processes, you get productivity amplified.
00:10:02.480 -->
00:10:13.039And so this really does shine a light on the organizations that have yet to get their software production under control with testing and security and quality assurance and all the things that we expect.
00:10:13.279 -->
00:10:31.840I think again, when you come back to the notion of we're going to generate more code with all of this, absolutely the the amount of software and the amount of backlog most organizations have in their desire to produce software and new systems is now being met and if not exceeded by the capability of some of the models.
00:10:32.000 -->
00:10:34.960So we're going to get an enormous amount more software.
00:10:35.200 -->
00:10:45.200I think what is not entirely clear yet is whether in that software there's going to be a greater or lower density of vulnerabilities.
00:10:45.360 -->
00:10:49.120So some models are better at producing secure code than others.
00:10:49.440 -->
00:11:07.679Most organizations, back to that point of the software development process, are getting better at post-training or prompting models to generate secure code using the libraries they expect and then not bringing in any unauthorized or extraneous um third-party libraries.
00:11:07.759 -->
00:11:09.919So that whole process is getting better all the time.
00:11:10.000 -->
00:11:20.000And I think, you know, if you if you'd put me on the spot now with a prediction, I I think ultimately the models are going to produce a lesser density of vulnerabilities.
00:11:20.080 -->
00:11:26.240There's still going to be vulnerabilities, but I don't think we're going to see a uh I don't think we're going to see a higher density of vulnerabilities.
00:11:26.320 -->
00:11:29.840I think so ultimately software is going to keep getting better.
00:11:30.159 -->
00:11:35.440But to your point, there's going to be a massive amount of software, and so we're going to see bigger attack surfaces.
00:11:35.679 -->
00:11:41.679The thing though, and you're correct to point it out, that we really do need to worry about how attackers use this.
00:11:41.840 -->
00:11:51.360I mean, the dirty secret of the entire security industry, as I mentioned before, is that there's been way more vulnerabilities that have gone unexploited than have been exploited.
00:11:51.519 -->
00:11:57.120And so most organizations are kind of targets of opportunity that have not actually been targeted.
00:11:57.360 -->
00:12:04.080What AI does in terms of the industrialization of attackers, it means they can just be relentless.
00:12:04.240 -->
00:12:14.720And so even if they don't get an immediate exploit from even a new vulnerability they've discovered, they can just at low cost just keep hammering away at organizations.
00:12:14.960 -->
00:12:24.399They can scale the amount of organizations they target, they can scale the amount of organizations that have a vulnerability that can be subsequently exploited in various ways.
00:12:24.639 -->
00:12:27.679They can scale the monetization of those attacks.
00:12:27.840 -->
00:12:35.200And so I think that's the real issue here is not necessarily that we'll discover more vulnerabilities, although that is an issue.
00:12:35.519 -->
00:12:42.879The real bigger issue is that this enables attackers to be relentlessly operating at much more significant scale.
00:12:43.120 -->
00:12:45.279So there's just no place to hide anymore.
00:12:45.600 -->
00:12:51.039You know, from a security standpoint, it's probably a bad thing, but it's pretty exciting, right, to see like how much this is growing.
00:12:51.279 -->
00:12:55.440I think we have to see attack innovation to innovate on defense as well.
00:12:55.679 -->
00:13:10.240So the exciting part, I think, about the entirety of AI is um while we do have this attack surface that is now, as we said, getting hammered relentlessly, um, there is the opportunities for teams to innovate in really interesting ways.
00:13:10.480 -->
00:13:19.279So, for example, um, when this first happened, open source repos were getting pounded by um PRs and bug fixes that were coming up all the time.
00:13:19.519 -->
00:13:28.000And I remember in some cases, these open source programs would close their um their bug programs and they would just say, no, we can't handle all this volume.
00:13:28.240 -->
00:13:34.480Well, now we're actually seeing a lot of these maybe go to like auto vulnerability fixes, right?
00:13:34.720 -->
00:13:53.039We're seeing some cases uh where teams are now leveraging AI to help them reduce some of that, like both the cognitive load of so much more noise, but also streamline some of those operational uh inefficiencies that existed just because uh it required so much human touch.
00:13:53.360 -->
00:14:06.879So I guess what are like some of those exciting parts of programs and maybe how is AI kind of like uh lifting up those teams that either didn't have the budget, didn't have the people, um, maybe didn't have the time to prioritize.
00:14:07.039 -->
00:14:10.960How do you feel like it's gonna kind of change the the landscape for them?
00:14:11.600 -->
00:14:31.600Yeah, so AI is a great kind of democratizer of capability, and so we see this quite a bit where organizations, and I I've always thought this that organizations want to be more secure, and they either can't afford to be or they don't know how to be.
00:14:31.759 -->
00:14:39.200And you know, when they don't know how to be, they can't afford to hire the security team that does help them understand what to do, and then they can't afford to pay for all of the controls.
00:14:39.360 -->
00:14:59.120And you know, there's some very high-end organizations and even some medium organizations that do this really well, but for most organizations, they're in this kind of permanent debt of not being able to deploy enough security capability, whether it's basic cyber hygiene all the way through to other types of more advanced things like continuous red teaming.
00:14:59.279 -->
00:15:14.080And now what you see is organizations um applying technology from particularly new startups that have built technology based around AI, using AI agents to provide almost infinitely scalable security capabilities.
00:15:14.240 -->
00:15:17.039So I mean, you've probably seen some of the announcements that we did recently.
00:15:17.120 -->
00:15:24.480So, for example, Kevin Mandia's new company, Armadin, is basically AI agents for full spectrum red teaming.
00:15:24.639 -->
00:15:30.320So most organizations don't have the resources or capability to continuously red team themselves.
00:15:30.480 -->
00:15:32.559And now you can now you can do that.
00:15:32.720 -->
00:15:44.000Um, another company we just invested in, a company called Above Security, is the same thesis, but for um but for insider threats, most companies would love to have a world-class insider threat program.
00:15:44.159 -->
00:15:47.200They generally can't afford to do that in the way they would like.
00:15:47.440 -->
00:15:53.039Now you can augment a small team with a large amount of agents to deliver you a world-class program.
00:15:53.519 -->
00:16:07.600Another one, exact same thesis, a company we invested in called BreachRX, which is AI and workflow support for managing incidents of various forms and coordinating incident response and incident disclosure.
00:16:07.679 -->
00:16:08.879Uh again, same thing.
00:16:08.960 -->
00:16:13.759Everybody would love a world-class uh multi-domain incident response team.
00:16:13.919 -->
00:16:16.960Not everybody can afford that, and not everybody can afford to scale that.
00:16:17.039 -->
00:16:19.679Now you can get agents to augment a team to do that.
00:16:19.759 -->
00:16:22.799And the same pattern, uh, the same pattern repeats everywhere.
00:16:22.960 -->
00:17:00.720But can stepping back a little bit though, it's worth reminding ourselves that while AI is going to be a massive boost to security, whether it's software security, operation security, or all the things I just talked about, we've also got to remember that good old-fashioned, basic high levels of cyber um cyber hygiene type defenses, strong multi-factor authentication, network segmentation, binary authorization, all of those least privilege access controls, many other things implemented relentlessly and implemented well, provide a defense to even AI-assisted attackers.
00:17:00.879 -->
00:17:19.759The main thing though is we've got to implement those at much higher rates of consistency, because back to that point of AI-driven attackers, they're gonna be relentless in finding that one misconfigured system in your otherwise wonderfully configured cyber hygiene, uh, and that they'll use that as a launching pad.
00:17:20.079 -->
00:17:28.400So um, so yeah, and I think AI is a tremendous boost, but you don't need AI, you don't only need AI to defend against AI-driven attackers.
00:17:28.480 -->
00:17:31.200You can you have to do all the basic stuff as well.
00:17:31.519 -->
00:17:33.839Yeah, no, that's again a great point.
00:17:33.920 -->
00:17:38.480And it's always like back to basics, making sure you have your foundational practices in place.
00:17:38.720 -->
00:17:41.200Um, it's just security tech debt, right?
00:17:41.279 -->
00:17:46.079If you don't have that fixed, how do you move on with getting a better world-class program?
00:17:46.319 -->
00:17:48.720Um, you've kind of got to dot all your eyes.
00:17:48.880 -->
00:17:53.119Um, but on that, I'd actually like to go back to agents because you brought up the keyword.
00:17:53.279 -->
00:17:59.119Um, everybody's really talking about them, even though it's been around or the concept's been around for a while.
00:17:59.359 -->
00:18:03.119It feels like right now agentic is just like taking flame.
00:18:03.359 -->
00:18:13.039And um, you mentioned augmenting staff with agents, which is the natural progression of how this technology is going and how I think uh we are envisioning it.
00:18:13.279 -->
00:18:31.279So when we look at agents and bringing those into uh the environment, I think it introduces uh a lot of unsolved problems that uh I think we've tried to apply traditional security mechanisms to um around like IM access management and role-based access controls, data loss prevention solutions.
00:18:31.440 -->
00:18:34.559But all of these were really designed around the humans, right?
00:18:34.799 -->
00:18:37.440So the threat model for an agent is totally different.
00:18:37.599 -->
00:18:45.359So when we think about the actor within our organization as an agent, what do you think are kind of the new ways we need to think about that trust layer?
00:18:45.440 -->
00:18:52.480Um, and maybe how we need to start reasoning about trust uh for AI systems that are working autonomously within our environment.
00:18:52.960 -->
00:18:56.000Yeah, so I I there's a number of ways of looking at this.
00:18:56.079 -->
00:19:07.039And I and it's interesting, we're at the very early days of this, and uh yeah, I, you know, some of your listeners may be old enough, I suspect not many will be, to remember the early days of the internet.
00:19:07.440 -->
00:19:19.599So in the kind of the late 90s, early 2000s, we had a set of technologies for the nascent commercial internet, browsers, web servers, load balancers, all of this type firewalls, intrusion detection systems.
00:19:19.759 -->
00:19:26.160But there wasn't there wasn't really a set of fixed design patterns of how all those things should be plugged together.
00:19:26.319 -->
00:19:37.440And so there was a lot of you know uncertainty around how the security models should work, and then over a period of years, the design patterns got locked in, and then we overlaid security onto those design patterns.
00:19:37.680 -->
00:19:54.000This feels like exactly the same moment where almost every day somebody's inventing a new design pattern for how agents should communicate or how agents should interact with resources or how agents should drive a business workflow or a technology workflow.
00:19:54.160 -->
00:20:05.359So I think we're not really going to see a stability in how we think about security until we start seeing some more coalescence of common agentic design patterns for particular problems.
00:20:05.519 -->
00:20:08.400And an example of that is exactly on identity.
00:20:08.559 -->
00:20:16.160So, you know, should an agent uh operate under a delegated identity and a delegated set of permissions from a human?
00:20:16.319 -->
00:20:21.039Yeah, probably for certain use cases where an agent is acting on your behalf.
00:20:21.119 -->
00:20:27.839Uh, but for other use cases, an agent should probably have its own permissions in the context of a business workflow.
00:20:28.000 -->
00:20:39.759And then maybe for other cases, it should only have an ephemeral identity that's spun up and spun down to deliver a particular subtask in a workflow coordinated by other agents.
00:20:39.920 -->
00:20:48.079And every one of those different use cases will have different properties of how you think about the identity, the permissions, the observability.
00:20:48.240 -->
00:20:51.359And so all of that is going to be different in every use case.
00:20:51.599 -->
00:20:58.960Then overlaid on top of that, we all know agents using models are by definition non-deterministic.
00:20:59.119 -->
00:21:04.880So you can you can ask a model a question ten times and you might not always get the right answer ten times.
00:21:05.119 -->
00:21:17.279But for deploying agents into business processes, particularly financial transactions, health transactions, other critical infrastructure, those need absolute determinism.
00:21:17.359 -->
00:21:39.039And so putting deterministic controls around these non-deterministic agents that aren't in the model itself or in the agent itself, but is in surrounding guardrails that enforce business policies, just like you would enforce business policies around a human's non-deterministic behavior, is what we still have to architect.
00:21:39.119 -->
00:21:49.759And there's, you know, again, there's many, there's many companies and many solutions and much work from the foundation model companies to look at how you augment agent activity with agent guardrails.
00:21:49.920 -->
00:21:52.000And it it's kind of very reminiscent.
00:21:52.079 -->
00:21:58.480Some of your listeners may be aware of this, very reminiscent of what banks do in high frequency trading systems.
00:21:58.559 -->
00:22:14.960You know, they have Um, you know, they have algorithms, you know, some machine learning derived, some not, but then they always have independent checks like circuit breakers to detect if the agent in in modern language is going off the rails than to kind of block its activity.
00:22:15.119 -->
00:22:22.079And so we're gonna need the same type of agentic enterprise control plane that we've built for other purposes.
00:22:22.240 -->
00:22:26.799And again, this is evolving as we speak because the design patterns are evolving.
00:22:27.039 -->
00:22:27.519Yeah.
00:22:27.839 -->
00:22:36.960Um, you know, like one of the big things that like we think about um at Alice is a lot about um how do you get an environment where agents can kind of play and you can understand.
00:22:37.119 -->
00:22:47.440There's this concept of a gym or like an RL gym where we have agents kind of like go at scale, they kind of just like run their business scenarios and we just observe them and we watch kind of just like an Antill farm.
00:22:47.519 -->
00:22:52.480Um and they go, they do their thing, and we recognize the behaviors, we see where it's going out of alignment.
00:22:52.640 -->
00:22:56.960But since it's a simulated environment, we're not seeing those same risks happen in a production environment.
00:22:57.119 -->
00:23:01.839But we can catch all those like really bad behaviors before it actually goes into production.
00:23:02.079 -->
00:23:06.079Unfortunately, this is not something I think every organization has.
00:23:06.240 -->
00:23:14.160Um, not at any fault for the organization, but because a lot of organizations are actually getting agents from another place, right?
00:23:14.319 -->
00:23:23.680So they may be going to a vendor and bringing agents from outside, kind of like uh almost like getting a bringing in a subcontractor or a contractor to work on one of your teams or a project.
00:23:24.079 -->
00:23:31.599So um as an app sec guy, I've always been trained to kind of shift left and move more left, but in this case, uh there is no left, right?
00:23:31.759 -->
00:23:44.480You kind of just have to um depend on processes that you have in GRC and your SOC to hope that like this new external party in your environment is kind of you know playing playing well with everything else.
00:23:44.720 -->
00:23:51.680So I'm wondering these controls that get embedded for agents that uh that are bringing in, are they really like holding up?
00:23:51.839 -->
00:24:00.559Are there maybe something something else that we need to think about in in this like runtime layer for agents that we're kind of consuming, not just creating?
00:24:01.119 -->
00:24:14.000Well, it's interesting you kind of use the shift left because you know, in other spaces, I'd advocate, and including this, that you're exactly right, we need to move from shift left, but instead of shift left, we need to shift down into the platform.
00:24:14.160 -->
00:24:26.799And so this could be you know where agents and in fact any other piece of software should inherit a set of security and other risk mitigating controls from the platforms that they operate within.
00:24:26.960 -->
00:24:35.119And that includes the runtime environment, the interfaces to other systems, the libraries, the frameworks that they pick up controls from.
00:24:35.279 -->
00:24:48.720Um, and so having that kind of shift down helps you with not just the agents you've developed, but the agents that turn up in your environment from third parties, from you know, potentially unexpected sources.
00:24:48.960 -->
00:25:00.960Because essentially, as a as an enterprise, you want to you want to be able to reason about to say, for example, let's say I've got a payments gateway or a stock ordering system or some other critical system.
00:25:01.279 -->
00:25:11.680Um, I don't want to build the controls only into the agents that may be interacting with that, just in the same way that organizations don't depend on correct human behavior.
00:25:11.839 -->
00:25:30.880You have a combination of trained humans, or in this case, trained and well-controlled agents, but you still build tremendous amounts of access control, transactional policies, auditing, observability, and many controls into the resources and systems that are being manipulated by those.
00:25:31.039 -->
00:25:36.559And then reasoning about that collective enterprise control plane is what's critical.
00:25:36.640 -->
00:25:42.880And that has to be in the you know, shifting down into the substrate of the organization in the runtime environment.
00:25:43.119 -->
00:25:50.720Now, the big question is what happens to that third-party agent that's running in your environment when your controls stop it doing something.
00:25:50.799 -->
00:25:55.680How do you signal back to that vendor, hey, I've blocked you because you were doing something crazy?
00:25:55.839 -->
00:26:01.839Um, all of these things are yet to be defined, and you know, and that's that's why this space is so exciting.
00:26:29.660 -->
00:26:30.380Yeah, exactly.
00:26:30.460 -->
00:26:36.059And I really like how you put it in shift down, you know, getting lower into the stack where agents are operating.
00:26:36.220 -->
00:26:48.220This is kind of fundamental, or it's it has to happen, especially when, like you said, you have agents that are likely agents within your organization from different vendors all kind of interacting with each other, right?
00:26:48.299 -->
00:26:55.740You don't necessarily control any of those interactions or have much observability on that shift down layer as to like what they're doing, right?
00:26:55.900 -->
00:27:08.859So if I bring in an agent from one platform and an agent to another, they happen to interact on a project, then we've got a problem where there's no human, it's just agents, and we don't understand the gravitational risk that one agent has on the other.
00:27:09.019 -->
00:27:18.700So being able to report that back, you know, I've said that guardrails are really value adding when you can take the signal from guardrails and turn it back into a flywheel process.
00:27:18.859 -->
00:27:27.819Um, because again, something gets caught by a guardrail, it's not necessarily training the model to get better, it's not training the agent to get better, it's just stopping a bad result from happening.
00:27:27.980 -->
00:27:36.299Um, but from there, turning that into signal where you can go and retrain or provide feedback, I think that's really important to be able to say, hey, we caught something bad happening.
00:27:36.460 -->
00:27:40.380This is how you um we would like to see this going forward, this is how you fix it.
00:27:40.619 -->
00:27:50.140So I think having that feedback loop or that flywheel cycle embedded within like uh when you work with other vendors or outside is gonna be really important in moving forward with shifting down.
00:27:50.380 -->
00:28:03.980No, I absolutely and I think you know it's gonna be just like any, just like all of the other security we've um invented and deployed in the past, it's all about how each layer interacts with each other in that kind of feedback loop.
00:28:04.059 -->
00:28:13.019So if you've got a resource that is continuously rejecting the attempted behaviors of an agent, then you can't just keep rejecting that.
00:28:13.099 -->
00:28:34.619You've got to think what went wrong in the identity and access management process for that agent that I've not appropriately permissioned it, or what went wrong in the enforcement layer that has an agent trying to do something that is against the policy that you want as detected by the resource that for which that kind of rogue access is being attempted, and how all of that ties together.
00:28:34.779 -->
00:28:44.700We've spent decades doing that in the kind of human to application to back-end system to re- other resources, and we're gonna have to do the same thing.
00:28:44.859 -->
00:29:05.500But the I think while it's gonna be conceptually similar, I think the nature and the scale, and to your point about kind of anthills, the extent to which we'll see emergent behavior from agent interactions that weren't quite predicted in our policy management systems, are gonna create some unusual behaviors that we also need to manage and monitor for.
00:29:05.819 -->
00:29:09.819There is this other piece that comes with um kind of like all these agents at scale.
00:29:09.980 -->
00:29:12.380The platforms that they're originating from or being developed on.
00:29:12.539 -->
00:29:13.740We call them foundational models.
00:29:13.819 -->
00:29:16.460Uh, I think Andreas had called them like a God model, right?
00:29:16.539 -->
00:29:18.619Like we have these really big models that do everything.
00:29:18.779 -->
00:29:25.500So now we're moving towards a place where um we have a couple of these big winners in the in the model space for enterprise.
00:29:25.740 -->
00:29:39.019And a lot of vendors and solutions providers are relying on these models to create these solutions, uh, whether it's providing an agent, providing an MCP layer for this particular platform, um, or so on.
00:29:39.339 -->
00:29:50.299So this concentration kind of reminds me of like these hidden dependencies or like these third-party dependencies that everyone kind of shares, but you don't see it at like uh at the high level because of where they are embedded.
00:29:50.539 -->
00:30:00.299So I may be interacting with an agent that may be using open AI's technology and it has the same kind of issues, regardless of what use case I put it in, right?
00:30:00.380 -->
00:30:02.779It's just like a basic benchmarking kind of thing.
00:30:02.940 -->
00:30:06.460But now we're seeing this at scale with multiple providers, multiple agents.
00:30:06.619 -->
00:30:15.180I mean, maybe I'm just like thinking about it wrong or I'm seeing it wrong, but does this problem like, do you think that this kind of exists at scale or how um AI has been rolling out?
00:30:15.660 -->
00:30:24.940Yeah, well, I I think in general, nobody quite knows how it's all gonna land, and that's just the nature of, you know, that's the nature of the beast we're in at the moment.
00:30:25.099 -->
00:30:28.940But I think when you look at the first of all on concentration risk.
00:30:29.099 -->
00:30:36.539So, you know, we've had concentration risks in multiple industries and technologies for years, and people figure out ways of managing it.
00:30:36.700 -->
00:30:58.299You look at the hyperscale cloud providers, never mind the foundation model providers that depend on the hyperscalers to run these things, you know, that's a degree of concentration risk that's managed in various ways by the hyperscalers or by companies figuring out ways to be able to deploy redundantly across multiple cloud and even on-premise infrastructure.
00:30:58.460 -->
00:31:00.299So there's there's ways to deal with that.
00:31:00.539 -->
00:31:29.420Then when you think about model dependency, most organizations I've seen over the past few years have been quite careful not to be wholly dependent on one model, mainly because they're always upgrading models anyway, just because they're trying to find the optimally priced model for the problem that they're trying to solve so that they're not solving every trivial problem with the highest model, and they're making sure they can refer up to a more sophisticated model when they need to.
00:31:29.660 -->
00:31:53.099Now, this does point to a dynamic where you see in multiple spaces in in all of these different vendors that sometimes get accused of just being an LLM wrapper is I think, you know, there are some vendors like that, but there's a lot that don't that they provide a layer to enable companies to have a lot of routing or portability across models.
00:31:53.180 -->
00:31:57.099And they pick up a lot of the work of model validation, model testing.
00:31:57.259 -->
00:32:03.900And so you see these companies like Rogo AI and investment banking or Harvey and Legal Services, all these other companies.
00:32:04.059 -->
00:32:22.539And essentially what they they provide a set of context, domain-specific knowledge, connectors, but also that layer that lets customers dynamically choose which model they're using and then leave it to that service provider to assess and validate whether that model is fit for purpose of what they're doing.
00:32:22.779 -->
00:32:43.339And I think a lot more organizations over time are gonna, you know, they'll have plenty of UK use cases where they directly use the models, but they'll also have plenty of use cases where they just say, look, I value that layer of kind of context connection, knowledge, model validation so much it's worth me paying a premium to not directly use the underlying models.
00:32:43.420 -->
00:32:46.220And I think you'll see various, various flavors of that.
00:32:46.380 -->
00:32:50.220How that shapes out in software security is gonna be gonna be interesting.
00:32:50.299 -->
00:32:55.740So you see plenty of harnesses around the models that help companies do software security.
00:32:55.900 -->
00:33:01.019And then you also see plenty of companies that are helping organizations deal with the output of that.
00:33:01.180 -->
00:33:18.460So I think there'll be multiple different frameworks of how you do this, uh, but not all organizations are gonna want to directly use the models for all things because they they just don't want to deal with all of the testing, the model routing, the updates, all of that kind of stuff.
00:33:18.940 -->
00:33:20.779I I completely uh in agreement.
00:33:20.940 -->
00:33:28.380I did recently see a post from Harvey where they um enabled uh Fable V within their environment as soon as it was released.
00:33:28.539 -->
00:33:33.660However, they released it with a benchmark that showed exactly how well it was performing on their stat.
00:33:34.140 -->
00:33:40.299So I think, like like you said, these providers, they're not always um just a wrapper for AI.
00:33:40.460 -->
00:33:47.099They're a context layer, and they provide um a layer of context that these foundation model companies don't.
00:33:47.339 -->
00:33:55.579So they would rather provide that context layer, allow you to figure out how you want to interact with it, but the consistency of results is kind of what you care about.
00:33:55.660 -->
00:33:58.380And that's why you choose the model that makes most sense to you.
00:33:58.619 -->
00:34:03.099Obviously, they figure out pricing on their end, and yeah, this one costs more because it costs us more.
00:34:03.259 -->
00:34:07.180But otherwise, it's really on you to figure out, okay, well, this is how we want to go for it.
00:34:07.339 -->
00:34:21.099And I think that's kind of uh that context piece, the cost, the value for the context, and then um the provider, all these three things makes it really difficult for leadership to kind of make choices on exactly what they need to be doing.
00:34:21.260 -->
00:34:35.820Um, all of these systems get more complex, especially as they get more interconnected, especially when I think even the conversation that you and I just had specifically around having a model, having the context, and figuring out which solution fits best based on all of this.
00:34:36.059 -->
00:34:42.860I think there's a lot of uh the it's almost like the answer is to get everything versus um get one thing.
00:34:43.179 -->
00:34:45.340And that's not very cost effective.
00:34:45.500 -->
00:34:58.059So I'm wondering if there's an easier way to provide leadership and engineering teams overall a more useful picture of like both the solutions that they can pick and the risks that are coming with all of these different things.
00:34:58.460 -->
00:35:13.019Yeah, well, I I I think I think in each layer of security, you'll see some companies acting as that that context layer over and above the models or what the models deliver to do that.
00:35:13.099 -->
00:35:14.860You know, you use the phrase kind of context.
00:35:14.940 -->
00:35:32.780Do you know one of the key differences you see in many companies is they they build and deliver a context graph for an organization that builds its knowledge up that is then used in the use of the model to make better grounded decisions in the context of that organization.
00:35:33.019 -->
00:35:34.460Um, you see this in plenty of places.
00:35:34.539 -->
00:35:41.980Uh, for example, we've got a company called Armacode, um, that they basically help companies do vulnerability operations.
00:35:42.059 -->
00:35:56.940And so they take in all of this information about vulnerabilities that have been discovered by AI models or from traditional sources and build this big graph of what your organization looks like to help you prioritize how to fix things and how to make sense of that.
00:35:57.179 -->
00:35:59.739We see similar things in in many other spaces.
00:35:59.900 -->
00:36:05.500So that that context graph is and the uh over and above the models is absolutely key.
00:36:05.659 -->
00:36:12.380Because ultimately, again, most organizations they don't just want to use the models for the model's sake, they're using it to solve a business problem.
00:36:12.539 -->
00:36:17.500And often the business problem is best solved by augmenting the model, not just using the raw model.
00:36:17.820 -->
00:36:18.059Yeah.
00:36:18.300 -->
00:36:31.980So based on this whole conversation, I guess if you had to take away one thing or give something to an organization to take away today, maybe an action that they could take or uh something that they should think about when they're either making their next purchase or building their next solution.
00:36:32.059 -->
00:36:35.179What's like the one key takeaway that you'd want them to have?
00:36:35.579 -->
00:36:44.139Yeah, I think it's to remember that while all this is new, it's not necessarily new from a first principles perspective.
00:36:44.300 -->
00:36:55.179So managing AI risk is about software lifecycle risk, it's about data governance, it's about the operational risk of putting the guardrails and hardrails around behaviors.
00:36:55.420 -->
00:37:05.340It's it's understanding who's got what identity, what resources being now, all of that is we've done that for decades in good and bad ways.
00:37:05.659 -->
00:37:12.139It's somewhat different and more pressured in an AI and agentic environment, but it's the same basic principles.
00:37:12.300 -->
00:37:26.619And I think sometimes in massive moments of change like this, we forget to go back to first principles, or we get, you know, get get dragged into the hype that this is all new and the first principles aren't relevant, which is just not true.
00:37:26.780 -->
00:37:32.619So just I would say to everybody, just remember, trust your instincts on going back to first principles.
00:37:32.780 -->
00:37:36.059And when you think like, why should I let an agent do all these things?
00:37:36.219 -->
00:37:37.820The answer is you shouldn't.
00:37:38.139 -->
00:37:46.780You should not just rely on the agent behaving problem, but you should put like controls around it, you should craft its privileges, you should put you know, access control enforcement in resource games.
00:37:47.019 -->
00:37:49.019You've got to do all of that just like we've done for years.
00:37:49.099 -->
00:37:53.659So trust your instincts and uh and keep sticking with first principles and everything will be fine.
00:37:53.980 -->
00:37:54.380Cool.
00:37:54.619 -->
00:38:01.420As a tech enthusiast, as a longtime software developer and everything, uh, what are you most personally excited about?
00:38:01.579 -->
00:38:04.539What would you most be excited to see in the next couple of years with AI?
00:38:04.699 -->
00:38:06.139What would you love to see emerge?
00:38:06.300 -->
00:38:07.820What's like the thing that gets used?
00:38:07.980 -->
00:38:13.659Yeah, it's not so it's kind of excited, but not necessarily in a good way, in a more of a curious way.
00:38:13.820 -->
00:38:18.219So I think we've not done enough yet to think about the second order effects.
00:38:18.460 -->
00:38:22.059So again, I I go back to these kind of previous waves of technology.
00:38:22.300 -->
00:38:28.940So, you know, when the smartphone really took off in the late 2000s, there was lots of discussions about the risks.
00:38:29.099 -->
00:38:33.659But nobody imagined, and they couldn't imagine at that point what the second order risks were.
00:38:33.820 -->
00:38:45.099So all of the things that came from what we built on mobile infrastructure, whether it's social media or kind of gig work or all this other stuff, there was loads of risks that came from that.
00:38:45.260 -->
00:38:54.059I think we've yet to really develop an understanding of what the second order effects are going to be from this first wave of AI deployment.
00:38:54.219 -->
00:38:58.059And this, you know, again, back to that Ant Hill's comment is what does it mean?
00:38:58.139 -->
00:39:08.780What does a world look like of billions of agents, all with different models and reward functions, interacting in different ways under competitive pressures?
00:39:09.019 -->
00:39:11.820Like, who knows what risks are going to emerge from that?
00:39:11.900 -->
00:39:26.619I mean, there are some things that could be quite predictable, like when is the first agentic flash crash going to happen when some website posts up an incorrect price and a billion agents descend on it to try and buy and lock in that contract.
00:39:26.780 -->
00:39:29.340You know, we're very close to that, I would think.
00:39:29.500 -->
00:39:31.500And then what's these other second order effects?
00:39:31.900 -->
00:39:47.179So I think what's going to be fascinating over the next few years that I'm uh excited to see how we figure out how to manage is just the the second order effects of emergent properties coming from a world of trillions of agents wired together in unpredictable ways.
00:39:47.340 -->
00:39:49.420It's uh it's gonna be wild.
00:39:49.739 -->
00:39:51.579Yeah, it is also gonna be very fun.
00:39:51.659 -->
00:39:55.019I know like I'm in the middle of organizing an agent-only conference.
00:39:55.179 -->
00:39:59.659So I'm excited to see what kind of uh what kind of talks they put together and stuff, right?
00:39:59.820 -->
00:40:01.099So cool, Phil.
00:40:01.260 -->
00:40:02.139Thank you so much.
00:40:02.219 -->
00:40:04.300Uh again, it was a real pleasure to have you today.
00:40:04.460 -->
00:40:05.500Where can people find you?
00:40:05.579 -->
00:40:06.860What do you do you got going on?
00:40:07.019 -->
00:40:07.579What's next?
00:40:07.820 -->
00:40:08.380Yeah, yeah.
00:40:08.539 -->
00:40:28.539So um, you know, my blog, you know, content out every two weeks is philvenables.com and uh I'm on X at Philvenables and uh and uh then beginning of next year, there's uh publishing a book on uh how to scale security uh for uh for organizations in a pre and post uh AI world.
00:40:28.699 -->
00:40:29.659So uh look out for that.
00:40:29.900 -->
00:40:34.699That'll be announced on uh all the uh all of the social channels in the in the coming quarters.
00:40:35.019 -->
00:40:35.500Sweet.
00:40:35.659 -->
00:40:36.940Thank you again so much.
00:40:37.019 -->
00:40:40.699I'll be looking out for that book for sure, and maybe we'll have you on again to talk about it.
00:40:40.860 -->
00:40:42.139Yeah, that'd be great.
00:40:42.380 -->
00:40:43.260Thank you, Phil.
00:40:43.340 -->
00:40:44.139Thanks for your time.
00:40:44.219 -->
00:40:47.340And yeah, uh to everybody stay curious and have a great day.
00:40:47.579 -->
00:40:51.659If this episode helped cut through the noise, like or subscribe so you don't miss what's next.
00:40:51.820 -->
00:40:53.179Thanks for spending time with us.
00:40:53.340 -->
00:40:55.820Until next time, stay curious.
