РАСШИФРОВКА 🔗
WEBVTT
00:00:00.080 --> 00:00:09.279
What changed it made you think the AI code security gap is now a real critical inflection point or problem for enterprises, rather than you know kind of marketing hype or a talking point or something like that?
00:00:09.519 --> 00:00:10.800
This feels bigger.
00:00:11.199 --> 00:00:12.000
But why is that?
00:00:12.160 --> 00:00:15.919
Part of the reason is because it's transforming so many different things at once.
00:00:16.000 --> 00:00:22.640
I feel like everyone in AppSteck was very nervous and anxious when the Frontier Labs started releasing native capabilities around AppSteck, and it was just one.
00:00:22.879 --> 00:00:30.079
I think there was this little voice on the side saying, you know, from a risk perspective, how do we manage this?
00:00:30.559 --> 00:00:32.640
Why can't the model itself be the control?
00:00:32.799 --> 00:00:36.399
Where exactly do you let the probabilistic layer kind of make calls versus not?
00:00:36.640 --> 00:00:44.799
There's a separation of church and state, which I think is really important in the security world, which is you have an independent control plane for security.
00:00:57.280 --> 00:00:59.280
Thank you for joining the Resilient Cyber Show.
00:00:59.359 --> 00:01:01.679
My name is Chris Hughes, and today I'm joined by Jonathan Rendi.
00:01:01.840 --> 00:01:02.799
Jonathan, thanks for being here.
00:01:03.039 --> 00:01:03.840
Great to be here, Chris.
00:01:03.920 --> 00:01:04.640
Thanks for the invite.
00:01:04.879 --> 00:01:09.120
Yeah, I'm excited to cat and dig into all things AI, AppSec, and all the things they're in.
00:01:09.200 --> 00:01:13.280
But for folks not familiar with you and your team, can you tell us a bit about your background as well as a team?
00:01:13.519 --> 00:01:15.599
Yeah, I've been in and out of security for a few years.
00:01:15.680 --> 00:01:16.719
We were just talking about that.
00:01:17.040 --> 00:01:21.120
Back in the day, I worked with uh Fortify and Spy Dynamics years ago.
00:01:21.280 --> 00:01:32.159
And the last uh kind of decade, I spent a lot of time in um in developer DevOps, developer-oriented tooling and products, leading product teams.
00:01:32.319 --> 00:01:34.560
But the last year and a half I've been at check marks.
00:01:34.719 --> 00:01:49.840
And so given the tsunami of AI, uh, I've been spending a lot of time working with the teams and um focused on how do we adapt to this like super dynamic, changing life cycle that is so disrupted.
00:01:50.000 --> 00:01:50.400
Uh uh.
00:01:50.480 --> 00:01:55.920
And in a good way, some great things are coming from it, but dramatically different than it was a year ago.
00:01:56.000 --> 00:01:58.000
Aaron Powell Yes, definitely a double-edged sword.
00:01:58.079 --> 00:02:02.319
We're gonna get into all that, uh, the the good and the bad of uh AI's intersection with AppSec.
00:02:02.640 --> 00:02:09.759
Um but you mentioned there you had a background you know on infrastructure side, software testing, incident response, you know, some other areas outside of security.
00:02:09.919 --> 00:02:16.319
You know, what were you seeing in those worlds that drew you back to AppSec or you know, wanted you to come back to this space to work on this problem now?
00:02:16.400 --> 00:02:19.840
Aaron Powell Yeah, it's um a couple of things.
00:02:20.080 --> 00:02:28.639
One, uh, you know, I have a pretty deep background in software quality and performance, and then uh spent time in security.
00:02:28.719 --> 00:02:36.479
And all these things are the attributes, you know, they're all the legs of the stool, so to speak, that are so important in a great user experience.
00:02:36.639 --> 00:02:38.319
And so it's great to build things.
00:02:38.560 --> 00:02:43.120
It's great that they then get actually used and used safely and securely.
00:02:43.199 --> 00:02:54.639
And so I just found myself gravitating to all the all the important elements that make something useful and valuable to everyone, and uh whether you're the provider or whether you're the user.
00:02:54.800 --> 00:03:02.560
And so I've come back to security because uh we I feel like we're in the heart of the um the the heart of the hurricane right now.
00:03:02.960 --> 00:03:08.560
Yeah, I mean it's uh it really is like a uh it could be an intimidating time, but also like it's so exciting.
00:03:08.639 --> 00:03:21.840
Like there's so much going on, opportunities to kind of uh revolutionize the way we do a lot of things in application security and vulnerability management, you know, all the kind of things that we've historically done don't really m fit well with the with the AI native development that we're gonna get into here.
00:03:22.000 --> 00:03:27.360
And you know, so I wanted to ask you, like, you know, you've described this the first half of this year as like an inflection point.
00:03:27.520 --> 00:03:42.000
I'm curious from your perspective as you look at AppSec, like in the first half of 2026, you know, what change that made you think uh the AI code security gap is now a real critical inflection point or problem for enterprises rather than you know kind of marketing hype or a talking point or something like that?
00:03:42.080 --> 00:03:51.919
Aaron Powell So a year and a half ago is when I walked in the doors in our office in Israel at check marks and we had a town hall, and I walked in and they were like, who's this guy coming in?
00:03:52.080 --> 00:04:01.599
And my comment was having just um been on the front end of a lot of the disruption of AI and seeing that and having lived through, you know, the introduction of the internet.
00:04:01.840 --> 00:04:08.159
I'm showing my age a little bit, but you know, and then SaaS and then all these and mobile and all these different disruptions.
00:04:08.319 --> 00:04:14.319
I I was trying to relate to the team that those were big and those were step function changes.
00:04:14.560 --> 00:04:16.639
This feels bigger, but why
Load full transcript
is that?
00:04:16.800 --> 00:04:22.399
And and part of the reason is because it's it's transforming so many different things at once.
00:04:22.560 --> 00:04:29.519
It impacts how people work, it impacts the process by which they work, it impacts their productivity and what they can do.
00:04:29.680 --> 00:04:34.480
Like every single aspect you look at is impacted in some way, shape, or form.
00:04:34.560 --> 00:04:54.800
It's changing the role of individuals from I'm a coder to I'm an architect, um, I'm a product manager to I'm an expert gatherer of information and assimilating that in the right way to define, you know, uh it changes designers, it changes everybody in their roles, and there was a lot of fear around that.
00:04:54.879 --> 00:05:01.680
I it feels to me like a lot of the dust is starting to settle now with security, with roles.
00:05:01.839 --> 00:05:17.439
People are seeing that wow, the opportunity I I think people are starting to get really excited about the opportunity in a lot of different areas versus just the initial shock or the fear that I I saw reflected in a lot of stocks uh with anthropics announcements several months ago.
00:05:17.759 --> 00:05:18.240
Yeah, definitely.
00:05:18.319 --> 00:05:24.399
I had written a piece at the time, I think I titled it like when the frontier lab sneezes uh cybersecurity catches a cold or something like that.
00:05:24.560 --> 00:05:26.480
Because just check up the whole industry.
00:05:26.560 --> 00:05:33.279
Like, you know, I feel like everyone in AppStech was very nervous and anxious when the Frontier Labs uh started releasing native capabilities around AppSteck.
00:05:33.519 --> 00:05:42.319
You actually said something there about the different waves of technology, and I've been uh you know doing this for some time myself, and we had cloud and mobile and SaaS and DevSecOps, all these kind of different trends.
00:05:42.480 --> 00:05:50.879
I feel like, you know, in the earlier eras of these waves, like cloud, mobile, and SaaS for particular, you know, security was kind of a a laggard, a late adopter, a little bit late to the party.
00:05:50.959 --> 00:05:53.279
Um, you know, we would show up and try to constrain everything.
00:05:53.439 --> 00:05:58.399
Uh and we of course we're we're trying to govern and secure things still, but I feel like we've maybe learned our lesson a little bit.
00:05:58.560 --> 00:06:02.560
And you know, I feel like security is leaning in very early in this one, or at least trying to, right?
00:06:02.639 --> 00:06:06.959
Yeah, uh is that would you say it's a fair uh kind of uh impression of how it's unfolding?
00:06:07.120 --> 00:06:08.079
At least that's how it feels to me.
00:06:08.399 --> 00:06:08.959
Yeah, yeah.
00:06:09.120 --> 00:06:11.759
And it this is a really interesting point.
00:06:11.920 --> 00:06:27.759
So if I go back just 18 months, uh at 18 months ago, I was starting to have a lot of conversations with CISOs, with some of our biggest clients and partners over the years, and the sense I was getting was at and you know, don't quote me on this, although who else is here besides just the two of us?
00:06:27.920 --> 00:06:39.199
But like, so I I got the sense that, hey, where does AppSet sick AppSec sit on the priority list of not just the CISO, but the CEO and maybe the board?
00:06:39.360 --> 00:06:44.000
And you know, I'd argue probably somewhere between eight and twelve on the list.
00:06:44.319 --> 00:06:46.720
You know, it's important, but it's not at the top.
00:06:46.959 --> 00:06:54.160
Given, you know, the growth of the frontier models and all of the things that we were just talking about, like the disruptions, I see two waves happening.
00:06:54.319 --> 00:07:10.000
Like wave number one was the CEO going to every head of engineering or CTO and saying, I I don't care what it costs, I don't care where the money comes from, get this product in-house because it's a competitive advantage.
00:07:10.240 --> 00:07:12.959
And everybody took kind of a backseat to that.
00:07:13.040 --> 00:07:15.680
And I heard that in numerous uh clients.
00:07:16.000 --> 00:07:21.839
It's almost like take a back seat, like I, you know, come hell or high water, run, run fast.
00:07:22.000 --> 00:07:34.160
And we did something very similar at Checkmarks where it was, you know, we're standardizing on Claude and Anthropic uh internally, and we looked at cursor and you know, we made some shifts really quick, but everybody's using it, right?
00:07:34.240 --> 00:07:35.360
And it was just run.
00:07:35.600 --> 00:07:45.600
And I think there was this little voice on the side saying, you know, from a from a risk perspective, how do we manage this?
00:07:45.839 --> 00:07:51.439
And and I think it was very much the first voice and then the second voice is the sense I got.
00:07:51.920 --> 00:08:29.360
What I felt has happened is the second part of this in the last quarter, I don't know, three, four months, where it's it's okay now with the you know, some of the announcements that have been made, um, with you know, glass swing and some of these things, now it's like uh the the pendulum, which was kind of like had swung completely over to just run um leaders of development quickly, the pendulum is swinging back very much towards kind of some thoughtful discussions about what should a program look like, what should posture look like, what is the right approach, um, how should I be thinking about this?
00:08:29.519 --> 00:08:40.000
Like and and people starting to really poke at like assumptions or maybe marketing messages that have been very um very uh visible.
00:08:40.240 --> 00:08:42.960
So it's I've seen these two steps, is my point.
00:08:43.039 --> 00:08:49.519
You know, that first run like crazy on development, and now the pendulum is kind of swinging back to when I said the dust is settling.
00:08:49.759 --> 00:08:52.799
A little bit more rational thinking about how these two things come together.
00:08:53.120
--> 00:08:57.120
Yeah, I've seen that in the discourse on, you know, uh social media and things like that in the community.
00:08:57.200 --> 00:09:06.879
It's like I feel like we're we're kind of back to thinking uh first principles, like, you know, rational thinking, uh trying to discern between my marketing hype and like what's real in terms of capabilities.
00:09:07.039 --> 00:09:15.440
There's one thing I see conflated still pretty often, I know you've spoken about this before too, is there's a difference between like functional AI generated code and then secure AI generated code.
00:09:15.600 --> 00:09:20.480
And I've seen research from vendors in academia alike uh that there's a big difference still between the two.
00:09:20.639 --> 00:09:26.720
Um, increasingly, like the code is functional, it works, like it could, you know, uh code velocity and volume is up, right?
00:09:26.799 --> 00:09:30.399
But security of the code is still problematic in many cases.
00:09:30.480 --> 00:09:42.000
Um, you know, where does the research say these two diverge and you know, why does the the model need to produce, you know, uh why does the model produce one reliably but not the other, for example, even through different model iterations and maturity and so on?
00:09:42.240 --> 00:09:53.120
I think it gets to kind of the heart of I mean w believe me, we debate this internally all the time in our research team and our engineering leadership, uh, executive leader team leadership.
00:09:53.440 --> 00:09:59.679
It's we're bringing this major disruption of probabilistic thinking, which is so powerful to everything.
00:09:59.840 --> 00:10:00.879
There's good and bad with it.
00:10:00.960 --> 00:10:06.960
You know, I have these conversations all the time, all the time, uh in work, outside of work, that is it bad, is it good?
00:10:07.120 --> 00:10:08.559
Well, the answer is yes.
00:10:08.720 --> 00:10:16.559
And and like it's it's the old, you know, um Ben Parker, Peter Parker, with great power comes great responsibility kind of phrase.
00:10:16.720 --> 00:10:20.720
And and and I feel that very much is is what's happening.
00:10:20.879 --> 00:10:23.600
And it's there's some really great things.
00:10:23.759 --> 00:10:27.120
What is initially being spoken about are all the great positives.
00:10:27.279 --> 00:10:30.320
More recently we're hearing about some of the not so positives.
00:10:30.399 --> 00:10:45.840
Uh but the you know what we talk about internally is just because a new disruption comes along doesn't mean it shouldn't be an integrated part of what's already been happening, and that's that comes from a kind of a how you think of a proper security program for an organization.
00:10:46.000 --> 00:10:47.360
Like that doesn't go away.
00:10:47.519 --> 00:10:55.120
There have been approaches for years about identification and rules-based approaches and how that how that works in a deterministic way.
00:10:55.279 --> 00:10:57.360
Should all that be thrown out the window?
00:10:58.000 --> 00:10:59.840
No, not necessarily.
00:11:00.080 --> 00:11:04.159
And after we started looking at benchmarks, definitely not.
00:11:04.320 --> 00:11:18.960
So this notion of how the two worlds come together actually allows us to get the good of both worlds and get more than what we got before and address some long-standing problems that have been around forever with noise and everything else.
00:11:19.120 --> 00:11:29.519
So I think if this is done again in with the right framework, with the right approach, we can address some of the biggest challenges that AppSec has had for the longest time.
00:11:29.679 --> 00:11:44.960
We can get earlier in the process, we can eliminate a lot of the waste and the inefficiencies and ultimately deliver, you know, less, you know, protect the organization from increased risk, which is what really our roles are all about.
00:11:45.279 --> 00:11:50.799
Yeah, I like what you said there about not throwing out, you know, kind of proved uh proven methodologies and things that we've learned over time.
00:11:50.960 --> 00:11:58.240
It feels like for a set for like you said for a while there we did that was just like go fast and like forget all the things that we've learned these painful lessons already.
00:11:58.399 --> 00:12:07.360
But at the same time, there are risks associated with kind of recreating the will or uh trying to apply old methodologies, right, to a new technology and a new new way of working.
00:12:07.519 --> 00:12:14.960
And you've said the biggest mistake is treating AI generated code exactly like human written code, you know, same review cadence, same trust assumption, same tooling.
00:12:15.120 --> 00:12:20.639
If it's a different process at a different velocity, you know, what does that security model look like for code that's AI generated?
00:12:20.720 --> 00:12:22.080
What does that look like in this new paradigm?
00:12:22.399 --> 00:12:28.960
You know, one of the an individual that I have a a ton of respect for in the industry is um Caleb Saima.
00:12:29.279 --> 00:12:46.480
And Caleb and I uh had the chance to work together many years ago, and I remember one of the things he said early on was this notion of um a layered approach and how security and when I got to work with a lot of the founders of uh of Fortify years ago, like they said the same thing.
00:12:46.639 --> 00:12:53.200
This notion of many layers is important as you try to go upstream to get preventative, and that's always stuck with me.
00:12:53.360 --> 00:13:01.679
And so here comes along this new disruption of AI and a whole accelerated life cycle, everything is condensed.
00:13:01.840 --> 00:13:15.919
The design, the ideation, the validation, the security, everything is condensed because to your point, the velocity is so great, the volume is great, the risk of that increased volume of code isn't necessarily getting any better.
00:13:16.159 --> 00:13:18.960
So, like when you look, at least today is the
case.
00:13:19.120 --> 00:13:23.679
So, like when you look at all of that, it's like hmm, everything's condensed.
00:13:24.000 --> 00:13:29.919
Most organizations aren't hiring more people, so that's not a winning proposition.
00:13:30.000 --> 00:13:39.919
So, how are you gonna basically automate and automate sooner and automate some of the things later that you may not have known in your backlog you had to deal with before?
00:13:40.159 --> 00:13:42.879
And so a couple things just came to the forefront.
00:13:43.039 --> 00:13:45.919
One, in this world of AI, context is everything.
00:13:46.080 --> 00:13:50.799
So, how can you ground what you're doing in in the most efficient, effective way?
00:13:50.960 --> 00:13:52.320
How can you make it consistent?
00:13:52.480 --> 00:13:54.080
How can you make it cost effective?
00:13:54.240 --> 00:14:02.960
And how can you create that kind of automation or orchestration layer that will marry to all the phases of the life cycle so you're catching things earlier, right?
00:14:03.039 --> 00:14:04.240
And it's a layered approach.
00:14:04.399 --> 00:14:07.600
So that's been kind of the way I've been thinking about it.
00:14:07.759 --> 00:14:15.200
Um everybody talked about shift left for so long, and now there's left left and being a part of the stack and everything else.
00:14:15.360 --> 00:14:16.879
But the concept is the same.
00:14:17.039 --> 00:14:27.759
It's uh how do you effectively create zero friction and not have to have your developers or architects now of software have to be security experts?
00:14:27.919 --> 00:14:31.679
And and I think that's the the huge opportunity and the power here.
00:14:31.919 --> 00:14:37.600
Yeah, something you said that really resonated with me is like HIF-left was this trend that, you know, it and on the surface it felt right.
00:14:37.679 --> 00:14:49.600
Like we were trying to do the right thing, but it often gets equated to dumping like noisy, volumeless uh, you know, scan scan findings on developers with little, little to no context and telling them like they had to fix XYZ before they go to production.
00:14:49.759 --> 00:14:54.159
It kind of bolstered the silos that we look to break down between us and developers with DevSecOps.
00:14:54.399 --> 00:14:58.480
Um, and then also you kind of framed, you know, we've been talking about the risk of AI, right?
00:14:58.639 --> 00:15:00.000
AI generated code, things like that.
00:15:00.080 --> 00:15:03.679
But there's also it's still it's dual nature in a sense it could be a force multiplier for us.
00:15:03.759 --> 00:15:06.320
We can leverage it to benefit us as well in AppSec.
00:15:06.559 --> 00:15:14.559
And I think your team has been doing some innovative stuff here around AppSec and uh leveraging agents, you know, for developers, for triage, for remediation, uh things like that.
00:15:14.639 --> 00:15:19.919
You know, what does that kind of look like with these agentec coding tools and you know how a developer kind of uses this capability?
00:15:20.240 --> 00:15:22.000
Yeah, I think it's really exciting.
00:15:22.080 --> 00:15:41.519
And you know, we we've sent even a lot of this conversation talking about the the improvements in productivity and efficiency and everything for designers and developers and and definers, product leaders, but then it like bringing it back to home to the AppSec professionals, I think it does a couple of really powerful, powerful things.
00:15:41.679 --> 00:15:51.120
Um one is it you know it it allows a super valuable part of the organization to be able to do higher value activities.
00:15:51.279 --> 00:15:58.879
It's almost back to the days of when I first started years and years ago in quality, you know, everything was manually being tested.
00:15:59.039 --> 00:16:02.480
And it was almost like the stepchild of development.
00:16:02.720 --> 00:16:19.679
And um the idea was if you can automate some of these capabilities, which is a highly valued activity, you free up the opportunity to really start thinking about the aspects of these important attributes of value delivered and quality and security and all these things.
00:16:19.759 --> 00:16:20.879
And it's no different here.
00:16:21.120 --> 00:16:23.600
AppSec is that role in the organization.
00:16:23.759 --> 00:16:24.559
They're experts.
00:16:24.720 --> 00:16:45.600
So I think it's one of those things where agents, models, agentic approaches kind of raise the tide for all boats, meaning there's a level of expertise that there are always going to be a classification of vulnerabilities that will need, I believe, as things get more and more sophisticated, a human touch.
00:16:45.840 --> 00:16:51.039
This isn't something that, this isn't completely the the dark factory for AppSec.
00:16:51.360 --> 00:17:08.319
This is something that needs to be vigilant, but what it does is it frees up the the mundane, the rote, the the you know, the the painful activities and automates those so that those higher order issues can be addressed and addressed at scale if done the right way by AppSec.
00:17:08.559 --> 00:17:16.960
So I think it actually, more than anything, it raises the profile of application security in the organization at a time when it's desperately needed.
00:17:17.119 --> 00:17:18.960
So I I think it's really exciting.
00:17:19.279 --> 00:17:20.079
Yeah, I agree with you.
00:17:20.160 --> 00:17:20.559
I'm excited.
00:17:20.640 --> 00:17:28.240
I think we have a chance to kind of make a dent in areas uh and problems that we've just struggled to truly, you know, make much progress in historically.
00:17:28.319 --> 00:17:35.519
And um, you talked about this a little bit when you talked about combining, you know, deterministic and probabilistic capabilities like for scanning, for example.
00:17:35.680 --> 00:17:39.039
And then of course, like the frontier labs are releasing their own capabilities.
00:17:39.119 --> 00:17:40.799
We talked uh talked about that a little bit.
00:17:40.960 --> 00:17:49.440
Um you had recently posted that, you know, Fable V, even though it's shipping stronger guardrails, for example, enterprises still need a deterministic independent control plane.
00:17:49.599 --> 00:17:52.480
You know, and this is something I've been harping on too, but I'm curious your perspective.
00:17:52.559 --> 00:17:54.880
Like, why can't the model itself be the control?
00:17:55.039 --> 00:17:59.039
And where exactly do you let the probabilistic layer kind of make calls versus not?
00:17:59.440 --> 00:17:59.680
Yeah.
00:17:59.839 --> 00:18:25.839
Um it it it gets back to the uh in in in my humble opinion, it gets back to the that that kind of foundation of um there's a separation of church and state, which I think is really important in the security world, which is you have a an independent control plane for security, and the model that you're using for productivity and for generation, uh there's there's a conflict of interest if that's also doing the validation for security.
00:18:25.920 --> 00:18:31.839
So that's kind of a basic, you know, important element to approach this in the right way, um, number one.
00:18:32.079 --> 00:18:43.200
Number two, um, as we were talking about, AI provides so many benefits in such a positive way for a classification of issues that couldn't be easily found before.
00:18:43.359 --> 00:18:59.440
I've been talking about in the teams that I work with, we've been uh our research team um uh called Checkmark Zero, some fantastic individuals there have been big proponents of this hybrid approach, bringing together deterministic and and probabilistic together.
00:18:59.759 --> 00:19:20.799
And in our benchmarking, I think this was the the the the greatest tell of all is w we looked at, you know, dozens and dozens of open source projects that are freely available to everyone and ran you know this combination approach uh with our own harness that we you know have have created against this just to see what the results would be.
00:19:21.039 --> 00:19:26.480
And it the the the output to me was like super interesting.
00:19:26.640 --> 00:19:31.039
It's it's this notion of if everybody gets in their head like a Venn diagram.
00:19:31.279 --> 00:19:35.519
And a Venn diagram can be heavily overlapping or very small overlap.
00:19:35.920 --> 00:19:43.279
And what we found over and over and over again was the overlap is like less than 10% consistently.
00:19:43.599 --> 00:19:56.160
So you find a whole classification of issues with a rules-based historical approach, um, and you find a whole class of issues uh with an AI approach, and like there's very little overlap.
00:19:56.319 --> 00:20:02.880
So if you talk about complete, like that's uh such a huge benefit to organizations, number one.
00:20:03.119 --> 00:20:18.240
Number two, then the question is like in our industry, as as you know, Chris, there isn't a lot of like standards or standard benchmarks, and everybody has their own, and for good
and and maybe more marketing purposes.
00:20:18.319 --> 00:20:22.799
Um and and we all accept that and we all understand that and we all get it.
00:20:23.119 --> 00:20:26.000
But I do think it's important to talk about fidelity.
00:20:26.160 --> 00:20:41.359
It's a long-standing issue in the in the industry where everybody talks about noise and you know, the number one going back to the comment, the question that you asked about development and and like that shift or shift shift left, the whole reason that's in place is because of friction.
00:20:41.519 --> 00:20:43.839
The last thing people want is noise.
00:20:44.079 --> 00:20:53.920
And so I think like fidelity is so important in that balancing act between true positives and false positives and that that balancing.
00:20:54.000 --> 00:21:12.240
And so we've tried very hard to focus on something called an F1 score in statistics, which is that precision and recall and that balance, and really be clear about what we're doing because it's it's for the v benefit of our audiences, you know, our our our clients, so that they have a measuring stick.
00:21:12.319 --> 00:21:20.160
So I think it's completeness, it's fidelity, it's consistency of results, it's cost effectiveness.
00:21:20.319 --> 00:21:29.200
You know, in our industry, we should hopefully be focused on um value and reducing risk, not selling tokens.
00:21:29.440 --> 00:21:41.279
And so along with that, along with how we think about the harness, I think we're incented in many ways to make sure that we provide all those value propositions uh in the most effective way.
00:21:41.519 --> 00:21:44.880
I wouldn't say that the the model providers don't care about that.
00:21:44.960 --> 00:21:48.160
Uh ultimately they will, but they're not incented for it.
00:21:48.319 --> 00:21:50.640
That is their that's their business model.
00:21:50.799 --> 00:21:59.759
And and so anyway, it's there's so many benefits of bringing the two together that just a model to your original question on its own isn't.
00:22:00.240 --> 00:22:00.480
enough.
00:22:00.640 --> 00:22:03.359
It's a huge step forward, but it just it isn't enough.
00:22:03.680 --> 00:22:06.799
Yeah, that's a that's a nuanced and again like a practical take.
00:22:06.880 --> 00:22:14.240
Uh I feel like when initially, like I feel like earlier this year, like LLM was kind of started to become really capable when it came to coding and code review and things like that.
00:22:14.400 --> 00:22:16.400
And uh you saw like two camps in the community.
00:22:16.480 --> 00:22:22.240
One was like, you know, uh it's non-deterministic, it's probabilistic, it can't it can't be used, you know, for code scanning, et cetera.
00:22:22.400 --> 00:22:24.240
The other one was like, oh, just use it for everything.
00:22:24.319 --> 00:22:25.759
We don't need the legacy approaches anymore.
00:22:25.839 --> 00:22:27.680
We don't need heuristics rule-based approaches.
00:22:27.839 --> 00:22:32.160
And it's like, no, like believe it or not, like there's a best of breed or best of both world scenario, right?
00:22:32.240 --> 00:22:35.119
Where you use both, you get the advantages of both.
00:22:35.279 --> 00:22:42.880
And I like where you mentioned um focusing on not just like uh you know findings and and and recall and precision, but also efficiency, token consumption.
00:22:43.039 --> 00:22:49.359
Uh these are things I think the industry is starting to kind of again feel our way through and start to get much more of a mature approach around.
00:22:49.519 --> 00:22:54.559
And you know there's also the fact that you know this new technology, it's like a new technology, a new attack surface.
00:22:54.640 --> 00:23:04.720
It brings with it all kinds of new considerations too, like prompt injection and the coding agents, you know, you have poisons or poisoned or hallucinated dependencies, models and supply chain concerns, right?
00:23:04.880 --> 00:23:07.839
Agents writing access to repos they shouldn't or deleting, et cetera.
00:23:08.000 --> 00:23:21.759
You know, as you think about these threat factors or kind of discuss with you know your your uh community of customers and so on, uh which ones do you think uh are the most critical or concerning and which ones do you think uh maybe we're overweighing in some cases or get kind of maybe blown out of proportion too.
00:23:22.000 --> 00:23:23.279
It's it's a great point to raise.
00:23:23.440 --> 00:23:41.839
It's it's interesting there's there's uh what what what I'm sensing and what I'm seeing and and kind of when I when I think about it kind of in in reflection I I I hear there are a couple different things happening and a lot of it is dictated by the maturity of the organization, what the organization is trying to do.
00:23:42.000 --> 00:23:58.000
So we work at the benefit of working with a few clients who are very cutting edge, the leaders in their industry and they're pushing the envelope going back to that first comment about like the the pounding on the head of developments or engineering's head, run, run fast.
00:23:58.799 --> 00:24:01.440
So because of that they are running really, really fast.
00:24:01.599 --> 00:24:03.680
It's maybe not as regulated of an industry.
00:24:03.839 --> 00:24:27.200
And so they're running really fast and they're running into many threat vectors before some of the other more regulated industries that maybe don't run quite as fast, like insurance or healthcare or for all the right reasons financial services in some ways, although they kind of skirt the line between super aggressive and and conservative better than anyone I've ever seen.
00:24:27.839 --> 00:24:45.359
But so so like some of like tech will run really fast in certain areas and um or or semiconductors or these kind of folks what I see is they're running into some really important issues that are highlighted before those other organizations.
00:24:45.519 --> 00:24:49.200
And so an example of this is you say new threat
vectors.
00:24:49.519 --> 00:25:04.799
So the building blocks of what they're building into their systems when it involves agents and when it involves models and when it involves skills files like these are all new threat vectors and you could argue something like skills files are ubiquitous.
00:25:04.880 --> 00:25:12.319
It's everybody's going to have this issue but not everybody is building models into their app right away or their customer facing services.
00:25:12.480 --> 00:25:21.119
And so like just knowing like the the way we've thought about that is the first step to improving something is just knowing the visibility to it.
00:25:21.200 --> 00:25:35.279
So that that notion of a a global kind of AI inventory of what's going on because you know in the past people in the world of of SaaS you know these other trends it was shadow IT shadow IT starting everywhere.
00:25:35.440 --> 00:26:03.759
Now I see shadow AI happening everywhere with like we we we released a a capability to to capture that AI inventory and then ultimately scan it and like the early access customers were finding out of like 5000 scans half of them they were identifying AI building blocks, models, MCP servers that the security team had no idea even existed.
00:26:04.000 --> 00:26:11.839
So I mean that's kind of very telling um that in these more aggressive companies, these building blocks and these threats are there.
00:26:12.079 --> 00:26:25.759
No one's shutting the door on them or shutting the window on them much more so than an organization which has very stated policies on AI, what they will allow, what they won't allow, what developers can do, they're auditing them all the time.
00:26:26.000 --> 00:26:30.960
So it going back to simple answer is it's really a maturity thing.
00:26:31.440 --> 00:26:40.880
We're seeing new threat vectors in the aggressive companies not as many new threat vectors more at the code level of the generation and the more kind of conservative companies.
00:26:41.200 --> 00:26:52.400
Yeah it's a good distinction between the two and if you are in those regulated environments where you see you know other industries tend to be early adopters or move out quicker with new new technology trends I would say learn from the mistakes of others.
00:26:52.480 --> 00:27:06.960
Like they're gonna they're gonna trip they're gonna step on something that hurts and they're you know you can learn from from their mistakes and and like they're sharing those lessons with the community a lot of times you just got nowhere to look of course and of course you know with your team you can share those lessons right across the customer base as well.
00:27:07.119 --> 00:27:10.000
So just you know lean into the community I would say on those things.
00:27:10.079 --> 00:27:45.519
And then you know last question for you say you're a CISO or an app tech lead you know with mostly say say mostly a human written code based traditional kind of development and you have a mandate to adopt coding agents or say you're an environment where you're finding out oh wait, we're already using these AI coding agents pretty extensively that we didn't know about you know what are some things that they should be thinking about changing first in terms of their app tech program to try to get it right or or what we think right looks like so far, basically as you know kind of we're still building this in flight as a community, but like you know, what are some things they need to be thinking about as they go to adopt AI coding agents and kind of change how they do app tech as a program basically.
00:27:45.839 --> 00:27:52.400
Yeah I you know it's it's interesting we we were posed and really faced with this internally ourselves.
00:27:52.480 --> 00:28:01.039
It's be really clear on what you want to measure and and establish up front what are the metrics that matter to you.
00:28:01.279 --> 00:28:16.720
And so there's there's been a lot of work done in the area of development of this which is measuring the productivity and the gains of AI which is complexity and volume of pull requests that are happening and those accepted PRs, right?
00:28:16.880 --> 00:28:21.519
And so that shows it shows a little bit about um you know the productivity of that.
00:28:21.759 --> 00:28:34.960
But then from a security posture perspective there I think there's a corollary to that which is okay um we have both incoming code we have existing vulnerabilities.
00:28:35.200 --> 00:28:41.519
There has to be a level of understanding where both are areas to focus on.
00:28:41.680 --> 00:28:47.039
Whereas in the past I think there was a lot of work on incoming code not so much on backlogs.
00:28:47.200 --> 00:29:01.039
And I think with AI that changes the equation where I I think I referenced this at the very beginning we're seeing that vulnerabilities low priority vulnerabilities can be chained together to show uh to have a much bigger impact or risk.
00:29:01.200 --> 00:29:12.400
And so I I think like looking equally at what is out there in the backlog versus that's really important to take a look at and make sure that you have an approach that addresses both.
00:29:12.559 --> 00:29:13.279
That's one.
00:29:13.599 --> 00:29:26.799
Two I would say in an AI world from an AppSec lens make sure that you're looking at in flow uh an in-workflow way to address both development and app sec together.
00:29:26.880 --> 00:29:28.160
It should be a part of what they do.
00:29:28.240 --> 00:29:35.039
They shouldn't have to change what they're doing which maybe that seems like simple hygiene but easier said than done.
00:29:35.200 --> 00:30:01.839
And then I think there needs to be kind of like a maybe a better process when I talk to a lot of CISOs by which the information the constant risk since risk is changing so quickly now how you're communicating and visualizing that to the rest of the organization, the executive team, the board I had a conversation with one executive that said that, you know, we used to get called in for a security review once a year.
00:30:02.160 --> 00:30:04.079
Now it's monthly it's monthly.
00:30:04.319 --> 00:30:10.160
Like it's uh that's a fundamental change to how risk is being assessed all the time.
00:30:10.319 --> 00:30:19.839
So I think thinking about how you can scale that kind of communication to the organization is a really important part to to help the business.
00:30:20.000 --> 00:30:41.839
So yeah I think uh you know the answer you gave was a good one because it kind of it strut out the the both the technical kind of uh in a technical tactical kind of world of developer workflows and native tooling but also the strategic executive kind of uh how we communicate this risk how we uh bring awareness to these tools and what they what they introduce and how we need to govern and secure them um so super practical advice there.
00:30:42.000 --> 00:30:43.839
So Jonathan thanks so much for jumping on.
00:30:44.000 --> 00:30:50.240
Again that's Jonathan from Checkmarks checkmarks.com check them out and uh really appreciate the conversation and joining me today.
00:30:50.559 --> 00:30:54.799
Chris thank you it was a pleasure good good to see you and thanks everyone appreciate your time.
00:30:55.039 --> 00:30:55.920
Absolutely take care