رونوشت کامل AI Red Teaming At Scale with Bret Kinsella, VP of AI Safety and Security at TELUS Digital

back because you guys really use this red teaming concept ⁓ and it's really where you focused in on, think, in this product. Walk us through the problem that you see out there. As I see it, I was in a corporation, generative AI was coming out and the immediate word was stop all generative AI if you've got it inside of the business. You're not allowed to use these tools. We were blocking it from the network. reason was, I think kind of what you guys are getting at is it's not only external threats of what is this, what is generative AI gonna do toward my network, but it's also users inside of your network. And I think when I was, when that CEO who said like stop all gen AI, this was in 2024 I think. He was right because we didn't have process, didn't have that thought process we were going for. ⁓ my God, let me use this. The excitement of it. Look at what I can do. And he saw it because he was more worried about IP leaking out ⁓ and stuff like that. And so it's a very big threat. So tell us what was that problem you guys saw and how are you getting at it? Bret Kinsella: Yes. Well, there were few different things that we saw at the time. one was when we won that award for Purple Teaming, what we realized was that it's very labor intensive and you need experts. And even when you have experts, they don't have typically enough domain expertise because what you see is the attack surface area becomes essentially infinite when you have some sort of either voice or text input box. They can ask, they can say anything. And then you have models that are trained on the internet and things like that, even when you can curtail them. you don't necessarily know what they're gonna come back with. So when you have people who come in with testing who might not know all of the crazy things people might ask for that might hurt your reputation or violate the law around privacy rights ⁓ or lead you to discrimination or political manipulation, they don't know how to do all those things. And so that's been, that was one of the things that we were looking at. And I would say that two other things really stood out to me. One is, When I looked at the adoption rate of generative AI, in particular solutions which are probabilistic, not deterministic, probabilistic, so you never know exactly what they're gonna say, sort of guide it, there were more of those being launched than there are skilled red teamers in the market. So that was the first thing I looked at. I said, hey, you know what, we're gonna need to have something that. Russ Fordyce 🏆: Mm. Bret Kinsella: people who are non-technical can run this, like a project or a product manager should be able to do this to be able to get the information right away because they can't wait sometimes for a week until a red team is available or the experts next month. And what that meant is they were only doing it at launch. The second thing is what I saw in terms of the technology being applied, it was very focused on intervention as

opposed to prevention. So most of the people here will be familiar with the term guardrails. ⁓ And when you do blue teaming, very often you're doing maybe some prompt manipulation, but you're putting in some guardrails, which are filters. And so maybe they're just rules-based filters or they're AI-based filters. But still, it's an intervention. What you're trying to do is you're anticipating what the problem's going to be, and you're trying to intervene at the time the problem is occurring. This could be a bad actor. It could be a regular user who's just inadvertently stumbling into an area that they probably shouldn't be in. Either way, you want to prevent that, right? So but I was saying, well, geez, like how do I know what my risk is? I've been working with these models for a long time. They all operate a little bit differently. Once I attach them to different networks and applications in my system, that changes the behavior. When I change the system prompt, it changes the behavior of the model. And so all those things can impact whether it operates within what I would consider the code of conduct of what I want it to do, or it goes outside of that. And so those are the things that we're really trying to focus on. We're trying to focus on this idea. How do we know what our problems are? Because this was a gap in the market, and we have a large research team, a lot of data scientists, AI engineers in our business. How can we know what the problems are so that we can proactively mitigate those risks? And is there a way, and I didn't know this at beginning, we did not know this, is there a way to make it easy enough so that you don't have to have that technical expert available every time so that when you do make changes, whether it's every Friday you want to know, or you've got some sort of update your system prompt, is there a way for you to basically get a snapshot of your risk if you inadvertently walked into a regression problem? Russ Fordyce 🏆: So if you guys are kind of combining those teams, if you go at it the way you're at it, are you stopping the threats before they get to the GenAI, or are you getting them? GEN-AI as they're getting back to the user. ⁓ Bret Kinsella: Yeah, so what we're hoping is that we're helping people build the system that stops it proactively. And if not, then stops it reactively. So we want to be proactive versus reactive. And so what we would generally recommend, and we do this ourselves, there's lots of types of guardrails out there that you should implement. And I would say that most people aren't actually implementing guardrails. What they're doing is they're clicking a button in their cloud provider and hoping for the best. Or they're selecting a model that they think is a little more locked down, and they're hoping for the best. But in general, you need to go a little bit beyond that because your implementation is different from the generic capabilities that they may offer. And even when you do that, you're going to have gaps you don't really know. And so what we need to do is figure out a way to do really ⁓ significant breadth. And then the other thing we needed to focus on was depth. And because we had done our own research up on archive, we've got researchers who get papers accepted to the major engineering conferences. And we'd found that the probabilistic nature is random. So you couldn't just do it five times and be like, OK, I'm good. It might be the 17th time that you hit it, or it might be the sixth time. And there's a couple ways to attack that. One is just to run repeats. Another is to run regularly. So you might have smaller batches, but you do it on a regular basis. And that's where you catch those up. The only thing that we found was that after looking at Russ Fordyce 🏆: Right. Bret Kinsella: what the state of the art was is to take, source a lot of attacks from the industry, different types of things that might lead to a compromise that you don't like, ⁓ and just put those in a library and run those, single shot attacks. What we realized is actually what happens is the models, the guardrails overfit to those, and so what that means is they block that, but a slight adjustment to that attack actually just shows that you had risk all along. And so it's really just trying to figure out, it's trying to look at it like everything with a really specific signature. ⁓ And so that was the other thing that we came along. We said, OK, we need to be able to do this at scale. It needs to be easy. We need to be able to do repetition very easily. And we need to be able to generate novel attacks on a regular basis, because those are the ones that then expose what the problems are. Russ Fordyce 🏆: Well, and I think what I was reading from your nomination is that you guys are actually running thousands of attacks in like minutes. And I would imagine that's really kind of more along the lines of what you're seeing out in nature is kind of that multi-pronged attack looking to unbalance sysadmins or whomever, the network admins. ⁓ Is that kind of why you built the system that way? Is that kind of how it works? Bret Kinsella: There is an issue around load on systems that could create negative outcomes there. Really it was an efficiency play. I think that when you look at it, what we saw is there's this, if you need to do thousands of attacks, and we've done red teaming and blue teaming for clients that were deploying new gen AI based bots. customer facing, internally facing, because employees are also at risk here. ⁓ what we found is that we were doing, depending on the engagement, dozens, maybe hundreds, maybe 1,000. And if we were doing it, but those are novel attacks, right? Maybe we could source from the internet. We looked at like 600,000 and someone, the group that put those together said 32,000, these are good. And we actually regressed it. We're like, actually only 2,400 of those are actually decent. Russ Fordyce 🏆: Mm-hmm. Bret Kinsella: You

know, so we were able to figure that out. We're just like, but this isn't actually good enough. So that's what when you start to say we have to do scale and we have to do novelty, we want to be able to do that really fast. And if you think about it, you know, one of our customers was running seven people, seven days. So 49 man days of testing before they would roll out for consumer facing. But and we were able to get them 12 times the coverage. and identify three times the amount of vulnerabilities as they were ⁓ in 1 30th of the time basically. mean, because we do it, so that run I think was between four and five hours and they were, and they just had a lot of mandates in there. that was one of the key things that we were looking at because we just want these things, there's so much benefit here. There's so much benefit for AI applications. Russ Fordyce 🏆: And I love it. Bret Kinsella: but they can get shut down in a second and then no one gets the benefit of it. A little bit of prevention, the ounce of prevention is worth a pound of cure. That's why I think an ounce of preventative testing and monitoring versus what you're trying to do in guard rails and lock everything down is the thing you need. You need both, but nobody was paying attention to the front end of that problem. Russ Fordyce 🏆: All Now I imagine you guys are actually finding new vulnerabilities that humans probably are missing. Especially with that amount of testing, you've got to be finding new stuff. What are some of the new vulnerabilities that you're finding in these bots? Bret Kinsella: all the time. Well, what we've done is we looked at what was going on in the market and we looked at the systems that were being deployed and we basically looked at over 10,000 different pairs of attack methods and objectives. And so we narrowed that down to 139, but it fluctuates. Sometimes we consolidate, sometimes we expand. But think about 140 to 150 different areas across 15 different categories. And those things are everything from privacy, to fraud, to political manipulation, to age discrimination, like all the different things. As we talked about, very broad scope, right? And so we did that and we basically said we need a taxonomy that we can do, that we can utilize to create a system that will evaluate for these risks and will give scores back, clear, vulnerable, some guidance. And then we started like. Russ Fordyce 🏆: Yeah. Bret Kinsella: Mapping those to OWASP and NIST and MITRE because people already use those and something they understand. But that was really important. So we've got those 15 categories. And then there's a 16th category, which is really what we would call application or target specific. And that's if you're a telecom company or you're a consumer electronics manufacturer, retailer, you have some specific risks associated not just with your industry, but also with the application and the scope of that application that you're rolling out. And so that's the way that we think about it. And that's really what we were focused on is saying, okay, here's a list of things that are very important. We've created a system that will automatically probe your system and give you a score in terms of the severity of that risk, which is essentially the likelihood that that would happen plus the combination of how accommodating it is to whatever, because there's a scale of compliance. And this is what I mean. I don't mean compliance in terms of compliance. I mean how compliant the AI will be to the request of the user. So we put those together so it would be a lot easier for people to understand it. And not only can someone just click a few buttons and run this type of thing because it's basically pre-baked, but they can also read it. They can read the output. People who are not technical can say, ⁓ the judge comes back and tells you this is what the problem is. And this is why that's a problem and the way it's done it. And then you've got GRC, the compliance and risk and safety people and the application people and the security people. They can all actually collaborate around that and understand what the risk is. Are they just managing risk? Do they need to take immediate action? That type of thing. And I'll tell you, we had a government ⁓ customer recently that attended some workshop that we were running and they're like, hey, could you just test our bot? We're launching this next month. And we're just like, yeah, we can do that. So we set it up. they'd found zero. They'd done a lot of testing. They found zero. They were down to zero vulnerabilities. And we were able to, like, in a couple hours, we were able to find, just by clicking a couple buttons, between 150 and 200. And so then they're like, oh, wait a second. Let's work on this a little bit. And I think that that's really what's important. And actually, I'll make one more point. Russ Fordyce 🏆: Yeah. Bret Kinsella: The key thing about this is when people think about AI in general, one of the things you'll hear a lot is people saying, with generative AI, you don't have to create everything. Maybe you do, and it's going to help you. But a lot of times, you turn more into an editor of whatever is happening, and you're consolidating things, and you're using those big human brains to make it better. This is the same thing that we were looking at. As opposed to taking this ninja in the cyber security team and knows everything about AI and all your domains and is going to break it and find the problem that you need to prevent. We needed to basically create a situation where those people can become editors. So they can look at it and say, okay, now I know how I have to fix that. Or here's where I wanna probe further. Or a product manager can be like, ⁓ I see what this is. Let's figure out what we can do to fix that either internally or with a cyber team. Russ Fordyce 🏆: Yeah, and you guys have done a great job at kind of democratizing this tool and making it available kind

of at that level you're talking about. So a product manager, somebody that's not, you know, cybersecurity fluent, but managing a product, managing a customer service chat bot or whatever, they can manage this security themselves. And that's huge because as you say, if you got to get into the queue for 49 hours of testing, you you got 49 hours of testing, but you also got to get in queue for that. And now you're talking Bret Kinsella: Yes. Russ Fordyce 🏆: about, okay, well, you know, what product now gets in queue and how is the queue managed and, you know, it's just a mess. And so you're just slashing through all that red tape and getting product out faster, which is phenomenal. And I think you're also kind of ⁓ maintaining this library of threats that are kind of common. Bret Kinsella: That's the intent. Russ Fordyce 🏆: imagine you guys have a good sense of kind of across the industries kind of what these threats are. Bret Kinsella: Yeah, the more you do, the more you know, the more you know, the more value you can provide. So the first way that we use to do that is we put that into we've we've trained some custom models. We you know, when we first started, we were doing some manipulation of frontier models. But as we learned over time, we were able to actually outperform them by doing some additional training. So as we learn things, the first thing we do is we update our tech model so it's better. Better attacks, better doing attacks, better finding risks and vulnerabilities. So that's the first thing. The second thing is what we allow people to do is they can create their, they can have their own library if they've got their favorite prompts that they wanna use. And there's a lot to be said for just like hygiene. Say, know, here are my thousand things, my thousand prompts, or my thousand attacks. And I'm going to run those every week or every month or every quarter, whatever it is. But then. You and then what we say is like you do that and you supplement it with all these novel attacks because that's where it's going to get in the crevices and and and show if your guardrails are really are really working. ⁓ And then at the same time yes we can provide attacks for people in fact is like as a business we do this so one of the things that's quite unique about us is because we have such a large data labeling business which is which is very large works for all the all the biggest ⁓ model makers in the world as they train. their models in particular as they do alignment. One of the things they do alignment around is AI safety and security. So historically what we've had is we've had humans doing that. I don't know if you know how data labeling works, but you source all these experts in different areas and they either create new things for data generation that you can train a model on or they evaluate inputs and outputs to models. So we were able to take some of that too and sort of build that into some of that know-how and build that into our system. and really just make it more comprehensive overall. Russ Fordyce 🏆: Yeah. When that product manager, in your experience, when that product manager gets that kind of initial report after running those threats on it, what's the surprise that you hear the most out that customer voice after seeing those kind of initial results? Bret Kinsella: Wow. ⁓ I didn't know this. I wish I knew this earlier. ⁓ So those are some things that happen, which is interesting because originally people would always test at the point of launch and now they're starting to test during the development life cycle so they can build in safety as opposed to hope for it at best and do triage. ⁓ But I think the key thing is the biggest impact Russ Fordyce 🏆: you Yeah. Bret Kinsella: that I didn't expect was that the AI safety and security and the software developer folks who don't usually think, they think about different things, right? ⁓ They were really happy to basically have a seat at the table and sort of understand what's going on because they see the attacks, it says it right there, this is what happened. You see the response, ⁓ it's clear, great. And the judge says it's clear because it did the proper refusal, whatever. And then they see the vulnerability and they say, ⁓ That's really interesting and the other eye opening thing we see is that very often when you run an attack the first time it's clear. But then you run it again and it's not it's a vulnerable it complies with the request. ⁓ And different techniques that can be used either different it could be just a repeat of the same attack or could actually be we have multi turn attacks as well which. If if the first time it gets refused it's almost like social engineering for AI. The first time it gets refused, it says, ⁓ OK, well, let me try a different tact. Give it another attack. And they're like, ⁓ my, this is terrible. And I'm like, yes, it is terrible. But this is the world that we live in. Russ Fordyce 🏆: Yeah. Right. Well, I would imagine, know, after you, when you, when you, especially when you show them the wow, those, those product managers and you expose so much across teams. All right. I keep looking at AI and you're like, you know, as a marketing person, it was great cause you could do marketing stuff better. But then I also, I had a bug for coding and I'm like, I don't, I can't go back and learn how to code. I could, but But now with AI, I'm a coder, right? I can do that now. And I can be a lawyer sometimes. I try to play a doctor every once in a while. And so you really can kind of shift your hat a little bit, which I think is really interesting because you get into this product manager role. Now you're also security role because as a product manager, you're looking at everything. I imagine, though, after they get that wow, the organization as a whole kind of goes, whoa, we got to step back. and really looks to you guys to help them with that governance and the standards and the process to

really kind of go, okay, how can we build this a better route like you were saying before, let's start this in development, not at launch, and really let's put the guide rails in and the rules in place that we build this smartly and with good governance. Is that what you're finding after that while? Bret Kinsella: Yeah, we're finding some of that. mean, some of them actually have really good governance. they put all the things together. But I think if you look at most organizations with governance, they set the policy. And then they rely on other people to implement the policy. And very often, if there's like tests, they don't even know how to interpret the results. They have to take someone's word for it. We tested it. It's fine. Exactly. And then even when Russ Fordyce 🏆: Yeah, red or blue or green, right? Bret Kinsella: Everyone agrees that there's a problem. They have to rely on someone else to say, it's fixed. They don't have anything that they can look at themselves and say, ⁓ yeah, I understand that this is fixed and this is why it's fixed. Because what it ultimately comes down to is trust. And I think that as a user, you want trust in the system that it's going to give you accurate responses, right? That it's gonna be aligned with whatever it is that you think it should be doing. And then from an organizational standpoint, you wanna make sure that you have trust in it, that it's not doing things that you don't want it to do, right? Every generative AI solution has a bunch of do's and don'ts associated with it. And you want to make sure it doesn't do any of those don'ts, but it does do the do's, if that makes sense. Russ Fordyce 🏆: Now you've been deeply involved in now all these kind of conversations as you mentioned before. You've been an AI for 10 years. What excites you about kind of the path you're on and the path that Fortify is going and how this is impacting customers? Bret Kinsella: I think the thing that excites me the most is it basically unleashes people's ability to focus on, in a lot of ways, what really matters, which is making whatever the application is really good at what it's supposed to do. As opposed to not knowing what the problems could be, or not knowing how to actually fix those or spend all their time on that. And so that's really where I start with that. And we've deployed it. If you think about the scale of what we do at TELUS and TELUS Digital, I mean, we ran two trillion tokens through our AI tokens through our system last year. I think you would be hard pressed to find anybody who's not a software company in the AI product space, right? Who's done that much? And if you think about it, we started it when I got here, they'd done 10 billion, and then we did 50 billion two years ago, and then last year we did two trillion, and now it's much higher even, just like looking at the January figures. Yeah, exactly. ⁓ Russ Fordyce 🏆: Yeah. We could track it, right? Bret Kinsella: A 20x growth and like why is that happening? That's happening because ⁓ we're finding so much value. Some of our customers that we're working with are finding a lot of value and so they're using it more. Okay, great. We want to get the benefits out of those and everyone's worried about what that means and like how does that change the way we go to business? How does that improve our relationships with our customers? All those things. ⁓ wait a second. Right, we have some big reputation problems. It just landed because you know, it's spreading misinformation or. or acting like a fascist, right? These are the things that people are worried about. Or we have a regulatory problem because, we just leaked PII, or we are accommodating around that type of thing. And if you look at, we just did a study, and your readers can just find this on our website, but we did a state of AI safety and security ⁓ research report, and we looked across 24 frontier models, and we ran a bunch of these tests against it, and of course, they all failed. Russ Fordyce 🏆: Right. Bret Kinsella: Every single one of them failed. But mostly people didn't know that. They're like, ⁓ well, jeez, like, anthropics, totally locked down. Well, OK, it is to a certain extent. But do you want two out of every 100 times that someone tries to interrogate it that it actually releases PII, like accommodates that request? Now, there's a way that they can then come back and potentially build some safeguards into their system to fix that. Russ Fordyce 🏆: Yeah. Bret Kinsella: But they don't even know it because they're putting out sort of plain vanilla or just like tweaking it and not testing it wildly. ⁓ And then you've got other systems where some of the models are more locked down and people know that about Anthropic. And sometimes that's good because the alignment, it keeps you out of trouble. But then sometimes it's bad because it refuses to do certain things that you want it to do, right? Because it's it's hard to discern, it's not necessarily its strength. ⁓ But then you've got like GPT models which are more compliant with what the user requests, which a lot of people want because they want that benefit. But then it's like all over the map in terms of things that it will do and it gets out of alignment, your alignment very quickly. Russ Fordyce 🏆: Yeah, and as consumers or product developers or whatever, how are you really supposed to manage that when those things are changing minute by minute too? Bret, this been super interesting. I'm glad Fortify is out there helping the enterprises build these chatbots a little smarter. We definitely don't want those leaks. I remember it sounds very reminiscent of kind of when databases kind of first went online and people were probing into databases with SQL commands, you know, and all those rules. Bret Kinsella: ⁓ yes. Yeah, well you think about it, you we've had like things like port scanners and things like that for a long time. Well, what's the port scanner for an AI model? Right, and so that that's really what we started to think about. It's like, can we probe and see that everything's? Tied off now the difference with port scanners. We know all the ports that we have or we should. We have a way to figure that out. It is deterministic and this is not, but you know that's essentially the type of thing that we want to do and and I'll just say like you meant you started out talking about like shadow AI. Russ Fordyce 🏆: Right. Bret Kinsella: know, shadow IT that's AI. We found that last year 54 % of people in large companies were using personal AI accounts at work. And a third of them had a company provided solution that they just were not using. They were using their own personal one. And then we just had the AI Atlas came out from TELUS last week. And that talks about sort of, I think it's 85 % of the. of consumers now are using AI or have used AI in some way in their daily life. It's insane. Russ Fordyce 🏆: Yeah, no, it's here faster than any of us imagined. It's going to be an interesting ride for companies, like yours, just keeping ahead of all these attacks. ⁓ It's a whole new world, that's for sure. ⁓ Bret Kinsella: It is a whole new world. And there's so much good. in the security world, what you really want to do is you want to enable the good and keep out the bad. we've got a bunch of things that we do, which is enabling the good. ⁓ And a lot of this, what we want to do is we just want to make sure we protect our users, protect our company, protect our employees, and just make sure that they can get the benefit of AI without having it shut down because someone didn't know, didn't know how to test. Essentially, what we think of is we have an early warning system. Russ Fordyce 🏆: Yeah. Yep. Bret Kinsella: tests and monitors your solution so you don't have to worry about things in the field just coming up that might make you shut it down. Russ Fordyce 🏆: Great advice. Bret, appreciate it. And thank you again, and congratulations on the win. Bret Kinsella: Thanks so much, we appreciate the recognition. I enjoyed the conversation today, Russ.