Anmol: Welcome to the AI Security Update, where we discuss the latest in AI and security. Today I'm so excited to introduce our guest for this episode, Yacine from Escape. So welcome to the show, Yacine. Can you please introduce yourself and your background?
Yacine: Yeah, so I'm currently an R &D engineer at Scape working in AI for security. My background, I think, started with machine learning, like many years ago before Child GPT. And then I pivoted into ⁓ cybersecurity. all AI things, I saw the opportunities both in AI for security and securing AI as well. And I also joined OSPI exchange as a member like Lester. And yeah, the goal of the OSPI exchange is to awareness around AI security, explain technical concepts to security professionals and also non-security professionals. yeah, we have a role in everything related to compliance, AI security compliance, European AI Act, et cetera. So yeah, my background is like really into AI and security, I would say.
Anmol: Yeah, I think the OWASP AI Exchange is a really great resource for those who are not aware. Oftentimes we refer to it in the industry as well, right? OWASP top 10, whether it's ML or AI or now AI agents. And they always have really great guidance for the industry that we actually refer to in even standards documents. So really, really great work from the OWASP AI Exchange. Yacine, based on your experience, whether it's at Escape or your involvement with OWASP, what are some challenges you've seen with AI security?
Yacine: Yeah, so I actually made a lot of discovery calls with many clients, enterprise. I don't think I saw like two scenarios. The first one is companies looking for quick wins to secure AI. So like they AI models and they want to filter out ⁓ what goes the model. using AI firewall, these kind of things. So I would say it's the first company profile ISO. The second one is companies that are more advanced in their AI usage. They are already deploying AI agents internally. So they are looking for ways to secure not only the model, but also the agents, the actions that the agent takes. you know, this ⁓ all around, less privilege, these kind of things. So ⁓ say those two types of companies are the ones I saw mainly on experience.
Anmol: Yeah, I think we have two of ⁓ companies, right? We have the AI enthusiasts that they just want to deploy AI right away. ⁓ then we have the more sophisticated AI companies that really want to understand securing the AI model, securing the AI agent. So based on your perspective, What is some advice you have in terms of securing AI agents and also securing AI models?
Yacine: Yeah, I think it can be really complicated for some security professionals because you're securing a big organization, a of developers are playing around with AI tools and maybe you don't know what they do with it. So would say the first thing is observability, like what is going on in the company, which AI tool is used. I think that still can be hard because some developers can... ⁓ maybe become frustrated because they get too much respected from AI usage. So I think that's where the balance comes in between letting enough innovation to be used and also be cautious about what's going on. So I say ⁓ the advice would be not limit yourself to model security. Quite early on, was advocating for pivoting attention to ⁓ securing because at that time, ⁓ early in AI security, majority of content, advices were around, this new prompt technique is out, ⁓ be of that. The real risks is can happen after this ⁓ bypass, this prompt injection thing. So first of all, ⁓ which you have internally in your company, whether they are coding agents or whatever. And then the ⁓ cybersecurity concepts least privilege. et cetera, security by design. So I'll say this is the first one. And also for the companies that are building now AI products, I would AI security is like a very topic for you because you're not only using AI internally, but you're trading clients data with your AI product. So you have to be even more cautious about that. So I would say, my main advice would be, like, don't get caught into all the noise around prompt injection, et cetera. This is important, I think, to keep track of what's happening, to have models that are up to date in their defense. it's just the first step. If this step is bypassed. then it can be more dangerous. So, yeah, based on what I saw, think that would be the main advice I would give.
Anmol: Yeah, and I think people need to also be aware that it's not just about generative AI prompt injection. It's also predictive analytics as well. So some of the same machine learning security concepts still apply even to generative AI. So there are some attacks we talk about in adversarial machine learning. I'm sure Yacine you're probably aware of these, but example, poisoning is the most popular one. So it also about data security as well as where is the AI model getting the data from and if it's learning incorrect information of course it will give you an incorrect result. you also mentioned security by design so there are also methods you can do to make your systems more secure. ⁓ security, adding model security, there are ways to add adversarial robustness to the machine learning models. maybe adversarial training is a good example. So you can train the AI model on of attacks. ⁓ You train it on these examples of attacks to try to see. what will make it more robust. an example is an adversarial example. The classic example we use to explain this security issue is it's an image of a panda, but then the AI model says that it's misclassified to be a gibbon. So you can add it to an model and then it will misclassify that image and think, ⁓ that's a gibbon. And that's an adversarial example because to people, it looks like a panda image, but than to others, it looks like a gibbon. And you can actually train your AI model to recognize these adversarial examples to also make your model more robust. then I would say a lot of it is around governance as well based on my experience. So it's about people understanding that these AI models can be attacked and having incident response plans in place as well.
Yacine: you Yeah, absolutely. think you're absolutely right regarding that. Like, majority of the attention is towards LLMs, LLM security, but you're right saying that a lot of models that are present since many, years are based on classic machine learning models. And those are still at risk and should be considered with that, especially regarding you know, image detection, take the example of a car, autonomous car. I that the risk can be really big for the safety of you know, humans, passengers, because if a car just makes a mistake regarding what's happening on the road, that can have huge consequences. Not many security professionals know the risks around that as I said, all hype and the attention is around LLMs of course.
Anmol: Yes, I know because so I did a lot of my research on adversarial machine learning before we had LLMs and generative AI as the hype. I've already talked about this in other episodes, but as you know, ⁓ an evolution of natural language processing. So a lot of these older machine learning and AI technologies, it's really think of it like your camera, how your camera has evolved is might be a good example. Like initially we all had to carry really big bulky cameras and now we have you know smartphones that can take pictures for us so it's like LLM is like just an evolution of the AI technologies but it's the same same underlying concepts I would say still apply even to LLMs they just call it different names or different terms
Yacine: Yeah, definitely.
Anmol: And one other question I have for you is, I know Escape does a lot related to API security. So based on your perspective, what are the interactions or the combination of AI and API security?
Yacine: Yeah, that's a really good question because, you know, I think today the many people understood that AI security is now like really about securing the agent, securing what the agent does. And what we see now is that the agent is using, you know, MCPs, CLI. But is like the technology that is behind all of that is, you know, API, classic APIs. So I think a lot of LLM security or AI security generally is about API security because if your underlying technology isn't secure, of course, if AI is going take an action that could be dangerous, of course, like take, for example, all things regarding the things that happened recently with some incidents we as well. The problem was that the agent was taking ⁓ being stopped because the least framework wasn't applied. The ⁓ API permissions were allowed. was given long-lived API keys, all sorts of things. So I think that a lot of problems we are seeing and will see. regarding agent security is around classic patterns, application security, API security, things here.
Anmol: Yeah, I agree. think there's a lot of hype around AI security, but a lot of the same principles apply. you give an example for the audience about some classic API security controls that they should be aware of?
Yacine: Yeah, so I think the first thing is to treat the agent like a new kind of identity. In the industry, we already talk about non-human identity, everything related to service accounts, et cetera. But now we have a new kind of non-human identity, which is AI agents. So would say, especially regarding tokens, API keys, those kind of things. Beware of treating them like humans and them too much ⁓ or even in the time, ⁓ that in the time and also in the permissions. I think ⁓ ⁓ first step. But course, before that, the action should be taken ⁓ is, first all, which agents are running, for example, inside your organization. properly the permissions for agent.
Anmol: So another question I have is based on what you've seen in the news related to the current events, ⁓ there one current event that you think the audience should know about related to AI security?
Yacine: Yeah, actually, think, like recently an incident happened with Lite-LLM. I think you heard about it. basically what was the incident is that like ⁓ is technology that helps like routing LLM models. And they also have ⁓ a Python And the thing is ⁓ a supply incident happened because the ⁓ the library using some compromised action that was from Trevi, I think. then ⁓ tokens accounts compromised, and then the attackers were allowed to publish malicious versions of LiteLLM on PyPy, which means that Each one of the developers was using Lite-LLM with the wrong version at the time was exposed to just being by this supply chain attack. And I think it's really interesting because it shows how AI security is not only about agents, model it's related to everything that we have known in the past. like supply chain security that is still a big subject. And it shows that the more dependencies you have, the more vulnerable you get, of course. And it's also scary ⁓ see how AI is accelerating hacking. But hopefully, AI will also accelerate defense. And ⁓ that be really interesting to track.
Anmol: another question I have ⁓ is to your work at OWASP exchange. ⁓ if could just explain what it's like be part of that exchange ⁓ for the audience interested or just some more information about it.
Yacine: Yeah, so first of all, of course, you can access the website oaspai.org. So basically what we do is that we provide free open source resources to understand what is really about AI security for security professionals and also non-security professionals because I know that... some folks in AI engineering, ML engineering are interested in securing the thing they build. So yeah, first of all, ⁓ what we do. And we also go to ⁓ conferences speak with people about everything related to AI security. also a big as I mentioned, is being being responsible for things related to compliance. The AI exchange has played an important role regarding the AI Act. And the deadline is coming fast for companies who want to comply to AI security regarding this act. So I think that was a really important work. So I encourage ⁓ to... ⁓ Just go to the website to learn more about the OWASP does.
Anmol: And you mentioned that you're providing guidance to both security and non-security professionals. ⁓ if we think about the non-security professionals, ⁓ an everyday scenario that explains an AI security concept?
Yacine: a good question. I think based my experience, there's something quite funny my work is around AI for security, so building agents, but security companies aren't immune AI security as well. we build agents ⁓ ourselves, we must make sure the first of all the agents cannot be controlled by external has right ⁓ permissions, right tools at the right moment. I think security professionals, like ⁓ people security solutions, we should be aware that we are also responsible for building secure AI solutions. And we've seen with security companies as well. So that can happen in all industries. And I think this dual identity is quite funny, but it's really important for at the same time to ensure that the agents we build for security are secure themselves.
Anmol: Yeah, great point. So Yacine, thank you so much for being on the show. ⁓ my last question for you is, ⁓ there any resources you recommend for people to learn more, whether they want to learn how to build these AI agents or they want to learn more about something else we discussed today?
Yacine: Yeah, sure. Of course, the first resource I would recommend is ⁓ to osp.ai.org. You can type slash roadmap and you will see everything listed about many subjects around security. And ⁓ I of best is just to do things by yourself, I would say. Try out new AI tools. Of course, Be aware of the risk. seen it with OpenCLO. But I think, just grab ⁓ a open source projects. You can deploy it locally, play with it, understand it, all things cloud code, et cetera. Like in those days, like ⁓ my start today, usually goes to, ⁓ you know, playing with those tools to stay updated. And yeah, I think by practicing, are also realizing the risks that could happen.
Anmol: Yeah, thanks so much. think that's great advice. Sometimes you can only learn by doing and by experimenting with these tools. And I always advise people if they want to learn Python, there are so many free tutorials on the internet as well. So thank you so much, Yacine, for being on the show. Is there anything else you want to add before we close?
Yacine: not really. I we covered many topics. I really like this conversation. So thank you, Anmol, for having me here. And yeah, just stay curious and just ⁓ continue to new things about AI and security. There's so much things to discover. This is an ⁓ industry that being evolving. So yeah, just ⁓ stay curious, think.
Anmol: Yeah, just say curious. Great, great advice. this concludes our episode of the AI Security Update. We'll be back next time with more great conversations. Thank you. ⁓
Yacine: Thank you.
Yacine: Yeah, so I'm currently an R &D engineer at Scape working in AI for security. My background, I think, started with machine learning, like many years ago before Child GPT. And then I pivoted into ⁓ cybersecurity. all AI things, I saw the opportunities both in AI for security and securing AI as well. And I also joined OSPI exchange as a member like Lester. And yeah, the goal of the OSPI exchange is to awareness around AI security, explain technical concepts to security professionals and also non-security professionals. yeah, we have a role in everything related to compliance, AI security compliance, European AI Act, et cetera. So yeah, my background is like really into AI and security, I would say.
Anmol: Yeah, I think the OWASP AI Exchange is a really great resource for those who are not aware. Oftentimes we refer to it in the industry as well, right? OWASP top 10, whether it's ML or AI or now AI agents. And they always have really great guidance for the industry that we actually refer to in even standards documents. So really, really great work from the OWASP AI Exchange. Yacine, based on your experience, whether it's at Escape or your involvement with OWASP, what are some challenges you've seen with AI security?
Yacine: Yeah, so I actually made a lot of discovery calls with many clients, enterprise. I don't think I saw like two scenarios. The first one is companies looking for quick wins to secure AI. So like they AI models and they want to filter out ⁓ what goes the model. using AI firewall, these kind of things. So I would say it's the first company profile ISO. The second one is companies that are more advanced in their AI usage. They are already deploying AI agents internally. So they are looking for ways to secure not only the model, but also the agents, the actions that the agent takes. you know, this ⁓ all around, less privilege, these kind of things. So ⁓ say those two types of companies are the ones I saw mainly on experience.
Anmol: Yeah, I think we have two of ⁓ companies, right? We have the AI enthusiasts that they just want to deploy AI right away. ⁓ then we have the more sophisticated AI companies that really want to understand securing the AI model, securing the AI agent. So based on your perspective, What is some advice you have in terms of securing AI agents and also securing AI models?
Yacine: Yeah, I think it can be really complicated for some security professionals because you're securing a big organization, a of developers are playing around with AI tools and maybe you don't know what they do with it. So would say the first thing is observability, like what is going on in the company, which AI tool is used. I think that still can be hard because some developers can... ⁓ maybe become frustrated because they get too much respected from AI usage. So I think that's where the balance comes in between letting enough innovation to be used and also be cautious about what's going on. So I say ⁓ the advice would be not limit yourself to model security. Quite early on, was advocating for pivoting attention to ⁓ securing because at that time, ⁓ early in AI security, majority of content, advices were around, this new prompt technique is out, ⁓ be of that. The real risks is can happen after this ⁓ bypass, this prompt injection thing. So first of all, ⁓ which you have internally in your company, whether they are coding agents or whatever. And then the ⁓ cybersecurity concepts least privilege. et cetera, security by design. So I'll say this is the first one. And also for the companies that are building now AI products, I would AI security is like a very topic for you because you're not only using AI internally, but you're trading clients data with your AI product. So you have to be even more cautious about that. So I would say, my main advice would be, like, don't get caught into all the noise around prompt injection, et cetera. This is important, I think, to keep track of what's happening, to have models that are up to date in their defense. it's just the first step. If this step is bypassed. then it can be more dangerous. So, yeah, based on what I saw, think that would be the main advice I would give.
Anmol: Yeah, and I think people need to also be aware that it's not just about generative AI prompt injection. It's also predictive analytics as well. So some of the same machine learning security concepts still apply even to generative AI. So there are some attacks we talk about in adversarial machine learning. I'm sure Yacine you're probably aware of these, but example, poisoning is the most popular one. So it also about data security as well as where is the AI model getting the data from and if it's learning incorrect information of course it will give you an incorrect result. you also mentioned security by design so there are also methods you can do to make your systems more secure. ⁓ security, adding model security, there are ways to add adversarial robustness to the machine learning models. maybe adversarial training is a good example. So you can train the AI model on of attacks. ⁓ You train it on these examples of attacks to try to see. what will make it more robust. an example is an adversarial example. The classic example we use to explain this security issue is it's an image of a panda, but then the AI model says that it's misclassified to be a gibbon. So you can add it to an model and then it will misclassify that image and think, ⁓ that's a gibbon. And that's an adversarial example because to people, it looks like a panda image, but than to others, it looks like a gibbon. And you can actually train your AI model to recognize these adversarial examples to also make your model more robust. then I would say a lot of it is around governance as well based on my experience. So it's about people understanding that these AI models can be attacked and having incident response plans in place as well.
Yacine: you Yeah, absolutely. think you're absolutely right regarding that. Like, majority of the attention is towards LLMs, LLM security, but you're right saying that a lot of models that are present since many, years are based on classic machine learning models. And those are still at risk and should be considered with that, especially regarding you know, image detection, take the example of a car, autonomous car. I that the risk can be really big for the safety of you know, humans, passengers, because if a car just makes a mistake regarding what's happening on the road, that can have huge consequences. Not many security professionals know the risks around that as I said, all hype and the attention is around LLMs of course.
Anmol: Yes, I know because so I did a lot of my research on adversarial machine learning before we had LLMs and generative AI as the hype. I've already talked about this in other episodes, but as you know, ⁓ an evolution of natural language processing. So a lot of these older machine learning and AI technologies, it's really think of it like your camera, how your camera has evolved is might be a good example. Like initially we all had to carry really big bulky cameras and now we have you know smartphones that can take pictures for us so it's like LLM is like just an evolution of the AI technologies but it's the same same underlying concepts I would say still apply even to LLMs they just call it different names or different terms
Yacine: Yeah, definitely.
Anmol: And one other question I have for you is, I know Escape does a lot related to API security. So based on your perspective, what are the interactions or the combination of AI and API security?
Yacine: Yeah, that's a really good question because, you know, I think today the many people understood that AI security is now like really about securing the agent, securing what the agent does. And what we see now is that the agent is using, you know, MCPs, CLI. But is like the technology that is behind all of that is, you know, API, classic APIs. So I think a lot of LLM security or AI security generally is about API security because if your underlying technology isn't secure, of course, if AI is going take an action that could be dangerous, of course, like take, for example, all things regarding the things that happened recently with some incidents we as well. The problem was that the agent was taking ⁓ being stopped because the least framework wasn't applied. The ⁓ API permissions were allowed. was given long-lived API keys, all sorts of things. So I think that a lot of problems we are seeing and will see. regarding agent security is around classic patterns, application security, API security, things here.
Anmol: Yeah, I agree. think there's a lot of hype around AI security, but a lot of the same principles apply. you give an example for the audience about some classic API security controls that they should be aware of?
Yacine: Yeah, so I think the first thing is to treat the agent like a new kind of identity. In the industry, we already talk about non-human identity, everything related to service accounts, et cetera. But now we have a new kind of non-human identity, which is AI agents. So would say, especially regarding tokens, API keys, those kind of things. Beware of treating them like humans and them too much ⁓ or even in the time, ⁓ that in the time and also in the permissions. I think ⁓ ⁓ first step. But course, before that, the action should be taken ⁓ is, first all, which agents are running, for example, inside your organization. properly the permissions for agent.
Anmol: So another question I have is based on what you've seen in the news related to the current events, ⁓ there one current event that you think the audience should know about related to AI security?
Yacine: Yeah, actually, think, like recently an incident happened with Lite-LLM. I think you heard about it. basically what was the incident is that like ⁓ is technology that helps like routing LLM models. And they also have ⁓ a Python And the thing is ⁓ a supply incident happened because the ⁓ the library using some compromised action that was from Trevi, I think. then ⁓ tokens accounts compromised, and then the attackers were allowed to publish malicious versions of LiteLLM on PyPy, which means that Each one of the developers was using Lite-LLM with the wrong version at the time was exposed to just being by this supply chain attack. And I think it's really interesting because it shows how AI security is not only about agents, model it's related to everything that we have known in the past. like supply chain security that is still a big subject. And it shows that the more dependencies you have, the more vulnerable you get, of course. And it's also scary ⁓ see how AI is accelerating hacking. But hopefully, AI will also accelerate defense. And ⁓ that be really interesting to track.
Anmol: another question I have ⁓ is to your work at OWASP exchange. ⁓ if could just explain what it's like be part of that exchange ⁓ for the audience interested or just some more information about it.
Yacine: Yeah, so first of all, of course, you can access the website oaspai.org. So basically what we do is that we provide free open source resources to understand what is really about AI security for security professionals and also non-security professionals because I know that... some folks in AI engineering, ML engineering are interested in securing the thing they build. So yeah, first of all, ⁓ what we do. And we also go to ⁓ conferences speak with people about everything related to AI security. also a big as I mentioned, is being being responsible for things related to compliance. The AI exchange has played an important role regarding the AI Act. And the deadline is coming fast for companies who want to comply to AI security regarding this act. So I think that was a really important work. So I encourage ⁓ to... ⁓ Just go to the website to learn more about the OWASP does.
Anmol: And you mentioned that you're providing guidance to both security and non-security professionals. ⁓ if we think about the non-security professionals, ⁓ an everyday scenario that explains an AI security concept?
Yacine: a good question. I think based my experience, there's something quite funny my work is around AI for security, so building agents, but security companies aren't immune AI security as well. we build agents ⁓ ourselves, we must make sure the first of all the agents cannot be controlled by external has right ⁓ permissions, right tools at the right moment. I think security professionals, like ⁓ people security solutions, we should be aware that we are also responsible for building secure AI solutions. And we've seen with security companies as well. So that can happen in all industries. And I think this dual identity is quite funny, but it's really important for at the same time to ensure that the agents we build for security are secure themselves.
Anmol: Yeah, great point. So Yacine, thank you so much for being on the show. ⁓ my last question for you is, ⁓ there any resources you recommend for people to learn more, whether they want to learn how to build these AI agents or they want to learn more about something else we discussed today?
Yacine: Yeah, sure. Of course, the first resource I would recommend is ⁓ to osp.ai.org. You can type slash roadmap and you will see everything listed about many subjects around security. And ⁓ I of best is just to do things by yourself, I would say. Try out new AI tools. Of course, Be aware of the risk. seen it with OpenCLO. But I think, just grab ⁓ a open source projects. You can deploy it locally, play with it, understand it, all things cloud code, et cetera. Like in those days, like ⁓ my start today, usually goes to, ⁓ you know, playing with those tools to stay updated. And yeah, I think by practicing, are also realizing the risks that could happen.
Anmol: Yeah, thanks so much. think that's great advice. Sometimes you can only learn by doing and by experimenting with these tools. And I always advise people if they want to learn Python, there are so many free tutorials on the internet as well. So thank you so much, Yacine, for being on the show. Is there anything else you want to add before we close?
Yacine: not really. I we covered many topics. I really like this conversation. So thank you, Anmol, for having me here. And yeah, just stay curious and just ⁓ continue to new things about AI and security. There's so much things to discover. This is an ⁓ industry that being evolving. So yeah, just ⁓ stay curious, think.
Anmol: Yeah, just say curious. Great, great advice. this concludes our episode of the AI Security Update. We'll be back next time with more great conversations. Thank you. ⁓
Yacine: Thank you.
