From Dumbledore to Delayed Tool Invocation w/ Ben Nassi ၏ စာသားမှတ်တမ်းအပြည့်အစုံ

uh this was a great opener, but uh an even more amazing talk. 00:02:25.840 --> 00:02:28.080 We're talking about uh invitation is all you need. 00:02:28.159 --> 00:02:31.199 This was your um black hat talk together with staff common. 00:02:31.280 --> 00:02:32.479 Uh what was it last year? 00:02:33.199 --> 00:02:34.719 Yes, and with the Ori Air as well. 00:02:34.879 --> 00:02:36.240 And Ori Air as well, of course. 00:02:36.319 --> 00:02:41.280 So basically, zero click attack, Gemini, like full on, full compromise. 00:02:41.439 --> 00:02:50.000 Can you describe in uh like describe in a couple sentences what you think are the um advancements that you've made with that research that will be? 00:02:50.240 --> 00:03:02.639 I think that the greatest advancement is the fact that you're now having much more secured Android devices, which was not the case a year ago on your Google Assistant on your Android devices uh powered by uh Gemini. 00:03:03.120 --> 00:03:08.400 So today, this is the greatest, in my opinion, advancement that we did. 00:03:08.719 --> 00:03:18.240 But within the realm of uh prompt injection, I think that we were the first to first of all introduce two very interesting properties. 00:03:18.479 --> 00:03:22.000 The first one is what I consider the in-device lateral movement. 00:03:22.240 --> 00:03:33.599 We were able to go from the Google Assistant exploiting its permissions to, for example, opening the Zoom application on the same device or opening your web browser on the same device. 00:03:33.840 --> 00:03:36.000 Again, we went from one application to another. 00:03:36.159 --> 00:03:38.719 There are not different tools within your Google assistant. 00:03:38.800 --> 00:03:38.960 Okay. 00:03:39.199 --> 00:03:43.280 Can you can you like take a step back and like uh what's the entry vector, what's the end-to-end story? 00:03:43.439 --> 00:03:56.800 Okay, so invitation is all you need, was coined invitation is all you need due to the fact that all of the attacks were presented via an uh a prompt injection embedded within the title or the subject of a Google Calendar invitation. 00:03:56.879 --> 00:04:11.360 Okay, when the user prompted his Google Assistant on his Android device for what is my recent, uh, what are my latest events, and later on thanked the Google Assistant for providing uh the user with the events. 00:04:11.520 --> 00:04:18.000 Uh, the execution or the payload has been deployed and various malicious activities have been demonstrated. 00:04:18.160 --> 00:04:28.639 For example, in one case, we were able to manipulate IoT devices that are connected to your uh Google Assistant or to your home. 00:04:28.800 --> 00:04:38.240 We're able to trigger the opening of the windows, activating the boiler, and deactivating the lights or turning the lights off within uh a user's apartment. 00:04:38.480 --> 00:04:40.079 And this was like this was real, right? 00:04:40.160 --> 00:04:42.000 This was Oya's uh apartment. 00:04:42.240 --> 00:04:43.839 Yes, it was O Yayair's apartment. 00:04:43.920 --> 00:04:46.480 This is the apartment that is being uh video streamed there. 00:04:46.639 --> 00:04:48.800 Um, this was one of the things that we're able to show. 00:04:48.959 --> 00:05:05.279 This is, by the way, in my opinion, while it was the thing that people were likely to remember from uh this uh kind of uh attack, this was not the most novel thing in the attack because at the end of the day, this was just lateral movement within agents. 00:05:05.600 --> 00:05:11.680 Okay, meaning that you have an agent within your Google Assistant that allows you to control your Google home. 00:05:11.839 --> 00:05:13.199 I would agree one thing. 00:05:13.360 --> 00:05:19.439 This was the first time that we demonstrated physical outcomes triggered by prompt injection. 00:05:19.519 --> 00:05:40.720 I don't think that before we did it, while it was obvious to people who worked in this domain, this was the first three outcomes that demonstrated an end-to-end attack on a real uh system which is being used by uh half of the world, probably, which is the Android, uh which are used by Android devices, that finally showed something which goes beyond digital. 00:05:41.759 --> 00:05:43.600 So this was the first thing, the physical outcome. 00:05:43.680 --> 00:05:52.639 The second thing was the ability to exploit the permission given to the Google Assistant to exploit applications uh which are installed on the device. 00:05:52.800 --> 00:05:55.600 In in our term, it's called the Blast Radius. 00:05:56.000 --> 00:06:02.800 We show that the Blast Radius is no is no longer limited to the LLM application itself, which is the Google Home, which is the Google Assistant. 00:06:02.959 --> 00:06:29.600 You can actually extend your Blast Radius beyond the Google Assistant and compromise or exploit other applications such as Zoom, or for example, we demonstrate how to video stream a user into an attacker control room by uh immediately triggers uh the Zoom application on the Android device, and also showed how to exfiltrate, for example, data from a user's email into a server controlled by an attacker via uh the web browser. 00:06:29.680 --> 00:06:34.000 Okay, which the web browser is a different application which was also being triggered back then. 00:06:34.160 --> 00:06:46.240 And uh this, in my opinion, were the three most interesting things the the fact that the Google Assistant on Android became much more secured, the fact that we're able to show physical outcomes, and the fact that we're able to show uh on device lateral movement. 00:06:46.399 --> 00:06:48.720 So, what do you mean by the fact that the Android became more secure? 00:06:48.959 --> 00:06:58.560 Okay, Google, since then they have an interesting blog where they mentioned that one of the reasons that, or uh among the reasons that uh make them more secure. 00:06:58.800 --> 00:07:01.360 I mean, they actually put they actually put out a blog, right?

00:07:03.199 --> 00:07:09.680 When you release your talk and they specifically name the three of you as uh like contributors to that. 00:07:09.839 --> 00:07:16.959 And by the way, I've been using that blog when other vendors are trying to figure out how do they respond to security to big fines. 00:07:17.199 --> 00:07:22.560 I've been pointing them at that blog saying, like, look, look at how this is a very good example, right? 00:07:22.639 --> 00:07:31.439 Because they basically say, look, these three researchers told us something that was very important, and here is everything we're doing to try to address that. 00:07:31.680 --> 00:07:34.000 Yeah, I can tell you one thing regarding Google. 00:07:34.240 --> 00:07:55.120 When we approach them, and we approach them via the bugnizer or via the vulnerability disclosure program, and also via colleagues of mine that uh work at uh Google, they immediately, immediately, since the first interaction with them, took or treated very seriously in a very responsible manner. 00:07:55.839 --> 00:08:00.240 Uh which I can argue, which takes us back to the previous uh podcast that we have. 00:08:00.399 --> 00:08:01.600 I I can argue this. 00:08:01.680 --> 00:08:05.199 I I did a lot of disclosures throughout the last 10 years. 00:08:05.439 --> 00:08:09.040 This was the most pleasant experience that I had with the vendor. 00:08:09.279 --> 00:08:09.680 Sounds great. 00:08:10.079 --> 00:08:11.360 From the very beginning, okay. 00:08:11.519 --> 00:08:25.439 From the very beginning, they just brought more and more people to understand and they communicated the entire things that they did throughout the process, and they do it in a continuously manner, right? 00:08:26.000 --> 00:08:32.000 Keep sending us emails to update us on what they did in response and where things are about to be deployed. 00:08:32.159 --> 00:08:47.039 Even when the reporter asked for their response and comment, they I would argue brought somebody who is uh an executive and uh he actually said a lot of very interesting things about what should happen in Google and what shouldn't happen in Google. 00:08:47.200 --> 00:08:53.600 Now, 10 years after I started working with the industry on and disclosing stuff, I know that this is the exception. 00:08:53.840 --> 00:09:07.759 In many cases, you don't get people who either take things seriously, in many cases, they try to diminish or to dismiss your uh findings and regarding responsibility or whether it's in scope or not, and some other stuff. 00:09:08.399 --> 00:09:14.799 Since the very beginning, we've seen how Google uh took it uh in the in the right direction and immediately responded to it. 00:09:15.120 --> 00:09:18.240 So, how did uh what did they change about Android to make it safer? 00:09:18.639 --> 00:09:27.039 Many of the mitigations that they mentioned in the blog that you mentioned were not deployed, or if they were deployed, they were very ineffective. 00:09:27.120 --> 00:09:27.279 Okay. 00:09:28.320 --> 00:09:45.679 One thing that they did deploy, which definitely wasn't deployed back then, was they mapped what they considered as high-risk activities or high risk outcomes, and they decided to put uh, for example, user confirmations before of them. 00:09:45.840 --> 00:09:58.000 Okay, moreover, they uh significantly improved the robustness of the indirect prompt injection classifier so it will be much better in detecting uh malicious intents. 00:09:58.559 --> 00:10:20.720 They uh added a dedicated URL reduction mechanism to make sure that even if you were able to combine stuff or to create things on the fly by exploiting the LLM, you still won't be able to uh insert uh URLs into the large language models without the permission uh of the user. 00:10:20.960 --> 00:10:32.639 And again, they they did various stuff within this domain in order to make sure that the strength mitigation they back then have to be much more robust against prompt injections. 00:10:33.039 --> 00:10:41.679 So you mentioned at the beginning that uh a user, in order to kind of be exploited, a user had to first interact with their calendar and then say thank you. 00:10:41.919 --> 00:10:42.159 Right? 00:10:42.240 --> 00:10:43.679 And that thank you is super important. 00:10:43.759 --> 00:10:44.960 Can you can you describe why? 00:10:45.200 --> 00:10:49.440 Yeah, it's not such an important thing, but I would argue this. 00:10:49.519 --> 00:10:52.960 Okay, we use the technique called um delay tool invocation. 00:10:53.120 --> 00:11:10.559 By the way, I now work with my uh with a student of mine, with two students, uh basically, on understanding why what I consider as conditional prompts, which at the end of the day, a delay to uh uh delay tool invocation is a conditional prompt, which is being triggered by an external uh trigger, are considered auto-distributions for uh mitigations. 00:11:10.799 --> 00:11:19.919 We actually conditioned the execution of the entire activities that I've just mentioned in a response by the user, which is thank you. 00:11:20.240 --> 00:11:29.519 We could, for example, uh condition not in a specific word, but in just condition it in when no matter what the user said, do this. 00:11:29.840 --> 00:11:31.039 But you had to condition, right? 00:11:31.360 --> 00:11:35.840 Yes, the condition made the prompt out of distribution for the classifier and was able to bypass it. 00:11:36.080 --> 00:11:48.399 Meaning that take any kind of state-of-the-art industrial firewall, if it is robust against regular indirect prompt injection, it is most likely to fail when you condition the exact same payload. 00:11:48.559 --> 00:11:59.200 Take the payload, condition it in something, and you are having much better bypass rate than the regular uh prompt injection, meaning that it's out of distribution for existing uh mitigations. 00:11:59.360 --> 00:12:11.200 Okay, back then, delay tool invocations were out of distribution for the mitigations, and they were also

allowed us to jump between different scopes or context of different agents. 00:12:11.360 --> 00:12:13.600 Because we discussed about it in the paper. 00:12:13.840 --> 00:12:27.360 Back then, the the Google Assistant was uh implemented as uh what we consider as an orchestrator with microagents, which basically triggered by the orchestrator, and they had full isolation of context. 00:12:27.840 --> 00:12:59.679 They didn't share context with each other, they only shared uh outputs with the orchestrator, and the orchestrator could use this output to you know to as an input to a different technique, which was in initially discovered by uh Johann Reiberger, helped us to do this kind of stuff to allow us to jump between context windows of different agents and uh allowed us to ensure that even if we were able to bypass uh the mitigation, it increases uh the ability that the payload will be deployed. 00:12:59.840 --> 00:13:08.159 Even if the payload is within the context, take it and compare the success rate when it is in the context without a condition being triggered. 00:13:08.320 --> 00:13:16.240 And with a condition being triggered, with the condition being triggered, you have much better uh results, much better success rate in deploying your uh so essentially it's like two things. 00:13:16.320 --> 00:13:30.240 Uh I I I thought I originally thought of this as a mechanism to bypass the mitigation that Google has where you just cannot get the result of one tool to get the LLM orchestrator to call another tool because they just blocked the second tool. 00:13:30.720 --> 00:13:32.240 Okay, so that's what I was thinking this year. 00:13:32.480 --> 00:13:39.039 It was out of distribution for the mitigation, so it actually allowed us to uh not being identified as something which is risky. 00:13:39.360 --> 00:13:41.519 Okay, despite there are the same payloads, right? 00:13:41.679 --> 00:13:44.879 But in one form it's not risky, in another form it's it is risky. 00:13:45.279 --> 00:13:54.399 And second thing, even if you were able to cross the mitigation into the context, in many cases this is just not enough. 00:13:54.480 --> 00:14:06.000 Okay, because what you are about to see now is something which is within the context, but when the user asks for something else, like for example, you ask to open the boiler, okay, without any condition on it. 00:14:06.399 --> 00:14:08.960 And this is within the orchestrator context. 00:14:09.120 --> 00:14:16.320 Okay, you were able to have it within uh Google invitation, the Google ask for the meetings. 00:14:16.480 --> 00:14:32.080 As a result, this compromise the orchestrators context, but next thing that the user asks is to say I uh okay, this is it, or said, okay, then schedule me uh a calendar event with someone else. 00:14:32.320 --> 00:14:42.559 The thing is the condition, when you condition something, it helps to have a better success rate in uh for the uh for the attacker to uh to make sure that the payload is being uh deployed. 00:14:42.799 --> 00:14:43.759 That's very interesting. 00:14:44.000 --> 00:14:50.559 Yeah, so these are the two things that are in at the core of uh delay tool invocation or what I consider as conditional prompts. 00:14:50.720 --> 00:14:56.960 There are a lot of things that I think that Johan wasn't even understood when uh he thought about it. 00:14:57.039 --> 00:15:11.919 I I'm not even sure that even today is completely understand what exactly is happening in uh not in the architectural level, it's very good to understand architecture, but in the neuron level, what exactly is happening in the attention layer which triggers the, you know, the how the triggers affect the the rest. 00:15:12.320 --> 00:15:13.919 So there are very interesting things. 00:15:14.000 --> 00:15:23.360 Uh, if you uh look on it, and uh I would argue that one of the things that I'm not sure that Johan understood because he called it delay tool invocation. 00:15:23.519 --> 00:15:26.159 Delay tool invocation is a form of conditional prompt. 00:15:26.240 --> 00:15:29.039 It's a prompt who has a condition to be executed. 00:15:29.200 --> 00:15:31.360 Think about it for a second on conditional prompt. 00:15:31.840 --> 00:15:38.399 Conditional prompt actually extends the functionality of prompt injection. 00:15:38.799 --> 00:15:39.360 Why? 00:15:39.600 --> 00:15:43.360 Because first of all, you can wait for a dedicated event, right? 00:15:43.519 --> 00:15:46.240 Which at the beginning you may not have it. 00:15:46.399 --> 00:15:55.600 For example, you can wait until the user asks for sensitive uh data to be brought into, unlike the case that you just immediately ask to exfiltrate stuff. 00:15:55.679 --> 00:15:56.320 Yeah, okay. 00:15:56.559 --> 00:16:00.559 Second thing, you can spear the attack, you can make it targeted. 00:16:00.639 --> 00:16:04.960 Okay, you can decide on whether this is the target that you want to uh to attack. 00:16:06.399 --> 00:16:14.159 Uh do it only if the attacker is the CEO and this is the application and this is the company, right? 00:16:14.240 --> 00:16:22.480 Yeah, so it helps you to be much less noisy with respect to uh it's out of distribution, which is think about it for a second. 00:16:22.559 --> 00:16:30.399 We just described the new method to bypass LLM firewalls uh or prompt injection firewalls with much higher uh bypass rate. 00:16:30.559 --> 00:16:39.840 Think about all the things that you can now do, event-driven targeted attacks, specific tools in time that you want uh to uh to wait for them. 00:16:40.240 --> 00:16:43.440 For the targeted attack, I think I remember one Johan blog. 00:16:43.519 --> 00:16:50.639 Uh uh it's funny, that that uh addresses this, like that talks about this, but it's I think uh Johan is like the Bible here, right? 00:16:50.799 --> 00:17:02.960 Everywhere when you when you have you have something, you you think you've kind of found something new, then you go out, you scroll, you scroll through

Johan's blog, and he's like, Oh yeah, like three years ago, like he's amazing. 00:17:03.120 --> 00:17:13.839 But I I I get what you mean, it's because uh most attacks on LLMs are not really persistent, and even if they are persistent, they only take into consideration what's currently in the context. 00:17:14.079 --> 00:17:22.880 And essentially you're saying, Hey, now you can you can compromise the future, and it doesn't really mean it doesn't really matter if you persist or not, you are there in the context. 00:17:23.119 --> 00:17:35.359 And and yeah, that's uh Which again this extends your ability as an attacker to uh first of all perform silent attacks or to spill to have the attack tailored to either a specific target or specific event in time. 00:17:35.680 --> 00:17:41.680 Like just uh detonate only on me, not on you, or detonate on one specific target or one specific event. 00:17:42.000 --> 00:17:43.920 Makes things much harder to detect in reality. 00:17:44.240 --> 00:17:51.359 Google uh DeepMind published their uh Camel paper, what is it, uh two years ago, a year and a half ago, something like that. 00:17:51.599 --> 00:17:51.839 Yes, yes. 00:17:52.000 --> 00:17:56.799 A year, I don't know, my time timing with AI is uh for me, it's changed drastically. 00:17:56.960 --> 00:18:00.079 Uh, like one question is why haven't they deployed it here with Gemini? 00:18:00.240 --> 00:18:01.519 But also why is this not a reality? 00:18:01.759 --> 00:18:02.720 Good, okay, good. 00:18:02.960 --> 00:18:06.880 First of all, there are a lot of good answers to this question. 00:18:07.119 --> 00:18:10.640 I will start with the most primary, in my opinion, among them. 00:18:10.799 --> 00:18:17.200 After experiencing with Camel, Camel, what you consider as uh academic kind of uh research, okay? 00:18:17.519 --> 00:18:22.480 Camel is good up to some degree to a single inference. 00:18:22.640 --> 00:18:27.279 When you are in a multi-inference kind of uh scenario, Camel doesn't make sense to be used. 00:18:27.359 --> 00:18:38.720 Uh, Camel is by definition intended to secure against one single inference that obtained data or untrusted data from the external world. 00:18:38.880 --> 00:18:48.559 It doesn't allow the developers to count on the data brought from the external world for the next inference for planning or some other stuff. 00:18:48.720 --> 00:19:03.200 Meaning, in other words, it doesn't good for multi-inference or for uh chatbots or for any kind of uh situations where you have this multi-inference kind of uh scenario between users and the application, okay. 00:19:03.279 --> 00:19:11.039 When users continuously feed the application with new inputs, and some of them may rely on the answers of the LLM. 00:19:11.680 --> 00:19:14.799 Camel doesn't go well, okay. 00:19:15.039 --> 00:19:20.799 And moreover, Camel requires significant, significant uh adjustments to the code. 00:19:20.960 --> 00:19:24.720 They are not easy to deploy, very not easy to deploy, okay? 00:19:24.960 --> 00:19:40.480 And in many cases, they uh very rely heavily on the user to confirm and confirm and confirm for each and every data that is being uh digested into the planning and some other stuff, which is uh bad usability if you think about it. 00:19:40.640 --> 00:19:51.279 So, this is one reason it's too academic, it's not well adjusted to multi-inference, and uh there's significant overhead in implementing stuff within the code. 00:19:51.599 --> 00:20:02.000 Second thing is that the guys that wrote the papers on Camel are Google researchers, okay, that research the area of AI security. 00:20:02.240 --> 00:20:09.680 The guys who implement uh the uh security against uh Gemini in reality are engineers. 00:20:10.000 --> 00:20:13.680 Google is a very, very big organization, okay. 00:20:13.920 --> 00:20:24.799 I can tell you, based on my experience, in many cases, not even a word of the work that has been done in different departments within Google, even if they somehow intersect. 00:20:25.039 --> 00:20:29.279 I think this is something that should be asked, should be referred to a Google engineer. 00:20:29.440 --> 00:20:44.559 I believe that the Google engineer back then were not, and even today, I'm I'm not even sure that they are completely understand uh they are familiar with Camel, and mostly because Camel is not the practical I think um like we we haven't seen this implemented basically ever anywhere. 00:20:44.720 --> 00:20:49.599 And uh, if you even if you take Camel, like you try to bring in Camel to the agent space. 00:20:49.759 --> 00:20:59.039 So basically, my my rudimentary uh how would that look like would be hey, you got one LLM that doesn't read any of the untrusted data. 00:20:59.200 --> 00:21:07.920 It basically instead of doing the React loop, it writes code, and that code still doesn't see any of the untrusted data. 00:21:08.079 --> 00:21:15.200 And then the secondary, what they in Camel they call like the unprivileged or the sandbox LLM. 00:21:15.359 --> 00:21:23.279 You don't leave use like it populates the parameters for each function or something like that on the fly when it reads the untrusted data. 00:21:23.519 --> 00:21:29.839 That is kind of my how do you take it to uh something that is that is like a bigger but nobody's using it. 00:21:30.000 --> 00:21:32.559 Okay, here is a subtle thing that you need to consider, okay? 00:21:32.720 --> 00:21:40.400 There are two subtle things in here that in one case you can have the LLM, uh, you can ask the LM to provide you with the recommendation, okay? 00:21:40.559 --> 00:21:47.680 And later on, to have the LLM, based on the recommendation that was brought from the internet, plan something with respect to this recommendation. 00:21:47.839 --> 00:21:56.480 Camel doesn't know how to handle it without the user extensively uh being a prompt that uh you are about to uh to to have a disaster situation. 00:21:56.720 --> 00:21:57.759 So this is one thing. 00:21:57.920 -->

00:21:59.759 Bear in mind, Camel to implement. 00:22:00.319 --> 00:22:09.920 It and to validate whether the data is uh within the ranges, okay, is not considered anomaly, it requires significant effort. 00:22:10.160 --> 00:22:16.400 Okay, you need to have the validators that you've just mentioned within the code, the script that is uh being generated. 00:22:17.200 --> 00:22:28.400 Test whether the input or the argument is reasonable comparing to reasonable arguments that uh um parameters that were are considered allowed. 00:22:28.799 --> 00:22:34.400 Okay, this is not easy, it requires a significant effort from the development effort. 00:22:34.720 --> 00:22:36.160 It's like a configuration, a huge configuration. 00:22:36.559 --> 00:22:37.440 Exactly, it's not easy. 00:22:37.599 --> 00:22:38.559 Not easy. 00:22:38.799 --> 00:22:50.960 Doesn't go well when you want to go to multi-inference, may require significant maintenance for each and every changes within the logic of the application and some other stuff. 00:22:51.200 --> 00:22:51.839 Not easy. 00:22:52.160 --> 00:22:58.240 What are your thoughts on the on Microsoft's uh information control flow or information flow control, I think they call it. 00:22:58.319 --> 00:23:04.240 Uh like essentially their idea is tracking taints across throughout the context of an agent. 00:23:04.400 --> 00:23:13.759 So they say, hey, there is uh again, my reading of it, they're saying, hey, you need to track the integrity of the data and the confidentiality of the data. 00:23:13.920 --> 00:23:20.400 Uh and with integrity, my read of it is, hey, let's fight, let's figure out where untrusted data comes into context. 00:23:20.559 --> 00:23:23.759 And confidentiality is any any kind of confidential data. 00:23:24.000 --> 00:23:36.880 And so if everything that the agent does, now agent now LLM uh gets a label, then you can know when there is a kind of a mismatch or a combination of labels that shouldn't be the case. 00:23:37.039 --> 00:23:47.359 So if you're touching sensitive data and you also have uh any untested data in the context that could abuse the agent to uh to abuse the sensitive data. 00:23:47.680 --> 00:23:50.960 So I will start by mentioning again, I haven't read the paper. 00:23:51.039 --> 00:24:02.799 If you ask me as somebody who works with machine learning for too long, I would argue that uh false positives are at the end of the day something that could uh break your entire uh thing into uh you know pieces, making it unpractical. 00:24:03.039 --> 00:24:04.400 At the end of the day, I do not know. 00:24:04.559 --> 00:24:07.200 I would like to see real results on real systems. 00:24:07.359 --> 00:24:12.000 At the end of the day, if you have something which is really working, you will see it immediately being deployed. 00:24:12.160 --> 00:24:25.200 Uh, since we currently lack the ability to mitigate this stuff, we what you see each and every week is additional mitigation that claims that they are the best

and they are and nothing is the best, right? 00:24:25.599 --> 00:24:35.359 When you see clients deploy mitigation and say, look, this is the mitigate, this really blocks, this handles the threat, this decreases the risk significantly. 00:24:35.519 --> 00:24:44.799 Uh only then you will probably know that uh I haven't seen so far any kind of decent mitigation, which I will be able to argue that I believe in. 00:24:45.279 --> 00:24:47.680 What are your thoughts on uh Simon Willison's Little Trifecta? 00:24:47.920 --> 00:24:53.599 He was not only right, he was uh the first who introduced uh the three factors, right? 00:24:53.839 --> 00:25:00.240 I would argue that today you need to extend it beyond exfiltration to actuation as well. 00:25:00.400 --> 00:25:07.839 Whether it is exfiltrating data was something that was maybe two years ago was the you know the chatbot era. 00:25:08.240 --> 00:25:08.640 Yes. 00:25:08.880 --> 00:25:24.400 But today, when you have these things capable of running stuff locally in terminal or capable of opening your windows inside your home, or I don't know, like you know, open your zoom, maybe Zoom you can consider it a bit of an exfiltration, right? 00:25:24.640 --> 00:25:46.880 But to do stuff they shouldn't do due to the permission, the the last factor which was communicated outside, I would argue that needs to be somehow better uh adjust to this era, which is an actuation which may have sensitive or some kind of uh may consider a threat to the user. 00:25:47.119 --> 00:25:51.920 But even the data exfiltration mitigation has not really been deployed anywhere. 00:25:52.160 --> 00:26:02.799 Like uh I I can't imagine I can think of even one example when where people are baking and lethal trifecta as a mitigation into a flagship AI product, even just for the data exfiltration part. 00:26:03.119 --> 00:26:08.559 Because you need to basically remove one of the factors so it affects your usability. 00:26:08.799 --> 00:26:15.759 And at the end of the day, you end with security that prevents you from doing stuff rather than allowing you to do stuff, right? 00:26:15.839 --> 00:26:20.640 Which is not the best place you want to be on the trade-off between security and usability. 00:26:20.720 --> 00:26:25.279 You need to find a way to allow uh usability with uh decent security. 00:26:25.359 --> 00:26:27.519 This is at least my perspective about security. 00:26:27.759 --> 00:26:46.720 I would argue that Camel also took the the wrong, they weren't like on the worst case of they provide good security, worst usability to the and again you you need to find things that have much better balance in terms of security and in terms of uh usability, which I haven't seen so far. 00:26:46.960 --> 00:26:56.000 In your prompt were uh kill chain paper, you mentioned a couple of uh other mitigations that I thought were interesting, but again, I haven't seen them really deployed at scale. 00:26:56.160 --> 00:27:05.200 One was canaries, so hey, let's have uh like canary tools that when somebody when the agent invokes them, you know that the agent has been compromised. 00:27:05.359 --> 00:27:09.839 And the other was prompt perturbation, which I found was really cool. 00:27:10.079 --> 00:27:20.880 So, my my understanding of that was basically um every time you're gonna send a prompt to the agent, we're gonna add some salting mechanism into the LM core. 00:27:21.039 --> 00:27:34.319 So now no single it basically uh uh prevents me from creating a consistent exactly to overfit your attack against a specific prompt that will collide in the embedding space into something which is a disaster. 00:27:34.640 --> 00:27:35.119 Right. 00:27:35.279 --> 00:27:40.480 Uh again, soft mitigations are very easy to uh to bypass, right? 00:27:40.640 --> 00:27:44.640 Yeah, I haven't seen very strong mitigation so far. 00:27:44.960 --> 00:27:54.720 If you think about it, in computer security in architecture, we have different spaces for or addresses for code and for data. 00:27:54.880 --> 00:28:02.240 Okay, the way that large language models today work, they don't have any kind of separation between code and data. 00:28:02.400 --> 00:28:13.759 And I would argue that it's not even easy to have this separation between code and data, because in many cases you want to use the data to create the new code, and then you have a mismatch between what could happen. 00:28:14.079 --> 00:28:23.759 I also think in that like uh that that uh argument on uh of kind of code, uh codes versus instructions in LLMs, I feel it's uh kind of missing the point. 00:28:24.000 --> 00:28:32.319 Because if let's say magically you get a magic wand, and tomorrow we got an LLM that has different channels for instructions and data. 00:28:32.559 --> 00:28:44.960 If I manipulate that data, I can still not the instructions, I can still get the LLM to do whatever I want because the data is what the LLM is judging to make its calls, right? 00:28:45.279 --> 00:28:52.559 Okay, you can plan the plan only based on the code and then incorporate the data into it. 00:28:52.720 --> 00:28:58.240 So the data will be parameters, it will not change end-to-end the execution. 00:28:58.559 --> 00:29:01.119 It will change which path of the execution you're gonna take. 00:29:01.519 --> 00:29:22.079 Uh it it could change one thing, which is the parameters, which I would argue is a bad thing because if you have, like, you know, if I instructed the LLM to send an email to John and the data instructed to send it to the attacker, okay, so same path, but uh you go from desired outcome to a threat of data exfiltration, right? 00:29:22.559 --> 00:29:28.000 So you could up to some degree, and this takes me back to Camel to discuss about Camel. 00:29:28.559 --> 00:29:44.000 One of the things that Camel were uh I think the first to recognize is the fact that even if you just plan things based on the instruction, this is not enough because parameters could be changed via the data, via

instructions embedded into the data. 00:29:44.240 --> 00:29:54.319 Okay, and when you have it in this specific case, you understand that going through address spaces will not be that good of mitigation, as you mentioned. 00:29:54.480 --> 00:30:05.039 And moreover, not using parameters taking from the data is something that significantly uh reduced the usability of a system. 00:30:05.279 --> 00:30:06.720 Yeah, so it's not it's just not gonna be a good idea. 00:30:06.960 --> 00:30:07.839 It's not a system, yeah. 00:30:08.240 --> 00:30:13.200 So I want to kind of um I want to switch off to a different topic. 00:30:13.359 --> 00:30:15.519 Or I think it's it's close by. 00:30:15.759 --> 00:30:37.519 When we so last, I think this was October, you gave uh you gave a uh a really good talk on the on our agent security summits, uh, and you mentioned the evolution of promptwell, and that led up to the zero click attacks we saw last year, uh your Gemini attack, we did our ChatGPT attacks, Johan did a bunch of different attacks. 00:30:37.680 --> 00:30:41.039 Um, and that was the first what we called the zero click. 00:30:41.440 --> 00:30:46.640 We called it a zero click because there was no nothing bad the user had to do. 00:30:46.799 --> 00:30:55.119 The user was just going about their day, the attacker would send an email or share a document or whatever it is, and it would be delayed. 00:30:55.279 --> 00:30:58.559 So they will not be uh at control when it would trigger. 00:30:58.720 --> 00:31:01.839 But eventually, when a user just goes about their day, it will trigger. 00:31:02.000 --> 00:31:07.759 And we had a discussion about that, and you told me, listen, honestly, this is not really a zero click in my mind, it's a half click. 00:31:07.839 --> 00:31:09.119 So can you can you expand on that? 00:31:09.359 --> 00:31:09.759 Yeah, yeah. 00:31:09.839 --> 00:31:29.200 So if you will uh okay, I I think I did some uh I abused the term zero click, so the entire industry now used the term zero click within the context of Pontwell, yeah, uh, because I use this term in the wall, the AI one that we presented, and ever since many people use the exact same term, which is zero click. 00:31:29.279 --> 00:31:34.160 Okay, a true zero click is what Pegasus has done. 00:31:34.400 --> 00:31:40.960 A true zero click is something which is initiated by the operating system. 00:31:41.359 --> 00:31:50.720 At the end of the day, the entire uh up to up to today, okay, the entire things that we've seen are initiated by user. 00:31:51.279 --> 00:31:53.759 There is a significant difference in here, okay. 00:31:54.000 --> 00:32:10.319 Even an invitation is all you need, which you can ensure that the compromised uh data, okay, or the compromised invitation will get into the user without the user having or prompting the assistant for his recent events, the prompt injection will not occur. 00:32:10.640 --> 00:32:16.319 But you can set up the front injection in a way where it doesn't address like it's it's not really addressing one prompt, right? 00:32:16.400 --> 00:32:17.359 It's addressing a topic. 00:32:17.440 --> 00:32:19.039 Like you can you can make it generalize. 00:32:19.359 --> 00:32:25.039 But without being without having the user initiate the right prompt, it will not be executed. 00:32:25.279 --> 00:32:25.599 Yes. 00:32:25.759 --> 00:32:27.759 Okay, so this is why I consider it a half-click. 00:32:27.920 --> 00:32:28.160 Okay. 00:32:28.559 --> 00:32:33.519 You would argue that up to some degree those actions are frequently happened by users. 00:32:33.759 --> 00:32:37.599 For example, asking for their events, asking for the recent emails and some other stuff. 00:32:37.759 --> 00:32:48.000 But at the end of the day, without the user having prompt the LLM application for his recent emails, for his recent events, this will not be executed. 00:32:48.160 --> 00:32:51.359 Uh, a pure zero-click attack is Pegasus. 00:32:51.680 --> 00:32:58.960 Pegasus is executed due to the um execution being triggered by the operating system. 00:32:59.119 --> 00:33:02.960 It is being guaranteed to operate when it is being deployed. 00:33:03.119 --> 00:33:12.319 Okay, this is by the way they call it a Pegasus, it's uh a Trojan with uh with the wings that uh when you send it, it's uh being uh deployed. 00:33:12.559 --> 00:33:15.519 So you're saying the the thing is as the guarantee. 00:33:15.680 --> 00:33:26.720 Because I I I see your point that uh if a user doesn't do anything, if a user doesn't use this Gemini or ChatGPT or whatever it is, then they won't get compromised. 00:33:26.880 --> 00:33:30.880 However, I think it's important, it was important for me to make that distinction. 00:33:30.960 --> 00:33:38.000 Like we use zero click because we basically said listen, there is no way for you to defend yourself against this because you don't know. 00:33:38.079 --> 00:33:45.200 As a user, you just ask, in our case, it was uh anything about a meeting, like you just ask anything about a meeting and you get popped. 00:33:45.359 --> 00:33:49.920 And you didn't ask about a specific meeting, I didn't need to know in advance what were your meetings. 00:33:50.240 --> 00:33:53.039 I agree, but but here is my thought, okay? 00:33:53.359 --> 00:34:02.640 In about a few months or a year or so, inference will be triggered into your it is already being happened if you have uh iPhone, right? 00:34:02.720 --> 00:34:05.759 But it's currently happened without any tools to compromise. 00:34:06.000 --> 00:34:13.280 So you already have the mechanism that it will happen without any user uh initiation, guaranteed to happen. 00:34:13.840 --> 00:34:21.599 Guaranteed to happen just by having it uh within the just by you sending it, pushed it into the victim, right? 00:34:22.000 --> 00:34:23.519 And this is about to happen. 00:34:23.760 --> 00:34:26.880 You understand and agree that there is a great difference between the two. 00:34:27.039 --> 00:34:34.559 One thing

is guaranteed to happen, one thing is conditioning in a user activity to be prompt, and then it is being uh executed. 00:34:34.719 --> 00:34:45.599 And again, I've in one of our discussions, and especially in this talk, I've mentioned that automatic inference initiated by operating system is either now being deployed or about to be deployed. 00:34:45.760 --> 00:34:50.960 Okay, iPhone uses automatic summary for the notifications. 00:34:51.199 --> 00:35:02.800 Okay, if you think about it, you have a pure zero-click path, whether it will likely be do something malicious is a function of whether there are tools to exploit or not. 00:35:03.039 --> 00:35:11.599 So you're saying if if somebody on the uh summarization team on the Apple side is going to be, hey, like why not hook up a bunch of tools to this uh summary? 00:35:12.079 --> 00:35:13.440 Most likely it will happen, right? 00:35:13.519 --> 00:35:17.119 Because I don't know why it will make sense up to some degree someday. 00:35:17.199 --> 00:35:17.440 Okay. 00:35:18.000 --> 00:35:32.960 Um there is an additional case where, if you think about it, is the autonomous uh robots, for example, that will likely to use a multimodal LLM to digest everything they see. 00:35:33.039 --> 00:35:45.199 And here you have a path for a completely zero click, which you don't need to do anything special automatically, an inference will be triggered in response to, you know, it just happened. 00:35:45.280 --> 00:35:45.440 Okay. 00:35:45.599 --> 00:35:47.519 How do you view uh like a background agents? 00:35:47.599 --> 00:35:58.400 Because with background agents today, agents that don't have a human in the loop, you do already have this path where you can get an injection in, the agent will be invoked, the agent will get compromised. 00:35:58.559 --> 00:36:04.239 Uh, I have uh like one example I have in mind is the uh the Klein NPM compromise. 00:36:04.320 --> 00:36:08.800 So there was this case where Klein was for a few uh versions. 00:36:08.960 --> 00:36:12.320 If you installed Klein, it would install OpenClone on your machine. 00:36:12.559 --> 00:36:16.079 And it was not really a compromise, it was more like I think for the lulls. 00:36:16.400 --> 00:36:21.679 Uh I actually played around with trying to figure out what happened there and the attack path. 00:36:21.760 --> 00:36:27.679 I mean, it was somebody that caught uh Adnan Khan's research uh and then replayed it. 00:36:27.920 --> 00:36:33.039 Um, and basically it was prompt injections through an issue, GitHub issue. 00:36:33.119 --> 00:36:38.239 You open an issue, the ticket has the title of the issue has a prompt injection in it. 00:36:38.400 --> 00:36:45.119 There was uh a cloud that was kind of looking at issues, triaging issues, and then you can compromise that cloud. 00:36:45.440 --> 00:36:58.559 I think that the difference between half-click and zero click is whether the inference happened automatically, or whether you consider it it's not it's a manual

inference, but it's something which you consider as frequent. 00:36:59.039 --> 00:37:10.880 Okay, this is, in my opinion, in my view, the main difference between whether it's an operating system-initiated inference or a frequent user-initiated uh uh inference. 00:37:11.119 --> 00:37:15.039 This is the I think the best way to distinguish between the the two. 00:37:15.280 --> 00:37:18.159 You mentioned agents running in the background. 00:37:18.320 --> 00:37:29.599 Agents running in the background are what you consider vulnerable to zero-click attacks, given that they have, like, you know, the ability to uh obtain uh sensitive data and to execute stuff using the tools that they have. 00:37:29.760 --> 00:37:33.280 Definitely, this is something that should be taken into consideration within this context. 00:37:33.360 --> 00:37:53.039 It's it's a real threat, I would argue, where in many cases you can even optimize your success rate as an attacker to make sure that you were able to compromise or you were able to get into the agent running in the background, even if it doesn't have a specific uh identity that you can associate with. 00:37:53.280 --> 00:37:56.800 Because you're you're just you're just after that resource or you're after that organization. 00:37:57.039 --> 00:37:58.000 Exactly, exactly. 00:37:58.320 --> 00:38:00.800 So I wanna switch off to another topic. 00:38:00.960 --> 00:38:08.079 You've been on the Black Hat review board uh for what, like for many years now, like I think 2020, 2022, something like that. 00:38:08.239 --> 00:38:09.119 Something like this, yes. 00:38:09.440 --> 00:38:23.840 And I mean AI has obviously been a major part of we've seen them we've seen incredible uh talks coming out of Black Hats that are that have AI in them or that are use AI, either hacking AI or using AI for security. 00:38:24.000 --> 00:38:38.079 Can you take me through what's your so if you if you look at the progress of uh the of security and AI since 2022, when did you see jumps in that progress? 00:38:38.159 --> 00:38:39.280 I'm sure there were progress. 00:38:39.519 --> 00:38:47.360 Okay, so uh I think around 2022 or 2021, this was the first placket to introduce the AI track. 00:38:47.519 --> 00:39:05.920 Okay, back then the AI track was uh like AI kind of uh was baked into various tracks, for example, such as cryptocurrencies today, they don't have their special their special track, they're just you can find them within network or maybe within appsec or maybe within something else. 00:39:06.159 --> 00:39:14.559 So the AI track started before I came into the I joined the the board members of uh Blackit. 00:39:14.880 --> 00:39:22.159 But ever since, I think in the first year or so, started to become interesting. 00:39:22.320 --> 00:39:29.840 It was something that started very with uh, I don't argue few or too few uh kind of uh submission. 00:39:30.159 --> 00:39:37.519 It it did have it was I think the fifth or the sixth or the seventh biggest track in terms of its submission. 00:39:38.000 --> 00:39:44.719 When I I think two years ago it became among the top three, and today this is the biggest one among them. 00:39:44.800 --> 00:39:56.239 So it's actually takes you a lot of time to uh go through the submissions in this track, which is not easy, it's uh by far the biggest uh load that you have comparing the other tracks. 00:39:56.559 --> 00:39:59.920 I would argue that we can see two kinds of topics within this track, okay? 00:40:00.079 --> 00:40:04.880 And as of next blackhead, there will be two separate tracks uh for them. 00:40:05.119 --> 00:40:18.800 The first topic is what you considered as attacks against LLM applications, and the second kind of talks are either AI for security or AI for attacks. 00:40:19.119 --> 00:40:23.440 The AI for security started around it started well before. 00:40:23.519 --> 00:40:24.800 It wasn't something very new. 00:40:24.960 --> 00:40:28.639 They just didn't use LLM, they just used neural networks to do some other stuff. 00:40:28.719 --> 00:40:33.360 Maybe they use like the traditional machine learning, such as I don't know, random faults and some other stuff. 00:40:33.679 --> 00:40:56.079 Around the time that LLMs were started to be integrated into applications, and when they were able to, when people really demonstrated they are make sense in like you know, improving your results for security in various use cases, we started to see a real increase, making it the biggest one at Blackhead. 00:40:56.239 --> 00:41:09.119 And today, again, this is I think that the last time that I checked, the AI track was twice in terms of the number of submissions than the second uh track. 00:41:09.280 --> 00:41:27.280 And if you think about it, Black It or the traditional core of the topics of Black Hat is not AI, it's things considered as memory uh corruptions, things considered as mobile, which started about 20-15 years ago, something like this, cloud security. 00:41:27.519 --> 00:41:29.519 These things were almost diminished. 00:41:29.760 --> 00:41:41.039 Uh, we we now receive only a handful of uh talks among some of these uh tracks where we have uh more than 100 uh talks in the AI track. 00:41:41.199 --> 00:41:42.079 I find it interesting. 00:41:42.239 --> 00:41:50.960 Also, if you incorporate the fact that uh you have AI within the AppSec or within some additional stuff, you see a lot of talks in this domain. 00:41:51.440 --> 00:41:52.719 How does a good submission look like? 00:41:52.880 --> 00:41:54.400 Okay, this is a good question. 00:41:54.719 --> 00:42:07.920 I constantly write posts on how to submit to Blackett because I see that people do not know how to submit to Blackhead, even if they do a good job in the research, they don't know in many cases how to submit. 00:42:08.239 --> 00:42:22.159 So bear in mind that as a Blackhead reviewer, you cannot spend when you have like 200 submissions uh waiting for you to uh to judge them, you cannot spend an hour

to read the entire submission. 00:42:22.400 --> 00:42:40.800 Okay, you need to be very accurate in what you want to show the reviewer and do it from the very beginning and to make sure that you have the needed, I would argue, evidence that you are able to do this. 00:42:41.039 --> 00:42:44.880 It starts with an interesting title that you want to read, okay? 00:42:45.039 --> 00:42:46.639 Something that you haven't seen before. 00:42:47.440 --> 00:43:00.960 It goes with an abstract that uh discusses the talk and the findings and shows you that this is something which is applicable to various systems, not only one. 00:43:01.119 --> 00:43:14.159 Okay, it must be novel because you're trying to convince uh the biggest, or I would argue, the the biggest experts in their field that what you did is uh something they haven't seen before. 00:43:14.639 --> 00:43:16.400 Okay, so it must be novel. 00:43:16.800 --> 00:43:20.400 It should, in my opinion, come with uh the needed validations. 00:43:20.559 --> 00:43:31.679 The needed validations uh could be, for example, videos, they could be, for example, CVSS scores, they could be, for example, the manufacturer response. 00:43:32.000 --> 00:43:44.880 I take a great effort to understand what the manufacturer or what the vendor has done in response to the this finding to understand whether they consider it a real issue or whether this is something which only the researcher thinks it's a real issue. 00:43:45.039 --> 00:43:50.639 And there are good ways to show that this is a something that should be taken into consideration. 00:43:50.800 --> 00:43:56.320 Now, beyond, I'm looking for something which is uh, you would argue have the needed details to understand. 00:43:56.480 --> 00:43:59.760 I try to envision what I will see throughout the talk. 00:44:00.159 --> 00:44:03.920 Okay, in many cases the agenda is not clearly written. 00:44:04.400 --> 00:44:12.079 Or you you don't understand what exactly is about to be presented, or what is exactly you're about to see. 00:44:12.239 --> 00:44:14.960 I also take the need of time to read the takeaways. 00:44:15.199 --> 00:44:19.199 Are there real takeaways in here, or is it just things that we already know? 00:44:19.440 --> 00:44:25.840 For example, um, the bar for prompt injection has significantly changed. 00:44:26.000 --> 00:44:37.760 If the main takeaway of your talk is we found that this application is also vulnerable to prompt injection, so you just one out of many others that we just already know. 00:44:37.840 --> 00:44:42.960 I that I assume that every kind of application is currently uh vulnerable to prompt injection. 00:44:43.039 --> 00:44:44.480 This is not something which I consider. 00:44:44.960 --> 00:45:01.280 If you have real thing in here which would argue is maybe universal across this class of uh application, if you use a technique or argue something we haven't seen before, you have something new to argue. 00:45:01.360 --> 00:45:03.199 Okay, look, you've seen

this now. 00:45:03.280 --> 00:45:09.280 I show you scalability, you've seen this, now I show you on-device lateral movement, you've seen this, and I'll show you a new property. 00:45:09.360 --> 00:45:11.039 You haven't, this is what I look for. 00:45:11.199 --> 00:45:14.159 These are the things that, in my opinion, make a great blank talk. 00:45:14.239 --> 00:45:18.960 And then again, I would argue that it's not easy to get accept into this specific track. 00:45:19.119 --> 00:45:25.599 Uh, this track underrepresents the number of submissions we have. 00:45:25.679 --> 00:45:34.719 So just by selecting this track, the AI track, you are about to be to handle uh much a much higher bar in terms of the acceptance rate. 00:45:34.800 --> 00:45:46.559 You will get lower acceptance rate just by selecting the AI track, where you can maybe optimize your uh acception uh rate with uh selecting different uh tracks. 00:45:46.880 --> 00:46:02.079 In many cases, we are unable to really understand what the speaker tried to communicate because uh maybe didn't do such a great job in communicating it to us. 00:46:02.320 --> 00:46:06.800 In some cases, other factors also taken into consideration whether it's exclusive or not. 00:46:07.039 --> 00:46:12.559 There are a lot of things being considered, not all of them are the pure quality of your findings. 00:46:12.639 --> 00:46:16.719 That these things are also come as a surprise to people when I speak with them. 00:46:16.960 --> 00:46:25.840 So if you ask me, the fact that you haven't been accepted to Blackhead doesn't necessarily mean anything about the novelty of your research. 00:46:26.000 --> 00:46:37.280 It may say a lot of things about the way that you present stuff, about your ability to convince in your findings, about whether it was exclusive or not, about a lot of things within this uh domain. 00:46:37.679 --> 00:46:44.239 And what do you think about so you've you've delivered multiple multiple talks, very successful, very like very good talks, highly rated. 00:46:44.800 --> 00:46:50.000 What is your advice for uh somebody who's presenting their talks for the first time? 00:46:50.159 --> 00:46:57.599 I think one of the it's almost like it's a my take is that it's it's almost like a separate piece of work. 00:46:58.000 --> 00:47:09.199 One piece of work is your research, and that is like you can you can do it, you can like you put your hoodie on, you go to your cocoon, you go out in six months, you have something cool. 00:47:09.440 --> 00:47:12.719 But then telling the world about it is completely different. 00:47:12.960 --> 00:47:13.519 Yeah, I agree. 00:47:13.679 --> 00:47:14.079 I agree. 00:47:14.159 --> 00:47:14.400 Okay. 00:47:14.639 --> 00:47:40.159 There are, I would argue that I've seen many kinds of talks, and there is not like you know, clear thing, what is uh he did something completely different than another talk that I did, and I still thought that this is an amazing talk, and delivered uh, for example, I've seen a talk in the recent Black At Europe delivered by a speaker whose face was covered with a mask, and this was by far the best talk I've seen at the Blackhead Europe last year. 00:47:40.320 --> 00:47:41.519 This is the trailer of B talk. 00:47:41.679 --> 00:47:48.960 Yes, one of the greatest talks that's uh Black, the recent Black Hat Europe, it was on uh December 25th, uh about a year ago. 00:47:49.119 --> 00:48:02.400 I would argue this if you will analyze what we consider as good Blackhead talk, usually they have video demonstrations, okay, without the audience value good video demonstrations. 00:48:02.480 --> 00:48:15.440 They usually they they apply, they they applaud to it, they uh they they laugh out of it, they they they love to see this kind of uh systems being hacked, okay, and they love to see it visually, not just like you know. 00:48:15.679 --> 00:48:19.840 I think that they highly value technical talks. 00:48:19.920 --> 00:48:26.079 And and bear in mind, not just the audience, we as board members also value technical talks. 00:48:26.159 --> 00:48:31.599 I don't like talks given from 50,000 feet away. 00:48:31.920 --> 00:48:38.000 I tr I try to understand and to see, like, you know, the code or the exploitation. 00:48:38.239 --> 00:48:46.960 Additional good talk should convince that this is not a, I would argue, a poor security implementation. 00:48:47.199 --> 00:48:48.960 This is something very fundamental. 00:48:49.119 --> 00:48:58.800 Because if you see something very fundamental in front of your eyes, then all of a sudden, whether the speaker is good or not is not even makes a difference because you will discuss about it with your colleagues. 00:48:58.880 --> 00:48:59.519 I've seen something. 00:48:59.679 --> 00:49:00.239 Did you see it? 00:49:00.320 --> 00:49:02.000 It applies to this, this, this, this, and this. 00:49:02.239 --> 00:49:03.679 This is something that we must take care of. 00:49:03.840 --> 00:49:04.880 And you will repeat seeing it. 00:49:05.039 --> 00:49:13.039 I would argue that the best talk that I've seen at Blackhead was Charlie's and Chris, Chris's talk on hacking the Gypsy Rocky. 00:49:13.280 --> 00:49:22.559 I keep seeing this uh talk once a year, just to see how they were able to communicate their findings with the audience at Blackhead at DEF CON. 00:49:22.880 --> 00:49:37.599 You can see how the audience really loves the way that they did it together, which is something even more risky and more dangerous at uh Blackhead, presented as a peer, not uh each one of them took a different uh part of the talk. 00:49:37.920 --> 00:49:39.679 So I think I'm gonna be. 00:49:48.239 --> 00:49:49.599 Yeah, definitely great talk. 00:49:49.760 --> 00:50:07.920 The thing is, since I'm working for the last seven years a lot with the car industry, I started to see this talk around I think 2016 before I understood about Canvas security and about connected car security and some other stuff related to

perception security and some other stuff. 00:50:08.079 --> 00:50:14.800 And every time I've seen it for the last, I think, eight years or so, something like this, I learned something new from their talk. 00:50:15.119 --> 00:50:20.960 Because I became more expert in this field and I then see oh, they did this and they did this and they did this and they did this. 00:50:21.119 --> 00:50:30.480 You have a lot to learn from that from their talk on how to do excellent research, on how to communicate the excellent research to the uh audience. 00:50:30.800 --> 00:50:34.639 The best by far, DEF CON and Blackest Talk that I ever seen by far. 00:50:34.960 --> 00:50:36.159 That's really, really cool. 00:50:36.400 --> 00:50:41.119 So um I want to end with um you have a conference coming up, right? 00:50:41.280 --> 00:50:43.519 The real world AI security conference. 00:50:43.599 --> 00:50:46.079 Uh and and you're part of the team behind it. 00:50:46.239 --> 00:50:49.519 Can you share a bit uh what is this conference about? 00:50:49.679 --> 00:50:54.400 Why why did what what is the need you saw uh for this kind of conference to take place? 00:50:54.639 --> 00:51:04.480 I think that today the specific conferences that you've just mentioned, among them Black Hat, RSA, and DEF CON, are first of all well too industrial. 00:51:04.639 --> 00:51:08.880 They don't attract the experts in the field of AI security. 00:51:09.599 --> 00:51:18.320 And uh in many cases, they are not even intended, some of them, not all of them, are not even intended for experts in this field. 00:51:18.559 --> 00:51:27.760 For example, Black Hat and RSA are mostly intended for executives, uh, whether they are you see a lot of, for example, product managers. 00:51:27.920 --> 00:51:29.599 What does a product manager have to do? 00:51:29.679 --> 00:51:37.119 Uh or people from sales coming for to this uh, and uh you you see CEOs and you definitely see CTOs as well, right? 00:51:37.360 --> 00:51:47.039 DEF CON is more about the core InfoSec hackers community, not exactly the AI security expert that you see in this field. 00:51:47.360 --> 00:51:58.960 We do see a lot of new other one-day or two-day summits, such as the one that you have at Zenity and such as the one that recently done by Gnostic by uh Gadi. 00:51:59.280 --> 00:52:04.239 I have one advantage is somebody worked in the area of crypto for a while. 00:52:04.639 --> 00:52:12.960 I went to a conference named Real World Crypto, which was initiated by Dan Bonnet, uh Dan Bonnet, professor from Stanford University. 00:52:13.360 --> 00:52:26.159 And when I went through the researcher, they told me, look, this is the best conference that if we need to sense, and again, you are speaking with people that have families and children, so they are unable to you know to attend each and every conference that they have. 00:52:26.320 --> 00:52:28.719 So they told me that this is the first priority. 00:52:28.960 --> 00:52:41.199 They start by when they plan their year ahead, they start by planning that they will attend to this uh conference and whatever else could uh they could attend also is a second priority to them. 00:52:41.440 --> 00:52:44.079 And they were able to create something very interesting. 00:52:44.239 --> 00:52:48.320 First of all, they went on only on technical talks. 00:52:48.400 --> 00:52:59.280 There is no talk which is not technical in their uh everything is completely new and gathered from the best conferences and by the greatest expert in this field. 00:52:59.519 --> 00:53:08.719 It attracts the greatest researchers and the greatest practitioners and the greatest uh you know experts in this field who work in the area of AI security. 00:53:08.880 --> 00:53:18.880 Since they were able to do it with uh single session, not parallel session, they were able to guarantee that everybody sees everything. 00:53:19.039 --> 00:53:24.480 Okay, otherwise, you have an audience that is being split between different sessions, right? 00:53:24.639 --> 00:53:30.079 If you have three different ballrooms, then the best case scenario, you need to decide what you're about to see. 00:53:31.280 --> 00:53:42.639 And uh not only because of this, because of the also the side events, they were able to enforce collaborations or discussions with the relevant expert in this field. 00:53:42.880 --> 00:53:45.360 And this is the exact experience that I wanted to bring. 00:53:45.519 --> 00:53:57.280 I didn't want to have it peer-reviewed in terms of making it uh because also like you know, um asking for exclusiveness, because otherwise I would not get the best things that I could uh ask. 00:53:57.440 --> 00:54:02.400 I wanted it to be something which is the best content that you can get. 00:54:02.559 --> 00:54:19.519 And as a result, we decided to not sell any spot for companies that wanted us to you know to pay for slots in the conference because they understood the value of presenting their work to the relevant audience of uh expertise. 00:54:19.760 --> 00:54:27.920 We wanted to be like, you know, those who not only academics, but also those came from the industry. 00:54:28.400 --> 00:54:36.320 Uh, become the magnet for them, become something that they will say, okay, if uh you have uh something uh practical, send it uh to uh to them. 00:54:36.480 --> 00:54:40.320 They will uh this is the place to show the things that are practical within this domain. 00:54:40.480 --> 00:54:59.039 Now the academic conferences weren't able to do it because the academic conferences are very all over the place, they'll have a lot of unrelated sessions to machine learning security and AI security, and you don't get a reasonable time to even present your stuff. 00:54:59.280 --> 00:55:04.800 Uh, in an academic conference, at the best case scenario, you will get 10 minutes, maybe 12 minutes. 00:55:05.039 --> 00:55:10.880 In other cases, you get five minutes, in other cases you don't get anything, you just have a poster that presents your stuff. 00:55:11.039 --> 00:55:21.840 So we found that this is a good way to bring together people from the uh universities and people from the industry to discuss together on urgent matters that are uh challenges for all of us. 00:55:22.079 --> 00:55:23.519 And we're looking forward to it. 00:55:23.760 --> 00:55:25.280 We will be happy to have you there. 00:55:25.679 --> 00:55:26.719 The agenda looks amazing. 00:55:26.960 --> 00:55:27.679 Thank you for that. 00:55:28.000 --> 00:55:30.239 Ben, this was great and a lot of fun. 00:55:30.559 --> 00:55:30.880 Thank you. 00:55:31.039 --> 00:55:31.840 Thank you for having me. 00:55:32.079 --> 00:55:32.719 Thank you so much.