From Dumbledore to Delayed Tool Invocation w/ Ben Nassi

WEBVTT
00:00:00.160 --> 00:00:09.599
We're able to trigger the opening of the windows, activating the boiler and deactivating the lights or turning the lights off within uh a user's apartment.
00:00:09.759 --> 00:00:11.359
And this was like this was real, right?
00:00:11.439 --> 00:00:13.359
This was Ori Air's uh apartment.
00:00:13.519 --> 00:00:15.119
Yes, it was Ori Air's apartment.
00:00:15.199 --> 00:00:17.519
This is the apartment that is being uh video streamed there.
00:00:17.679 --> 00:00:23.920
This was the first time that we demonstrated physical outcomes triggered by prompt injection.
00:00:27.280 --> 00:00:29.760
Initializing wild environment.
00:00:30.559 --> 00:00:32.799
Welcome to In the Wild.
00:00:37.520 --> 00:00:42.079
User authenticated, Michael Margaret.
00:00:47.759 --> 00:00:48.799
Hi everyone.
00:00:48.960 --> 00:00:53.039
I'm here uh with Ben Nasi again, uh picking up on our last conversation.
00:00:53.200 --> 00:00:54.240
Ben, welcome back.
00:00:54.399 --> 00:00:56.320
Thank you again for inviting me.
00:00:56.560 --> 00:01:03.200
So I want to start with uh what I think is the best style for a black hat talk I've ever seen.
00:01:03.359 --> 00:01:06.640
Uh uh this is yours, but let's let's watch this for a moment.
00:01:06.879 --> 00:01:09.040
Hello, black hat attendees.
00:01:09.280 --> 00:01:18.480
In today's lesson, you shall learn how to open windows, not those of your operating system, but real windows in the muggles world.
00:01:18.640 --> 00:01:29.920
We'll activate the boiler in a muggles apartment, remotely, of course, and stream their every move, all without a single ounce of consent.
00:01:30.079 --> 00:01:36.239
And as for a muggle's emails and meetings, oh yes, we will exfiltrate them with ease.
00:01:36.400 --> 00:01:38.719
All of this I mean, how long did it take you to generate?
00:01:39.680 --> 00:01:40.000
Too much.
00:01:40.319 --> 00:01:41.840
More than the prompt injections.
00:01:42.879 --> 00:01:44.000
More than the prompt injections.
00:01:44.239 --> 00:01:44.959
It was not ready.
00:01:45.280 --> 00:01:45.840
Like, right?
00:01:45.920 --> 00:01:49.040
This was just the this was the beginning of these kind of videos, right?
00:01:49.280 --> 00:01:51.840
Yeah, again, it's too much, too much.
00:01:52.159 --> 00:02:04.400
Took me some time to find the platform that I could use to uh to do this kind of stuff, and the platform to duplicate to deep fake Dumbledore's uh voice, and later on the text part was somehow easy.
00:02:04.480 --> 00:02:07.040
It was actually being written by an LLM.
00:02:07.120 --> 00:02:07.519
Oh wow.
00:02:07.680 --> 00:02:11.759
Uh I told him, look, uh, this is the and write it to describe it as Dumbledore would.
00:02:12.159 --> 00:02:19.759
And uh, but it it took a lot of effort, you know, to put stuff in the right uh too much editing and too much um I mean it was worth it.
00:02:19.919 --> 00:02:21.439
Yeah, definitely, definitely.
00:02:21.680 --> 00:02:25.759
Um it's uh this was a great opener, but uh an even more amazing talk.
00:02:25.840 --> 00:02:28.080
We're talking about uh invitation is all you need.
00:02:28.159 --> 00:02:31.199
This was your um black hat talk together with staff common.
00:02:31.280 --> 00:02:32.479
Uh what was it last year?
00:02:33.199 --> 00:02:34.719
Yes, and with the Ori Air as well.
00:02:34.879 --> 00:02:36.240
And Ori Air as well, of course.
00:02:36.319 --> 00:02:41.280
So basically, zero click attack, Gemini, like full on, full compromise.
00:02:41.439 --> 00:02:50.000
Can you describe in uh like describe in a couple sentences what you think are the um advancements that you've made with that research that will be?
00:02:50.240 --> 00:03:02.639
I think that the greatest advancement is the fact that you're now having much more secured Android devices, which was not the case a year ago on your Google Assistant on your Android devices uh powered by uh Gemini.
00:03:03.120 --> 00:03:08.400
So today, this is the greatest, in my opinion, advancement that we did.
00:03:08.719 --> 00:03:18.240
But within the realm of uh prompt injection, I think that we were the first to first of all introduce two very interesting properties.
00:03:18.479 --> 00:03:22.000
The first one is what I consider the in-device lateral movement.
00:03:22.240 --> 00:03:33.599
We were able to go from the Google Assistant exploiting its permissions to, for example, opening the Zoom application on the same device or opening your web browser on the same device.
00:03:33.840 --> 00:03:36.000
Again, we went from one application to another.
00:03:36.159 --> 00:03:38.719
There are not different tools within your Google assistant.
00:03:38.800 --> 00:03:38.960
Okay.
00:03:39.199 --> 00:03:43.280
Can you can you like take a step back and like uh what's the entry vector, what's the end-to-end story?
00:03:43.439 --> 00:03:56.800
Okay, so invitation is all you need, was coined invitation is all you need due to the fact that all of the attacks were presented via an uh a prompt injection embedded within the title or the subject of a Google Calendar invitation.
00:03:56.879 --> 00:04:11.360
Okay, when the user prompted his Google Assistant on his Android device for what is my recent, uh, what are my latest events, and later on thanked the Google Assistant for providing uh the user with the events.
00:04:11.520 --> 00:04:18.000
Uh, the execution or the payload has been deployed and various malicious activities have been demonstrated.
00:04:18.160 --> 00:04:28.639
For example, in one case, we were able to manipulate IoT devices that are connected to your uh Google Assistant or to your home.
00:04:28.800 --> 00:04:38.240
We're able to trigger the opening of the windows, activating the boiler, and deactivating the lights or turning the lights off within uh a user's apartment.
00:04:38.480 --> 00:04:40.079
And this was like this was real, right?
00:04:40.160 --> 00:04:42.000
This was Oya's uh apartment.
00:04:42.240 --> 00:04:43.839
Yes, it was O Yayair's apartment.
00:04:43.920 --> 00:04:46.480
This is the apartment that is being uh video streamed there.
00:04:46.639 --> 00:04:48.800
Um, this was one of the things that we're able to show.
00:04:48.959 --> 00:05:05.279
This is, by the way, in my opinion, while it was the thing that people were likely to remember from uh this uh kind of uh attack, this was not the most novel thing in the attack because at the end of the day, this was just lateral movement within agents.
00:05:05.600 --> 00:05:11.680
Okay, meaning that you have an agent within your Google Assistant that allows you to control your Google home.
00:05:11.839 --> 00:05:13.199
I would agree one thing.
00:05:13.360 --> 00:05:19.439
This was the first time that we demonstrated physical outcomes triggered by prompt injection.
00:05:19.519 --> 00:05:40.720
I don't think that before we did it, while it was obvious to people who worked in this domain, this was the first three outcomes that demonstrated an end-to-end attack on a real uh system which is being used by uh half of the world, probably, which is the Android, uh which are used by Android devices, that finally showed something which goes beyond digital.
00:05:41.759 --> 00:05:43.600
So this was the first thing, the physical outcome.
00:05:43.680 --> 00:05:52.639
The second thing was the ability to exploit the permission given to the Google Assistant to exploit applications uh which are installed on the device.
00:05:52.800 --> 00:05:55.600
In in our term, it's called the Blast Radius.
00:05:56.000 --> 00:06:02.800
We show that the Blast Radius is no is no longer limited to the LLM application itself, which is the Google Home, which is the Google Assistant.
00:06:02.959 --> 00:06:29.600
You can actually extend your Blast Radius beyond the Google Assistant and compromise or exploit other applications such as Zoom, or for example, we demonstrate how to video stream a user into an attacker control room by uh immediately triggers uh the Zoom application on the Android device, and also showed how to exfiltrate, for example, data from a user's email into a server controlled by an attacker via uh the web browser.
00:06:29.680 --> 00:06:34.000
Okay, which the web browser is a different application which was also being triggered back then.
00:06:34.160 --> 00:06:46.240
And uh this, in my opinion, were the three most interesting things the the fact that the Google Assistant on Android became much more secured, the fact that we're able to show physical outcomes, and the fact that we're able to show uh on device lateral movement.
00:06:46.399 --> 00:06:48.720
So, what do you mean by the fact that the Android became more secure?
00:06:48.959 --> 00:06:58.560
Okay, Google, since then they have an interesting blog where they mentioned that one of the reasons that, or uh among the reasons that uh make them more secure.
00:06:58.800 --> 00:07:01.360
I mean, they actually put they actually put out a blog, right?
00:07:03.199 --> 00:07:09.680
When you release your talk and they specifically name the three of you as uh like contributors to that.
00:07:09.839 --> 00:07:16.959
And by the way, I've been using that blog when other vendors are trying to figure out how do they respond to security to big fines.
00:07:17.199 --> 00:07:22.560
I've been pointing them at that blog saying, like, look, look at how this is a very good example, right?
00:07:22.639 --> 00:07:31.439
Because they basically say, look, these three researchers told us something that was very important, and here is everything we're doing to try to address that.
00:07:31.680 --> 00:07:34.000
Yeah, I can tell you one thing regarding Google.
00:07:34.240 --> 00:07:55.120
When we approach them, and we approach them via the bugnizer or via the vulnerability disclosure program, and also via colleagues of mine that uh work at uh Google, they immediately, immediately, since the first interaction with them, took or treated very seriously in a very responsible manner.
00:07:55.839 --> 00:08:00.240
Uh which I can argue, which takes us back to the previous uh podcast that we have.
00:08:00.399 --> 00:08:01.600
I I can argue this.
00:08:01.680 --> 00:08:05.199
I I did a lot of disclosures throughout the last 10 years.
00:08:05.439 --> 00:08:09.040
This was the most pleasant experience that I had with the vendor.
00:08:09.279 --> 00:08:09.680
Sounds great.
00:08:10.079 --> 00:08:11.360
From the very beginning, okay.
00:08:11.519 --> 00:08:25.439
From the very beginning, they just brought more and more people to understand and they communicated the entire things that they did throughout the process, and they do it in a continuously manner, right?
00:08:26.000 --> 00:08:32.000
Keep sending us emails to update us on what they did in response and where things are about to be deployed.
00:08:32.159 --> 00:08:47.039
Even when the reporter asked for their response and comment, they I would argue brought somebody who is uh an executive and uh he actually said a lot of very interesting things about what should happen in Google and what shouldn't happen in Google.
00:08:47.200 --> 00:08:53.600
Now, 10 years after I started working with the industry on and disclosing stuff, I know that this is the exception.
00:08:53.840 --> 00:09:07.759
In many cases, you don't get people who either take things seriously, in many cases, they try to diminish or to dismiss your uh findings and regarding responsibility or whether it's in scope or not, and some other stuff.
00:09:08.399 --> 00:09:14.799
Since the very beginning, we've seen how Google uh took it uh in the in the right direction and immediately responded to it.
00:09:15.120 --> 00:09:18.240
So, how did uh what did they change about Android to make it safer?
00:09:18.639 --> 00:09:27.039
Many of the mitigations that they mentioned in the blog that you mentioned were not deployed, or if they were deployed, they were very ineffective.
00:09:27.120 --> 00:09:27.279
Okay.
00:09:28.320 --> 00:09:45.679
One thing that they did deploy, which definitely wasn't deployed back then, was they mapped what they considered as high-risk activities or high risk outcomes, and they decided to put uh, for example, user confirmations before of them.
00:09:45.840 --> 00:09:58.000
Okay, moreover, they uh significantly improved the robustness of the indirect prompt injection classifier so it will be much better in detecting uh malicious intents.
00:09:58.559 --> 00:10:20.720
They uh added a dedicated URL reduction mechanism to make sure that even if you were able to combine stuff or to create things on the fly by exploiting the LLM, you still won't be able to uh insert uh URLs into the large language models without the permission uh of the user.
00:10:20.960 --> 00:10:32.639
And again, they they did various stuff within this domain in order to make sure that the strength mitigation they back then have to be much more robust against prompt injections.
00:10:33.039 --> 00:10:41.679
So you mentioned at the beginning that uh a user, in order to kind of be exploited, a user had to first interact with their calendar and then say thank you.
00:10:41.919 --> 00:10:42.159
Right?
00:10:42.240 --> 00:10:43.679
And that thank you is super important.
00:10:43.759 --> 00:10:44.960
Can you can you describe why?
00:10:45.200 --> 00:10:49.440
Yeah, it's not such an important thing, but I would argue this.
00:10:49.519 --> 00:10:52.960
Okay, we use the technique called um delay tool invocation.
00:10:53.120 --> 00:11:10.559
By the way, I now work with my uh with a student of mine, with two students, uh basically, on understanding why what I consider as conditional prompts, which at the end of the day, a delay to uh uh delay tool invocation is a conditional prompt, which is being triggered by an external uh trigger, are considered auto-distributions for uh mitigations.
00:11:10.799 --> 00:11:19.919
We actually conditioned the execution of the entire activities that I've just mentioned in a response by the user, which is thank you.
00:11:20.240 --> 00:11:29.519
We could, for example, uh condition not in a specific word, but in just condition it in when no matter what the user said, do this.
00:11:29.840 --> 00:11:31.039
But you had to condition, right?
00:11:31.360 --> 00:11:35.840
Yes, the condition made the prompt out of distribution for the classifier and was able to bypass it.
00:11:36.080 --> 00:11:48.399
Meaning that take any kind of state-of-the-art industrial firewall, if it is robust against regular indirect prompt injection, it is most likely to fail when you condition the exact same payload.
00:11:48.559 --> 00:11:59.200
Take the payload, condition it in something, and you are having much better bypass rate than the regular uh prompt injection, meaning that it's out of distribution for existing uh mitigations.
00:11:59.360 --> 00:12:11.200
Okay, back then, delay tool invocations were out of distribution for the mitigations, and they were also allowed us to jump between different scopes or context of different agents.
00:12:11.360 --> 00:12:13.600
Because we discussed about it in the paper.
00:12:13.840 --> 00:12:27.360
Back then, the the Google Assistant was uh implemented as uh what we consider as an orchestrator with microagents, which basically triggered by the orchestrator, and they had full isolation of context.
00:12:27.840 --> 00:12:59.679
They didn't share context with each other, they only shared uh outputs with the orchestrator, and the orchestrator could use this output to you know to as an input to a different technique, which was in initially discovered by uh Johann Reiberger, helped us to do this kind of stuff to allow us to jump between context windows of different agents and uh allowed us to ensure that even if we were able to bypass uh the mitigation, it increases uh the ability that the payload will be deployed.
00:12:59.840 --> 00:13:08.159
Even if the payload is within the context, take it and compare the success rate when it is in the context without a condition being triggered.
00:13:08.320 --> 00:13:16.240
And with a condition being triggered, with the condition being triggered, you have much better uh results, much better success rate in deploying your uh so essentially it's like two things.
00:13:16.320 --> 00:13:30.240
Uh I I I thought I originally thought of this as a mechanism to bypass the mitigation that Google has where you just cannot get the result of one tool to get the LLM orchestrator to call another tool because they just blocked the second tool.
00:13:30.720 --> 00:13:32.240
Okay, so that's what I was thinking this year.
00:13:32.480 --> 00:13:39.039
It was out of distribution for the mitigation, so it actually allowed us to uh not being identified as something which is risky.
00:13:39.360 --> 00:13:41.519
Okay, despite there are the same payloads, right?
00:13:41.679 --> 00:13:44.879
But in one form it's not risky, in another form it's it is risky.
00:13:45.279 --> 00:13:54.399
And second thing, even if you were able to cross the mitigation into the context, in many cases this is just not enough.
00:13:54.480 --> 00:14:06.000
Okay, because what you are about to see now is something which is within the context, but when the user asks for something else, like for example, you ask to open the boiler, okay, without any condition on it.
00:14:06.399 --> 00:14:08.960
And this is within the orchestrator context.
00:14:09.120 --> 00:14:16.320
Okay, you were able to have it within uh Google invitation, the Google ask for the meetings.
00:14:16.480 --> 00:14:32.080
As a result, this compromise the orchestrators context, but next thing that the user asks is to say I uh okay, this is it, or said, okay, then schedule me uh a calendar event with someone else.
00:14:32.320 --> 00:14:42.559
The thing is the condition, when you condition something, it helps to have a better success rate in uh for the uh for the attacker to uh to make sure that the payload is being uh deployed.
00:14:42.799 --> 00:14:43.759
That's very interesting.
00:14:44.000 --> 00:14:50.559
Yeah, so these are the two things that are in at the core of uh delay tool invocation or what I consider as conditional prompts.
00:14:50.720 --> 00:14:56.960
There are a lot of things that I think that Johan wasn't even understood when uh he thought about it.
00:14:57.039 --> 00:15:11.919
I I'm not even sure that even today is completely understand what exactly is happening in uh not in the architectural level, it's very good to understand architecture, but in the neuron level, what exactly is happening in the attention layer which triggers the, you know, the how the triggers affect the the rest.
00:15:12.320 --> 00:15:13.919
So there are very interesting things.
00:15:14.000 --> 00:15:23.360
Uh, if you uh look on it, and uh I would argue that one of the things that I'm not sure that Johan understood because he called it delay tool invocation.
00:15:23.519 --> 00:15:26.159
Delay tool invocation is a form of conditional prompt.
00:15:26.240 --> 00:15:29.039
It's a prompt who has a condition to be executed.
00:15:29.200 --> 00:15:31.360
Think about it for a second on conditional prompt.
00:15:31.840 --> 00:15:38.399
Conditional prompt actually extends the functionality of prompt injection.
00:15:38.799 --> 00:15:39.360
Why?
00:15:39.600 --> 00:15:43.360
Because first of all, you can wait for a dedicated event, right?
00:15:43.519 --> 00:15:46.240
Which at the beginning you may not have it.
00:15:46.399 --> 00:15:55.600
For example, you can wait until the user asks for sensitive uh data to be brought into, unlike the case that you just immediately ask to exfiltrate stuff.
00:15:55.679 --> 00:15:56.320
Yeah, okay.
00:15:56.559 --> 00:16:00.559
Second thing, you can spear the attack, you can make it targeted.
00:16:00.639 --> 00:16:04.960
Okay, you can decide on whether this is the target that you want to uh to attack.
00:16:06.399 --> 00:16:14.159
Uh do it only if the attacker is the CEO and this is the application and this is the company, right?
00:16:14.240 --> 00:16:22.480
Yeah, so it helps you to be much less noisy with respect to uh it's out of distribution, which is think about it for a second.
00:16:22.559 --> 00:16:30.399
We just described the new method to bypass LLM firewalls uh or prompt injection firewalls with much higher uh bypass rate.
00:16:30.559 --> 00:16:39.840
Think about all the things that you can now do, event-driven targeted attacks, specific tools in time that you want uh to uh to wait for them.
00:16:40.240 --> 00:16:43.440
For the targeted attack, I think I remember one Johan blog.
00:16:43.519 --> 00:16:50.639
Uh uh it's funny, that that uh addresses this, like that talks about this, but it's I think uh Johan is like the Bible here, right?
00:16:50.799 --> 00:17:02.960
Everywhere when you when you have you have something, you you think you've kind of found something new, then you go out, you scroll, you scroll through Johan's blog, and he's like, Oh yeah, like three years ago, like he's amazing.
00:17:03.120 --> 00:17:13.839
But I I I get what you mean, it's because uh most attacks on LLMs are not really persistent, and even if they are persistent, they only take into consideration what's currently in the context.
00:17:14.079 --> 00:17:22.880
And essentially you're saying, Hey, now you can you can compromise the future, and it doesn't really mean it doesn't really matter if you persist or not, you are there in the context.
00:17:23.119 --> 00:17:35.359
And and yeah, that's uh Which again this extends your ability as an attacker to uh first of all perform silent attacks or to spill to have the attack tailored to either a specific target or specific event in time.
00:17:35.680 --> 00:17:41.680
Like just uh detonate only on me, not on you, or detonate on one specific target or one specific event.
00:17:42.000 --> 00:17:43.920
Makes things much harder to detect in reality.
00:17:44.240 --> 00:17:51.359
Google uh DeepMind published their uh Camel paper, what is it, uh two years ago, a year and a half ago, something like that.
00:17:51.599 --> 00:17:51.839
Yes, yes.
00:17:52.000 --> 00:17:56.799
A year, I don't know, my time timing with AI is uh for me, it's changed drastically.
00:17:56.960 --> 00:18:00.079
Uh, like one question is why haven't they deployed it here with Gemini?
00:18:00.240 --> 00:18:01.519
But also why is this not a reality?
00:18:01.759 --> 00:18:02.720
Good, okay, good.
00:18:02.960 --> 00:18:06.880
First of all, there are a lot of good answers to this question.
00:18:07.119 --> 00:18:10.640
I will start with the most primary, in my opinion, among them.
00:18:10.799 --> 00:18:17.200
After experiencing with Camel, Camel, what you consider as uh academic kind of uh research, okay?
00:18:17.519 --> 00:18:22.480
Camel is good up to some degree to a single inference.
00:18:22.640 --> 00:18:27.279
When you are in a multi-inference kind of uh scenario, Camel doesn't make sense to be used.
00:18:27.359 --> 00:18:38.720
Uh, Camel is by definition intended to secure against one single inference that obtained data or untrusted data from the external world.
00:18:38.880 --> 00:18:48.559
It doesn't allow the developers to count on the data brought from the external world for the next inference for planning or some other stuff.
00:18:48.720 --> 00:19:03.200
Meaning, in other words, it doesn't good for multi-inference or for uh chatbots or for any kind of uh situations where you have this multi-inference kind of uh scenario between users and the application, okay.
00:19:03.279 --> 00:19:11.039
When users continuously feed the application with new inputs, and some of them may rely on the answers of the LLM.
00:19:11.680 --> 00:19:14.799
Camel doesn't go well, okay.
00:19:15.039 --> 00:19:20.799
And moreover, Camel requires significant, significant uh adjustments to the code.
00:19:20.960 --> 00:19:24.720
They are not easy to deploy, very not easy to deploy, okay?
00:19:24.960 --> 00:19:40.480
And in many cases, they uh very rely heavily on the user to confirm and confirm and confirm for each and every data that is being uh digested into the planning and some other stuff, which is uh bad usability if you think about it.
00:19:40.640 --> 00:19:51.279
So, this is one reason it's too academic, it's not well adjusted to multi-inference, and uh there's significant overhead in implementing stuff within the code.
00:19:51.599 --> 00:20:02.000
Second thing is that the guys that wrote the papers on Camel are Google researchers, okay, that research the area of AI security.
00:20:02.240 --> 00:20:09.680
The guys who implement uh the uh security against uh Gemini in reality are engineers.
00:20:10.000 --> 00:20:13.680
Google is a very, very big organization, okay.
00:20:13.920 --> 00:20:24.799
I can tell you, based on my experience, in many cases, not even a word of the work that has been done in different departments within Google, even if they somehow intersect.
00:20:25.039 --> 00:20:29.279
I think this is something that should be asked, should be referred to a Google engineer.
00:20:29.440 --> 00:20:44.559
I believe that the Google engineer back then were not, and even today, I'm I'm not even sure that they are completely understand uh they are familiar with Camel, and mostly because Camel is not the practical I think um like we we haven't seen this implemented basically ever anywhere.
00:20:44.720 --> 00:20:49.599
And uh, if you even if you take Camel, like you try to bring in Camel to the agent space.
00:20:49.759 --> 00:20:59.039
So basically, my my rudimentary uh how would that look like would be hey, you got one LLM that doesn't read any of the untrusted data.
00:20:59.200 --> 00:21:07.920
It basically instead of doing the React loop, it writes code, and that code still doesn't see any of the untrusted data.
00:21:08.079 --> 00:21:15.200
And then the secondary, what they in Camel they call like the unprivileged or the sandbox LLM.
00:21:15.359 --> 00:21:23.279
You don't leave use like it populates the parameters for each function or something like that on the fly when it reads the untrusted data.
00:21:23.519 --> 00:21:29.839
That is kind of my how do you take it to uh something that is that is like a bigger but nobody's using it.
00:21:30.000 --> 00:21:32.559
Okay, here is a subtle thing that you need to consider, okay?
00:21:32.720 --> 00:21:40.400
There are two subtle things in here that in one case you can have the LLM, uh, you can ask the LM to provide you with the recommendation, okay?
00:21:40.559 --> 00:21:47.680
And later on, to have the LLM, based on the recommendation that was brought from the internet, plan something with respect to this recommendation.
00:21:47.839 --> 00:21:56.480
Camel doesn't know how to handle it without the user extensively uh being a prompt that uh you are about to uh to to have a disaster situation.
00:21:56.720 --> 00:21:57.759
So this is one thing.
00:21:57.920 --> 00:21:59.759
Bear in mind, Camel to implement.
00:22:00.319 --> 00:22:09.920
It and to validate whether the data is uh within the ranges, okay, is not considered anomaly, it requires significant effort.
00:22:10.160 --> 00:22:16.400
Okay, you need to have the validators that you've just mentioned within the code, the script that is uh being generated.
00:22:17.200 --> 00:22:28.400
Test whether the input or the argument is reasonable comparing to reasonable arguments that uh um parameters that were are considered allowed.
00:22:28.799 --> 00:22:34.400
Okay, this is not easy, it requires a significant effort from the development effort.
00:22:34.720 --> 00:22:36.160
It's like a configuration, a huge configuration.
00:22:36.559 --> 00:22:37.440
Exactly, it's not easy.
00:22:37.599 --> 00:22:38.559
Not easy.
00:22:38.799 --> 00:22:50.960
Doesn't go well when you want to go to multi-inference, may require significant maintenance for each and every changes within the logic of the application and some other stuff.
00:22:51.200 --> 00:22:51.839
Not easy.
00:22:52.160 --> 00:22:58.240
What are your thoughts on the on Microsoft's uh information control flow or information flow control, I think they call it.
00:22:58.319 --> 00:23:04.240
Uh like essentially their idea is tracking taints across throughout the context of an agent.
00:23:04.400 --> 00:23:13.759
So they say, hey, there is uh again, my reading of it, they're saying, hey, you need to track the integrity of the data and the confidentiality of the data.
00:23:13.920 --> 00:23:20.400
Uh and with integrity, my read of it is, hey, let's fight, let's figure out where untrusted data comes into context.
00:23:20.559 --> 00:23:23.759
And confidentiality is any any kind of confidential data.
00:23:24.000 --> 00:23:36.880
And so if everything that the agent does, now agent now LLM uh gets a label, then you can know when there is a kind of a mismatch or a combination of labels that shouldn't be the case.
00:23:37.039 --> 00:23:47.359
So if you're touching sensitive data and you also have uh any untested data in the context that could abuse the agent to uh to abuse the sensitive data.
00:23:47.680 --> 00:23:50.960
So I will start by mentioning again, I haven't read the paper.
00:23:51.039 --> 00:24:02.799
If you ask me as somebody who works with machine learning for too long, I would argue that uh false positives are at the end of the day something that could uh break your entire uh thing into uh you know pieces, making it unpractical.
00:24:03.039 --> 00:24:04.400
At the end of the day, I do not know.
00:24:04.559 --> 00:24:07.200
I would like to see real results on real systems.
00:24:07.359 --> 00:24:12.000
At the end of the day, if you have something which is really working, you will see it immediately being deployed.
00:24:12.160 --> 00:24:25.200
Uh, since we currently lack the ability to mitigate this stuff, we what you see each and every week is additional mitigation that claims that they are the best and they are and nothing is the best, right?
00:24:25.599 --> 00:24:35.359
When you see clients deploy mitigation and say, look, this is the mitigate, this really blocks, this handles the threat, this decreases the risk significantly.
00:24:35.519 --> 00:24:44.799
Uh only then you will probably know that uh I haven't seen so far any kind of decent mitigation, which I will be able to argue that I believe in.
00:24:45.279 --> 00:24:47.680
What are your thoughts on uh Simon Willison's Little Trifecta?
00:24:47.920 --> 00:24:53.599
He was not only right, he was uh the first who introduced uh the three factors, right?
00:24:53.839 --> 00:25:00.240
I would argue that today you need to extend it beyond exfiltration to actuation as well.
00:25:00.400 --> 00:25:07.839
Whether it is exfiltrating data was something that was maybe two years ago was the you know the chatbot era.
00:25:08.240 --> 00:25:08.640
Yes.
00:25:08.880 --> 00:25:24.400
But today, when you have these things capable of running stuff locally in terminal or capable of opening your windows inside your home, or I don't know, like you know, open your zoom, maybe Zoom you can consider it a bit of an exfiltration, right?
00:25:24.640 --> 00:25:46.880
But to do stuff they shouldn't do due to the permission, the the last factor which was communicated outside, I would argue that needs to be somehow better uh adjust to this era, which is an actuation which may have sensitive or some kind of uh may consider a threat to the user.
00:25:47.119 --> 00:25:51.920
But even the data exfiltration mitigation has not really been deployed anywhere.
00:25:52.160 --> 00:26:02.799
Like uh I I can't imagine I can think of even one example when where people are baking and lethal trifecta as a mitigation into a flagship AI product, even just for the data exfiltration part.
00:26:03.119 --> 00:26:08.559
Because you need to basically remove one of the factors so it affects your usability.
00:26:08.799 --> 00:26:15.759
And at the end of the day, you end with security that prevents you from doing stuff rather than allowing you to do stuff, right?
00:26:15.839 --> 00:26:20.640
Which is not the best place you want to be on the trade-off between security and usability.
00:26:20.720 --> 00:26:25.279
You need to find a way to allow uh usability with uh decent security.
00:26:25.359 --> 00:26:27.519
This is at least my perspective about security.
00:26:27.759 --> 00:26:46.720
I would argue that Camel also took the the wrong, they weren't like on the worst case of they provide good security, worst usability to the and again you you need to find things that have much better balance in terms of security and in terms of uh usability, which I haven't seen so far.
00:26:46.960 --> 00:26:56.000
In your prompt were uh kill chain paper, you mentioned a couple of uh other mitigations that I thought were interesting, but again, I haven't seen them really deployed at scale.
00:26:56.160 --> 00:27:05.200
One was canaries, so hey, let's have uh like canary tools that when somebody when the agent invokes them, you know that the agent has been compromised.
00:27:05.359 --> 00:27:09.839
And the other was prompt perturbation, which I found was really cool.
00:27:10.079 --> 00:27:20.880
So, my my understanding of that was basically um every time you're gonna send a prompt to the agent, we're gonna add some salting mechanism into the LM core.
00:27:21.039 --> 00:27:34.319
So now no single it basically uh uh prevents me from creating a consistent exactly to overfit your attack against a specific prompt that will collide in the embedding space into something which is a disaster.
00:27:34.640 --> 00:27:35.119
Right.
00:27:35.279 --> 00:27:40.480
Uh again, soft mitigations are very easy to uh to bypass, right?
00:27:40.640 --> 00:27:44.640
Yeah, I haven't seen very strong mitigation so far.
00:27:44.960 --> 00:27:54.720
If you think about it, in computer security in architecture, we have different spaces for or addresses for code and for data.
00:27:54.880 --> 00:28:02.240
Okay, the way that large language models today work, they don't have any kind of separation between code and data.
00:28:02.400 --> 00:28:13.759
And I would argue that it's not even easy to have this separation between code and data, because in many cases you want to use the data to create the new code, and then you have a mismatch between what could happen.
00:28:14.079 --> 00:28:23.759
I also think in that like uh that that uh argument on uh of kind of code, uh codes versus instructions in LLMs, I feel it's uh kind of missing the point.
00:28:24.000 --> 00:28:32.319
Because if let's say magically you get a magic wand, and tomorrow we got an LLM that has different channels for instructions and data.
00:28:32.559 --> 00:28:44.960
If I manipulate that data, I can still not the instructions, I can still get the LLM to do whatever I want because the data is what the LLM is judging to make its calls, right?
00:28:45.279 --> 00:28:52.559
Okay, you can plan the plan only based on the code and then incorporate the data into it.
00:28:52.720 --> 00:28:58.240
So the data will be parameters, it will not change end-to-end the execution.
00:28:58.559 --> 00:29:01.119
It will change which path of the execution you're gonna take.
00:29:01.519 --> 00:29:22.079
Uh it it could change one thing, which is the parameters, which I would argue is a bad thing because if you have, like, you know, if I instructed the LLM to send an email to John and the data instructed to send it to the attacker, okay, so same path, but uh you go from desired outcome to a threat of data exfiltration, right?
00:29:22.559 --> 00:29:28.000
So you could up to some degree, and this takes me back to Camel to discuss about Camel.
00:29:28.559 --> 00:29:44.000
One of the things that Camel were uh I think the first to recognize is the fact that even if you just plan things based on the instruction, this is not enough because parameters could be changed via the data, via instructions embedded into the data.
00:29:44.240 --> 00:29:54.319
Okay, and when you have it in this specific case, you understand that going through address spaces will not be that good of mitigation, as you mentioned.
00:29:54.480 --> 00:30:05.039
And moreover, not using parameters taking from the data is something that significantly uh reduced the usability of a system.
00:30:05.279 --> 00:30:06.720
Yeah, so it's not it's just not gonna be a good idea.
00:30:06.960 --> 00:30:07.839
It's not a system, yeah.
00:30:08.240 --> 00:30:13.200
So I want to kind of um I want to switch off to a different topic.
00:30:13.359 --> 00:30:15.519
Or I think it's it's close by.
00:30:15.759 --> 00:30:37.519
When we so last, I think this was October, you gave uh you gave a uh a really good talk on the on our agent security summits, uh, and you mentioned the evolution of promptwell, and that led up to the zero click attacks we saw last year, uh your Gemini attack, we did our ChatGPT attacks, Johan did a bunch of different attacks.
00:30:37.680 --> 00:30:41.039
Um, and that was the first what we called the zero click.
00:30:41.440 --> 00:30:46.640
We called it a zero click because there was no nothing bad the user had to do.
00:30:46.799 --> 00:30:55.119
The user was just going about their day, the attacker would send an email or share a document or whatever it is, and it would be delayed.
00:30:55.279 --> 00:30:58.559
So they will not be uh at control when it would trigger.
00:30:58.720 --> 00:31:01.839
But eventually, when a user just goes about their day, it will trigger.
00:31:02.000 --> 00:31:07.759
And we had a discussion about that, and you told me, listen, honestly, this is not really a zero click in my mind, it's a half click.
00:31:07.839 --> 00:31:09.119
So can you can you expand on that?
00:31:09.359 --> 00:31:09.759
Yeah, yeah.
00:31:09.839 --> 00:31:29.200
So if you will uh okay, I I think I did some uh I abused the term zero click, so the entire industry now used the term zero click within the context of Pontwell, yeah, uh, because I use this term in the wall, the AI one that we presented, and ever since many people use the exact same term, which is zero click.
00:31:29.279 --> 00:31:34.160
Okay, a true zero click is what Pegasus has done.
00:31:34.400 --> 00:31:40.960
A true zero click is something which is initiated by the operating system.
00:31:41.359 --> 00:31:50.720
At the end of the day, the entire uh up to up to today, okay, the entire things that we've seen are initiated by user.
00:31:51.279 --> 00:31:53.759
There is a significant difference in here, okay.
00:31:54.000 --> 00:32:10.319
Even an invitation is all you need, which you can ensure that the compromised uh data, okay, or the compromised invitation will get into the user without the user having or prompting the assistant for his recent events, the prompt injection will not occur.
00:32:10.640 --> 00:32:16.319
But you can set up the front injection in a way where it doesn't address like it's it's not really addressing one prompt, right?
00:32:16.400 --> 00:32:17.359
It's addressing a topic.
00:32:17.440 --> 00:32:19.039
Like you can you can make it generalize.
00:32:19.359 --> 00:32:25.039
But without being without having the user initiate the right prompt, it will not be executed.
00:32:25.279 --> 00:32:25.599
Yes.
00:32:25.759 --> 00:32:27.759
Okay, so this is why I consider it a half-click.
00:32:27.920 --> 00:32:28.160
Okay.
00:32:28.559 --> 00:32:33.519
You would argue that up to some degree those actions are frequently happened by users.
00:32:33.759 --> 00:32:37.599
For example, asking for their events, asking for the recent emails and some other stuff.
00:32:37.759 --> 00:32:48.000
But at the end of the day, without the user having prompt the LLM application for his recent emails, for his recent events, this will not be executed.
00:32:48.160 --> 00:32:51.359
Uh, a pure zero-click attack is Pegasus.
00:32:51.680 --> 00:32:58.960
Pegasus is executed due to the um execution being triggered by the operating system.
00:32:59.119 --> 00:33:02.960
It is being guaranteed to operate when it is being deployed.
00:33:03.119 --> 00:33:12.319
Okay, this is by the way they call it a Pegasus, it's uh a Trojan with uh with the wings that uh when you send it, it's uh being uh deployed.
00:33:12.559 --> 00:33:15.519
So you're saying the the thing is as the guarantee.
00:33:15.680 --> 00:33:26.720
Because I I I see your point that uh if a user doesn't do anything, if a user doesn't use this Gemini or ChatGPT or whatever it is, then they won't get compromised.
00:33:26.880 --> 00:33:30.880
However, I think it's important, it was important for me to make that distinction.
00:33:30.960 --> 00:33:38.000
Like we use zero click because we basically said listen, there is no way for you to defend yourself against this because you don't know.
00:33:38.079 --> 00:33:45.200
As a user, you just ask, in our case, it was uh anything about a meeting, like you just ask anything about a meeting and you get popped.
00:33:45.359 --> 00:33:49.920
And you didn't ask about a specific meeting, I didn't need to know in advance what were your meetings.
00:33:50.240 --> 00:33:53.039
I agree, but but here is my thought, okay?
00:33:53.359 --> 00:34:02.640
In about a few months or a year or so, inference will be triggered into your it is already being happened if you have uh iPhone, right?
00:34:02.720 --> 00:34:05.759
But it's currently happened without any tools to compromise.
00:34:06.000 --> 00:34:13.280
So you already have the mechanism that it will happen without any user uh initiation, guaranteed to happen.
00:34:13.840 --> 00:34:21.599
Guaranteed to happen just by having it uh within the just by you sending it, pushed it into the victim, right?
00:34:22.000 --> 00:34:23.519
And this is about to happen.
00:34:23.760 --> 00:34:26.880
You understand and agree that there is a great difference between the two.
00:34:27.039 --> 00:34:34.559
One thing is guaranteed to happen, one thing is conditioning in a user activity to be prompt, and then it is being uh executed.
00:34:34.719 --> 00:34:45.599
And again, I've in one of our discussions, and especially in this talk, I've mentioned that automatic inference initiated by operating system is either now being deployed or about to be deployed.
00:34:45.760 --> 00:34:50.960
Okay, iPhone uses automatic summary for the notifications.
00:34:51.199 --> 00:35:02.800
Okay, if you think about it, you have a pure zero-click path, whether it will likely be do something malicious is a function of whether there are tools to exploit or not.
00:35:03.039 --> 00:35:11.599
So you're saying if if somebody on the uh summarization team on the Apple side is going to be, hey, like why not hook up a bunch of tools to this uh summary?
00:35:12.079 --> 00:35:13.440
Most likely it will happen, right?
00:35:13.519 --> 00:35:17.119
Because I don't know why it will make sense up to some degree someday.
00:35:17.199 --> 00:35:17.440
Okay.
00:35:18.000 --> 00:35:32.960
Um there is an additional case where, if you think about it, is the autonomous uh robots, for example, that will likely to use a multimodal LLM to digest everything they see.
00:35:33.039 --> 00:35:45.199
And here you have a path for a completely zero click, which you don't need to do anything special automatically, an inference will be triggered in response to, you know, it just happened.
00:35:45.280 --> 00:35:45.440
Okay.
00:35:45.599 --> 00:35:47.519
How do you view uh like a background agents?
00:35:47.599 --> 00:35:58.400
Because with background agents today, agents that don't have a human in the loop, you do already have this path where you can get an injection in, the agent will be invoked, the agent will get compromised.
00:35:58.559 --> 00:36:04.239
Uh, I have uh like one example I have in mind is the uh the Klein NPM compromise.
00:36:04.320 --> 00:36:08.800
So there was this case where Klein was for a few uh versions.
00:36:08.960 --> 00:36:12.320
If you installed Klein, it would install OpenClone on your machine.
00:36:12.559 --> 00:36:16.079
And it was not really a compromise, it was more like I think for the lulls.
00:36:16.400 --> 00:36:21.679
Uh I actually played around with trying to figure out what happened there and the attack path.
00:36:21.760 --> 00:36:27.679
I mean, it was somebody that caught uh Adnan Khan's research uh and then replayed it.
00:36:27.920 --> 00:36:33.039
Um, and basically it was prompt injections through an issue, GitHub issue.
00:36:33.119 --> 00:36:38.239
You open an issue, the ticket has the title of the issue has a prompt injection in it.
00:36:38.400 --> 00:36:45.119
There was uh a cloud that was kind of looking at issues, triaging issues, and then you can compromise that cloud.
00:36:45.440 --> 00:36:58.559
I think that the difference between half-click and zero click is whether the inference happened automatically, or whether you consider it it's not it's a manual inference, but it's something which you consider as frequent.
00:36:59.039 --> 00:37:10.880
Okay, this is, in my opinion, in my view, the main difference between whether it's an operating system-initiated inference or a frequent user-initiated uh uh inference.
00:37:11.119 --> 00:37:15.039
This is the I think the best way to distinguish between the the two.
00:37:15.280 --> 00:37:18.159
You mentioned agents running in the background.
00:37:18.320 --> 00:37:29.599
Agents running in the background are what you consider vulnerable to zero-click attacks, given that they have, like, you know, the ability to uh obtain uh sensitive data and to execute stuff using the tools that they have.
00:37:29.760 --> 00:37:33.280
Definitely, this is something that should be taken into consideration within this context.
00:37:33.360 --> 00:37:53.039
It's it's a real threat, I would argue, where in many cases you can even optimize your success rate as an attacker to make sure that you were able to compromise or you were able to get into the agent running in the background, even if it doesn't have a specific uh identity that you can associate with.
00:37:53.280 --> 00:37:56.800
Because you're you're just you're just after that resource or you're after that organization.
00:37:57.039 --> 00:37:58.000
Exactly, exactly.
00:37:58.320 --> 00:38:00.800
So I wanna switch off to another topic.
00:38:00.960 --> 00:38:08.079
You've been on the Black Hat review board uh for what, like for many years now, like I think 2020, 2022, something like that.
00:38:08.239 --> 00:38:09.119
Something like this, yes.
00:38:09.440 --> 00:38:23.840
And I mean AI has obviously been a major part of we've seen them we've seen incredible uh talks coming out of Black Hats that are that have AI in them or that are use AI, either hacking AI or using AI for security.
00:38:24.000 --> 00:38:38.079
Can you take me through what's your so if you if you look at the progress of uh the of security and AI since 2022, when did you see jumps in that progress?
00:38:38.159 --> 00:38:39.280
I'm sure there were progress.
00:38:39.519 --> 00:38:47.360
Okay, so uh I think around 2022 or 2021, this was the first placket to introduce the AI track.
00:38:47.519 --> 00:39:05.920
Okay, back then the AI track was uh like AI kind of uh was baked into various tracks, for example, such as cryptocurrencies today, they don't have their special their special track, they're just you can find them within network or maybe within appsec or maybe within something else.
00:39:06.159 --> 00:39:14.559
So the AI track started before I came into the I joined the the board members of uh Blackit.
00:39:14.880 --> 00:39:22.159
But ever since, I think in the first year or so, started to become interesting.
00:39:22.320 --> 00:39:29.840
It was something that started very with uh, I don't argue few or too few uh kind of uh submission.
00:39:30.159 --> 00:39:37.519
It it did have it was I think the fifth or the sixth or the seventh biggest track in terms of its submission.
00:39:38.000 --> 00:39:44.719
When I I think two years ago it became among the top three, and today this is the biggest one among them.
00:39:44.800 --> 00:39:56.239
So it's actually takes you a lot of time to uh go through the submissions in this track, which is not easy, it's uh by far the biggest uh load that you have comparing the other tracks.
00:39:56.559 --> 00:39:59.920
I would argue that we can see two kinds of topics within this track, okay?
00:40:00.079 --> 00:40:04.880
And as of next blackhead, there will be two separate tracks uh for them.
00:40:05.119 --> 00:40:18.800
The first topic is what you considered as attacks against LLM applications, and the second kind of talks are either AI for security or AI for attacks.
00:40:19.119 --> 00:40:23.440
The AI for security started around it started well before.
00:40:23.519 --> 00:40:24.800
It wasn't something very new.
00:40:24.960 --> 00:40:28.639
They just didn't use LLM, they just used neural networks to do some other stuff.
00:40:28.719 --> 00:40:33.360
Maybe they use like the traditional machine learning, such as I don't know, random faults and some other stuff.
00:40:33.679 --> 00:40:56.079
Around the time that LLMs were started to be integrated into applications, and when they were able to, when people really demonstrated they are make sense in like you know, improving your results for security in various use cases, we started to see a real increase, making it the biggest one at Blackhead.
00:40:56.239 --> 00:41:09.119
And today, again, this is I think that the last time that I checked, the AI track was twice in terms of the number of submissions than the second uh track.
00:41:09.280 --> 00:41:27.280
And if you think about it, Black It or the traditional core of the topics of Black Hat is not AI, it's things considered as memory uh corruptions, things considered as mobile, which started about 20-15 years ago, something like this, cloud security.
00:41:27.519 --> 00:41:29.519
These things were almost diminished.
00:41:29.760 --> 00:41:41.039
Uh, we we now receive only a handful of uh talks among some of these uh tracks where we have uh more than 100 uh talks in the AI track.
00:41:41.199 --> 00:41:42.079
I find it interesting.
00:41:42.239 --> 00:41:50.960
Also, if you incorporate the fact that uh you have AI within the AppSec or within some additional stuff, you see a lot of talks in this domain.
00:41:51.440 --> 00:41:52.719
How does a good submission look like?
00:41:52.880 --> 00:41:54.400
Okay, this is a good question.
00:41:54.719 --> 00:42:07.920
I constantly write posts on how to submit to Blackett because I see that people do not know how to submit to Blackhead, even if they do a good job in the research, they don't know in many cases how to submit.
00:42:08.239 --> 00:42:22.159
So bear in mind that as a Blackhead reviewer, you cannot spend when you have like 200 submissions uh waiting for you to uh to judge them, you cannot spend an hour to read the entire submission.
00:42:22.400 --> 00:42:40.800
Okay, you need to be very accurate in what you want to show the reviewer and do it from the very beginning and to make sure that you have the needed, I would argue, evidence that you are able to do this.
00:42:41.039 --> 00:42:44.880
It starts with an interesting title that you want to read, okay?
00:42:45.039 --> 00:42:46.639
Something that you haven't seen before.
00:42:47.440 --> 00:43:00.960
It goes with an abstract that uh discusses the talk and the findings and shows you that this is something which is applicable to various systems, not only one.
00:43:01.119 --> 00:43:14.159
Okay, it must be novel because you're trying to convince uh the biggest, or I would argue, the the biggest experts in their field that what you did is uh something they haven't seen before.
00:43:14.639 --> 00:43:16.400
Okay, so it must be novel.
00:43:16.800 --> 00:43:20.400
It should, in my opinion, come with uh the needed validations.
00:43:20.559 --> 00:43:31.679
The needed validations uh could be, for example, videos, they could be, for example, CVSS scores, they could be, for example, the manufacturer response.
00:43:32.000 --> 00:43:44.880
I take a great effort to understand what the manufacturer or what the vendor has done in response to the this finding to understand whether they consider it a real issue or whether this is something which only the researcher thinks it's a real issue.
00:43:45.039 --> 00:43:50.639
And there are good ways to show that this is a something that should be taken into consideration.
00:43:50.800 --> 00:43:56.320
Now, beyond, I'm looking for something which is uh, you would argue have the needed details to understand.
00:43:56.480 --> 00:43:59.760
I try to envision what I will see throughout the talk.
00:44:00.159 --> 00:44:03.920
Okay, in many cases the agenda is not clearly written.
00:44:04.400 --> 00:44:12.079
Or you you don't understand what exactly is about to be presented, or what is exactly you're about to see.
00:44:12.239 --> 00:44:14.960
I also take the need of time to read the takeaways.
00:44:15.199 --> 00:44:19.199
Are there real takeaways in here, or is it just things that we already know?
00:44:19.440 --> 00:44:25.840
For example, um, the bar for prompt injection has significantly changed.
00:44:26.000 --> 00:44:37.760
If the main takeaway of your talk is we found that this application is also vulnerable to prompt injection, so you just one out of many others that we just already know.
00:44:37.840 --> 00:44:42.960
I that I assume that every kind of application is currently uh vulnerable to prompt injection.
00:44:43.039 --> 00:44:44.480
This is not something which I consider.
00:44:44.960 --> 00:45:01.280
If you have real thing in here which would argue is maybe universal across this class of uh application, if you use a technique or argue something we haven't seen before, you have something new to argue.
00:45:01.360 --> 00:45:03.199
Okay, look, you've seen this now.
00:45:03.280 --> 00:45:09.280
I show you scalability, you've seen this, now I show you on-device lateral movement, you've seen this, and I'll show you a new property.
00:45:09.360 --> 00:45:11.039
You haven't, this is what I look for.
00:45:11.199 --> 00:45:14.159
These are the things that, in my opinion, make a great blank talk.
00:45:14.239 --> 00:45:18.960
And then again, I would argue that it's not easy to get accept into this specific track.
00:45:19.119 --> 00:45:25.599
Uh, this track underrepresents the number of submissions we have.
00:45:25.679 --> 00:45:34.719
So just by selecting this track, the AI track, you are about to be to handle uh much a much higher bar in terms of the acceptance rate.
00:45:34.800 --> 00:45:46.559
You will get lower acceptance rate just by selecting the AI track, where you can maybe optimize your uh acception uh rate with uh selecting different uh tracks.
00:45:46.880 --> 00:46:02.079
In many cases, we are unable to really understand what the speaker tried to communicate because uh maybe didn't do such a great job in communicating it to us.
00:46:02.320 --> 00:46:06.800
In some cases, other factors also taken into consideration whether it's exclusive or not.
00:46:07.039 --> 00:46:12.559
There are a lot of things being considered, not all of them are the pure quality of your findings.
00:46:12.639 --> 00:46:16.719
That these things are also come as a surprise to people when I speak with them.
00:46:16.960 --> 00:46:25.840
So if you ask me, the fact that you haven't been accepted to Blackhead doesn't necessarily mean anything about the novelty of your research.
00:46:26.000 --> 00:46:37.280
It may say a lot of things about the way that you present stuff, about your ability to convince in your findings, about whether it was exclusive or not, about a lot of things within this uh domain.
00:46:37.679 --> 00:46:44.239
And what do you think about so you've you've delivered multiple multiple talks, very successful, very like very good talks, highly rated.
00:46:44.800 --> 00:46:50.000
What is your advice for uh somebody who's presenting their talks for the first time?
00:46:50.159 --> 00:46:57.599
I think one of the it's almost like it's a my take is that it's it's almost like a separate piece of work.
00:46:58.000 --> 00:47:09.199
One piece of work is your research, and that is like you can you can do it, you can like you put your hoodie on, you go to your cocoon, you go out in six months, you have something cool.
00:47:09.440 --> 00:47:12.719
But then telling the world about it is completely different.
00:47:12.960 --> 00:47:13.519
Yeah, I agree.
00:47:13.679 --> 00:47:14.079
I agree.
00:47:14.159 --> 00:47:14.400
Okay.
00:47:14.639 --> 00:47:40.159
There are, I would argue that I've seen many kinds of talks, and there is not like you know, clear thing, what is uh he did something completely different than another talk that I did, and I still thought that this is an amazing talk, and delivered uh, for example, I've seen a talk in the recent Black At Europe delivered by a speaker whose face was covered with a mask, and this was by far the best talk I've seen at the Blackhead Europe last year.
00:47:40.320 --> 00:47:41.519
This is the trailer of B talk.
00:47:41.679 --> 00:47:48.960
Yes, one of the greatest talks that's uh Black, the recent Black Hat Europe, it was on uh December 25th, uh about a year ago.
00:47:49.119 --> 00:48:02.400
I would argue this if you will analyze what we consider as good Blackhead talk, usually they have video demonstrations, okay, without the audience value good video demonstrations.
00:48:02.480 --> 00:48:15.440
They usually they they apply, they they applaud to it, they uh they they laugh out of it, they they they love to see this kind of uh systems being hacked, okay, and they love to see it visually, not just like you know.
00:48:15.679 --> 00:48:19.840
I think that they highly value technical talks.
00:48:19.920 --> 00:48:26.079
And and bear in mind, not just the audience, we as board members also value technical talks.
00:48:26.159 --> 00:48:31.599
I don't like talks given from 50,000 feet away.
00:48:31.920 --> 00:48:38.000
I tr I try to understand and to see, like, you know, the code or the exploitation.
00:48:38.239 --> 00:48:46.960
Additional good talk should convince that this is not a, I would argue, a poor security implementation.
00:48:47.199 --> 00:48:48.960
This is something very fundamental.
00:48:49.119 --> 00:48:58.800
Because if you see something very fundamental in front of your eyes, then all of a sudden, whether the speaker is good or not is not even makes a difference because you will discuss about it with your colleagues.
00:48:58.880 --> 00:48:59.519
I've seen something.
00:48:59.679 --> 00:49:00.239
Did you see it?
00:49:00.320 --> 00:49:02.000
It applies to this, this, this, this, and this.
00:49:02.239 --> 00:49:03.679
This is something that we must take care of.
00:49:03.840 --> 00:49:04.880
And you will repeat seeing it.
00:49:05.039 --> 00:49:13.039
I would argue that the best talk that I've seen at Blackhead was Charlie's and Chris, Chris's talk on hacking the Gypsy Rocky.
00:49:13.280 --> 00:49:22.559
I keep seeing this uh talk once a year, just to see how they were able to communicate their findings with the audience at Blackhead at DEF CON.
00:49:22.880 --> 00:49:37.599
You can see how the audience really loves the way that they did it together, which is something even more risky and more dangerous at uh Blackhead, presented as a peer, not uh each one of them took a different uh part of the talk.
00:49:37.920 --> 00:49:39.679
So I think I'm gonna be.
00:49:48.239 --> 00:49:49.599
Yeah, definitely great talk.
00:49:49.760 --> 00:50:07.920
The thing is, since I'm working for the last seven years a lot with the car industry, I started to see this talk around I think 2016 before I understood about Canvas security and about connected car security and some other stuff related to perception security and some other stuff.
00:50:08.079 --> 00:50:14.800
And every time I've seen it for the last, I think, eight years or so, something like this, I learned something new from their talk.
00:50:15.119 --> 00:50:20.960
Because I became more expert in this field and I then see oh, they did this and they did this and they did this and they did this.
00:50:21.119 --> 00:50:30.480
You have a lot to learn from that from their talk on how to do excellent research, on how to communicate the excellent research to the uh audience.
00:50:30.800 --> 00:50:34.639
The best by far, DEF CON and Blackest Talk that I ever seen by far.
00:50:34.960 --> 00:50:36.159
That's really, really cool.
00:50:36.400 --> 00:50:41.119
So um I want to end with um you have a conference coming up, right?
00:50:41.280 --> 00:50:43.519
The real world AI security conference.
00:50:43.599 --> 00:50:46.079
Uh and and you're part of the team behind it.
00:50:46.239 --> 00:50:49.519
Can you share a bit uh what is this conference about?
00:50:49.679 --> 00:50:54.400
Why why did what what is the need you saw uh for this kind of conference to take place?
00:50:54.639 --> 00:51:04.480
I think that today the specific conferences that you've just mentioned, among them Black Hat, RSA, and DEF CON, are first of all well too industrial.
00:51:04.639 --> 00:51:08.880
They don't attract the experts in the field of AI security.
00:51:09.599 --> 00:51:18.320
And uh in many cases, they are not even intended, some of them, not all of them, are not even intended for experts in this field.
00:51:18.559 --> 00:51:27.760
For example, Black Hat and RSA are mostly intended for executives, uh, whether they are you see a lot of, for example, product managers.
00:51:27.920 --> 00:51:29.599
What does a product manager have to do?
00:51:29.679 --> 00:51:37.119
Uh or people from sales coming for to this uh, and uh you you see CEOs and you definitely see CTOs as well, right?
00:51:37.360 --> 00:51:47.039
DEF CON is more about the core InfoSec hackers community, not exactly the AI security expert that you see in this field.
00:51:47.360 --> 00:51:58.960
We do see a lot of new other one-day or two-day summits, such as the one that you have at Zenity and such as the one that recently done by Gnostic by uh Gadi.
00:51:59.280 --> 00:52:04.239
I have one advantage is somebody worked in the area of crypto for a while.
00:52:04.639 --> 00:52:12.960
I went to a conference named Real World Crypto, which was initiated by Dan Bonnet, uh Dan Bonnet, professor from Stanford University.
00:52:13.360 --> 00:52:26.159
And when I went through the researcher, they told me, look, this is the best conference that if we need to sense, and again, you are speaking with people that have families and children, so they are unable to you know to attend each and every conference that they have.
00:52:26.320 --> 00:52:28.719
So they told me that this is the first priority.
00:52:28.960 --> 00:52:41.199
They start by when they plan their year ahead, they start by planning that they will attend to this uh conference and whatever else could uh they could attend also is a second priority to them.
00:52:41.440 --> 00:52:44.079
And they were able to create something very interesting.
00:52:44.239 --> 00:52:48.320
First of all, they went on only on technical talks.
00:52:48.400 --> 00:52:59.280
There is no talk which is not technical in their uh everything is completely new and gathered from the best conferences and by the greatest expert in this field.
00:52:59.519 --> 00:53:08.719
It attracts the greatest researchers and the greatest practitioners and the greatest uh you know experts in this field who work in the area of AI security.
00:53:08.880 --> 00:53:18.880
Since they were able to do it with uh single session, not parallel session, they were able to guarantee that everybody sees everything.
00:53:19.039 --> 00:53:24.480
Okay, otherwise, you have an audience that is being split between different sessions, right?
00:53:24.639 --> 00:53:30.079
If you have three different ballrooms, then the best case scenario, you need to decide what you're about to see.
00:53:31.280 --> 00:53:42.639
And uh not only because of this, because of the also the side events, they were able to enforce collaborations or discussions with the relevant expert in this field.
00:53:42.880 --> 00:53:45.360
And this is the exact experience that I wanted to bring.
00:53:45.519 --> 00:53:57.280
I didn't want to have it peer-reviewed in terms of making it uh because also like you know, um asking for exclusiveness, because otherwise I would not get the best things that I could uh ask.
00:53:57.440 --> 00:54:02.400
I wanted it to be something which is the best content that you can get.
00:54:02.559 --> 00:54:19.519
And as a result, we decided to not sell any spot for companies that wanted us to you know to pay for slots in the conference because they understood the value of presenting their work to the relevant audience of uh expertise.
00:54:19.760 --> 00:54:27.920
We wanted to be like, you know, those who not only academics, but also those came from the industry.
00:54:28.400 --> 00:54:36.320
Uh, become the magnet for them, become something that they will say, okay, if uh you have uh something uh practical, send it uh to uh to them.
00:54:36.480 --> 00:54:40.320
They will uh this is the place to show the things that are practical within this domain.
00:54:40.480 --> 00:54:59.039
Now the academic conferences weren't able to do it because the academic conferences are very all over the place, they'll have a lot of unrelated sessions to machine learning security and AI security, and you don't get a reasonable time to even present your stuff.
00:54:59.280 --> 00:55:04.800
Uh, in an academic conference, at the best case scenario, you will get 10 minutes, maybe 12 minutes.
00:55:05.039 --> 00:55:10.880
In other cases, you get five minutes, in other cases you don't get anything, you just have a poster that presents your stuff.
00:55:11.039 --> 00:55:21.840
So we found that this is a good way to bring together people from the uh universities and people from the industry to discuss together on urgent matters that are uh challenges for all of us.
00:55:22.079 --> 00:55:23.519
And we're looking forward to it.
00:55:23.760 --> 00:55:25.280
We will be happy to have you there.
00:55:25.679 --> 00:55:26.719
The agenda looks amazing.
00:55:26.960 --> 00:55:27.679
Thank you for that.
00:55:28.000 --> 00:55:30.239
Ben, this was great and a lot of fun.
00:55:30.559 --> 00:55:30.880
Thank you.
00:55:31.039 --> 00:55:31.840
Thank you for having me.
00:55:32.079 --> 00:55:32.719
Thank you so much.