WEBVTT
00:00:03.699 --> 00:00:06.240
Welcome to the Azure Security Podcast, where
00:00:06.240 --> 00:00:08.779
we discuss topics relating to security, privacy,
00:00:09.039 --> 00:00:11.480
reliability, and compliance on the Microsoft
00:00:11.480 --> 00:00:15.839
Cloud Platform. Hey everybody, welcome to episode
00:00:15.839 --> 00:00:20.239
124. This week it's just me, Michael. Everyone
00:00:20.239 --> 00:00:23.260
else is busy this week. And our guest this week
00:00:23.260 --> 00:00:26.100
is Raji, who's here to talk to us about the Microsoft
00:00:26.100 --> 00:00:29.219
Security Response Center AI, safety and security.
00:00:29.940 --> 00:00:31.420
This is certainly an area that's going to be
00:00:31.420 --> 00:00:34.799
very interesting. But before we get to Raji,
00:00:34.820 --> 00:00:37.259
I want to just talk about one little news item
00:00:37.259 --> 00:00:39.200
that I have, and I literally only have one news
00:00:39.200 --> 00:00:42.039
item. And that is, in fact, I really do wish
00:00:42.039 --> 00:00:45.200
Sarah was here because this is right up in her
00:00:45.200 --> 00:00:48.600
wheelhouse. That is, we now have in general availability
00:00:48.600 --> 00:00:53.439
deployment safeguards for pod security in AKS,
00:00:53.439 --> 00:00:55.179
or more accurately, the pod security security
00:00:55.179 --> 00:00:58.119
standard. So it turns out, I didn't know this,
00:00:58.219 --> 00:01:01.250
but now I do. There are some pod security standards
00:01:01.250 --> 00:01:03.710
upon the Kubernetes website that have different
00:01:03.710 --> 00:01:06.010
profiles. There's privileged baseline and restricted.
00:01:06.569 --> 00:01:10.730
And there's now abilities built into AKS that
00:01:10.730 --> 00:01:13.709
allow you to deploy these in a consistent way
00:01:13.709 --> 00:01:16.349
and to enforce pod security configurations across
00:01:16.349 --> 00:01:19.510
clusters. I understand the words. I kind of understand
00:01:19.510 --> 00:01:21.790
the security implications of it. But again, I
00:01:21.790 --> 00:01:23.530
really wish Sarah was here to go into a bit more
00:01:23.530 --> 00:01:26.230
detail because it's her thing. All right, that's
00:01:26.230 --> 00:01:29.120
all I have in the news. So now let's turn our
00:01:29.120 --> 00:01:30.719
attention to our guest. As I mentioned, our guest
00:01:30.719 --> 00:01:32.859
this week is Raji, who's here to talk to us about
00:01:32.859 --> 00:01:36.040
MSRC AI safety and security. So Raji, welcome
00:01:36.040 --> 00:01:39.340
to the episode. We'd like to take a moment and
00:01:39.340 --> 00:01:42.099
introduce yourself to our listeners. Absolutely.
00:01:42.180 --> 00:01:45.340
Thank you so much, Michael, for having me. Exciting
00:01:45.340 --> 00:01:48.400
to be here. I've always been a big fan of the
00:01:48.400 --> 00:01:51.519
podcast, so really excited to be here, specifically
00:01:51.519 --> 00:01:54.060
talking about something really near and dear
00:01:54.060 --> 00:01:57.439
to my heart, the MSRC AI safety and security
00:01:57.439 --> 00:02:01.319
team. So basically, my role at MSRC is that I'm
00:02:01.319 --> 00:02:04.680
a principal engineering manager, and I have been
00:02:04.680 --> 00:02:08.860
with Microsoft completing three years in Feb
00:02:08.860 --> 00:02:12.400
of this year. It feels like a long time, but
00:02:12.400 --> 00:02:16.039
I've been here for three years. My team mainly
00:02:16.039 --> 00:02:18.900
is responsible for driving the end -to -end lifecycle
00:02:18.900 --> 00:02:22.680
for vulnerabilities response for Microsoft AI
00:02:22.680 --> 00:02:26.139
services and products. So this is anything to
00:02:26.139 --> 00:02:30.979
do with Copilot, Copilot Studio, Consumer Copilot,
00:02:31.060 --> 00:02:34.500
M365 Copilot, all the different AI interfaces
00:02:34.500 --> 00:02:38.080
you deal with as you are working with Microsoft
00:02:38.080 --> 00:02:42.639
products. So my team actually, we also lead the
00:02:42.639 --> 00:02:45.979
engagement with a global AI research community.
00:02:46.419 --> 00:02:51.659
This spans both academia and industry. And prior
00:02:51.659 --> 00:02:55.740
to Microsoft, I have held other security leadership
00:02:55.740 --> 00:02:58.659
roles and worked in multiple domains of security
00:02:58.659 --> 00:03:04.879
at Apple and Adobe. So yeah, it's fascinating
00:03:04.879 --> 00:03:06.860
for me because I've seen the transition to the
00:03:06.860 --> 00:03:10.719
cloud, multi -cloud, and now AI and AI agents.
00:03:11.099 --> 00:03:14.719
I just want to end by saying with great connectivity
00:03:14.719 --> 00:03:19.460
comes great risk. I would say with the ambiguity
00:03:19.460 --> 00:03:22.240
of an LLM comes great risk, but let's just leave
00:03:22.240 --> 00:03:24.990
it at that for the moment. I think it's great
00:03:24.990 --> 00:03:27.530
to see a specialized branch of the Microsoft
00:03:27.530 --> 00:03:30.870
Security Response Center coming into focus. Obviously,
00:03:30.889 --> 00:03:33.370
it is an area that needs to be treated in its
00:03:33.370 --> 00:03:36.189
own way. So maybe worthwhile just spending a
00:03:36.189 --> 00:03:37.330
couple of minutes and just talking about the
00:03:37.330 --> 00:03:39.310
difference between the Microsoft Security Response
00:03:39.310 --> 00:03:44.219
Center, the classic MSRC, and MSRC AI. Has anybody
00:03:44.219 --> 00:03:45.900
been here three years? I've been here just a
00:03:45.900 --> 00:03:48.199
little bit longer than that. Yeah, 33 years.
00:03:48.340 --> 00:03:51.400
But anyway, that's another discussion. But yeah,
00:03:51.479 --> 00:03:53.840
so I was there at the beginning with the MSRC.
00:03:53.979 --> 00:03:57.620
I remember the MSRC first forming. People like
00:03:57.620 --> 00:03:59.500
Steve Lipner and Scott Culp will remember some
00:03:59.500 --> 00:04:01.620
of those names who started the MSRC back in the
00:04:01.620 --> 00:04:04.719
day. And obviously the big focus back then was
00:04:04.719 --> 00:04:06.539
on literally security vulnerabilities in products
00:04:06.539 --> 00:04:09.879
like Windows and Office and SQL Server and Exchange
00:04:09.879 --> 00:04:13.430
and Visual Studio and so on and so forth. So,
00:04:13.430 --> 00:04:17.149
I mean, is this like an adjunct to the MSRC?
00:04:17.370 --> 00:04:21.029
Is it part of the MSRC? How does that all fit?
00:04:21.410 --> 00:04:24.829
Yeah, great question. And it always fascinates
00:04:24.829 --> 00:04:28.470
me, you know, people I meet either at Microsoft
00:04:28.470 --> 00:04:30.730
or across the industry. There's like so many
00:04:30.730 --> 00:04:33.490
people just across different companies right
00:04:33.490 --> 00:04:36.050
now who all started their careers at MSRC. So
00:04:36.050 --> 00:04:39.750
it's really fascinating to hear the stories and
00:04:39.750 --> 00:04:43.100
kind of, you know, the... period for which this
00:04:43.100 --> 00:04:47.519
specific part of Microsoft has been, I would
00:04:47.519 --> 00:04:52.360
say, continuing to grow and bring together the
00:04:52.360 --> 00:04:54.740
nuance of coordinated vulnerability disclosure,
00:04:55.120 --> 00:04:57.519
specifically when it comes to external researchers.
00:04:59.149 --> 00:05:01.269
Just fascinated by that. Specifically to your
00:05:01.269 --> 00:05:05.389
question about, you know, AI versus MSRC AI versus
00:05:05.389 --> 00:05:09.790
MSRC. Yes, MSRC AI is part of the MSRC. We have
00:05:09.790 --> 00:05:15.449
two... big groups I would say. One is the security
00:05:15.449 --> 00:05:19.110
assurance engineers who are basically working
00:05:19.110 --> 00:05:23.649
on triaging the bugs which come in. We also look
00:05:23.649 --> 00:05:28.069
at ensuring that it is transferred to the right
00:05:28.069 --> 00:05:31.449
teams within Microsoft and get the right level
00:05:31.449 --> 00:05:35.860
of visibility. Then we also obviously go through
00:05:35.860 --> 00:05:39.779
and ensure that this is in line with our AI bug
00:05:39.779 --> 00:05:42.360
bar. We can talk a little bit about that in just
00:05:42.360 --> 00:05:47.459
a little bit. But then we assess those bugs to
00:05:47.459 --> 00:05:53.920
ensure what kind of severity it is in the space.
00:05:54.180 --> 00:05:56.199
And so that's basically what this team has been
00:05:56.199 --> 00:05:59.060
doing. And we've also built a relationship with
00:05:59.060 --> 00:06:02.680
external researchers, specifically who are doing
00:06:02.680 --> 00:06:06.810
research. in AI safety and security. Some of
00:06:06.810 --> 00:06:10.550
the names are Johan Rehberger. He has published
00:06:10.550 --> 00:06:13.550
the article on Embrace the Red. He has a great
00:06:13.550 --> 00:06:18.110
blog on that. And Zenity, we worked quite closely
00:06:18.110 --> 00:06:21.410
with Michael Bargeri and his team. So those are
00:06:21.410 --> 00:06:23.730
some of the well -known researchers, but we also
00:06:23.730 --> 00:06:28.250
have others we've worked on with them. Yeah,
00:06:28.310 --> 00:06:29.990
I think the thing that makes it interesting is
00:06:29.990 --> 00:06:32.509
that it's such a... brand new area, relatively
00:06:32.509 --> 00:06:34.670
speaking. I mean, security vulnerabilities, SQL
00:06:34.670 --> 00:06:37.769
injection, memory corruption, et cetera, et cetera,
00:06:37.910 --> 00:06:41.470
have been around forever. AI -specific vulnerabilities
00:06:41.470 --> 00:06:44.610
are relatively new in the overall scheme of things.
00:06:44.689 --> 00:06:47.870
However, in many cases, what is old is new again.
00:06:48.029 --> 00:06:49.370
I mean, if you look at prompt injection, you
00:06:49.370 --> 00:06:53.600
could argue it's just input validation. Just
00:06:53.600 --> 00:06:56.740
on steroids, though. Yeah, exactly. Well, you
00:06:56.740 --> 00:06:58.540
know, as you and I were talking about earlier,
00:06:58.660 --> 00:07:01.060
you know, there's a security mantra, which is,
00:07:01.120 --> 00:07:03.459
you know, thou shalt not mix the data plane with
00:07:03.459 --> 00:07:06.199
the control plane. And yet with prompts, that
00:07:06.199 --> 00:07:10.100
is precisely what we're doing, right? Yeah. Absolutely.
00:07:10.360 --> 00:07:14.220
And that leads to the different types of attack
00:07:14.220 --> 00:07:18.279
surfaces, as well as almost sneaky attacks in
00:07:18.279 --> 00:07:21.980
some ways. I mean, we all know, everybody who's
00:07:21.980 --> 00:07:25.120
listening knows about AI and generative AI and
00:07:25.120 --> 00:07:29.120
the impact this has had on our lives from 2023,
00:07:29.519 --> 00:07:33.660
at least, since ChatGPT has been released. There's
00:07:33.660 --> 00:07:40.180
a complete era booming. Everybody, businesses
00:07:40.180 --> 00:07:45.120
are wanting to adapt and obviously it's the right
00:07:45.120 --> 00:07:48.360
thing to do because it brings so much additional
00:07:48.360 --> 00:07:54.819
goodness and efficiencies. But like I said before,
00:07:55.100 --> 00:07:58.279
as the goodness comes along, there's always some
00:07:58.279 --> 00:08:02.420
risk involved. And here we have expanded the
00:08:02.420 --> 00:08:06.720
threat surface and the threat landscape in completely
00:08:06.720 --> 00:08:09.600
new areas. Yeah. I mean, at the end of the day,
00:08:09.620 --> 00:08:10.980
I mean, let's be brutally honest. At the end
00:08:10.980 --> 00:08:12.939
of the day, you know, for every use case, there's
00:08:12.939 --> 00:08:16.000
probably N abuse cases where something can be
00:08:16.000 --> 00:08:17.720
used for good. And then all of a sudden, you
00:08:17.720 --> 00:08:19.740
know, well, if you do it this way, it can actually
00:08:19.740 --> 00:08:22.939
be used for bad. Yeah. So just going back to
00:08:22.939 --> 00:08:27.939
the MSRC versus MSRC AI. I mean, so the MSRC
00:08:27.939 --> 00:08:30.899
handles classic air quotes, classic security
00:08:30.899 --> 00:08:35.360
vulnerabilities and the MSRC AI handles. safety
00:08:35.360 --> 00:08:39.259
-related vulnerabilities. And security, AI security
00:08:39.259 --> 00:08:42.200
vulnerabilities as well, yeah. Yeah. So a classic
00:08:42.200 --> 00:08:46.120
bug is a security bug, but it might not be a
00:08:46.120 --> 00:08:50.720
safety bug necessarily, right? Yes, it may not
00:08:50.720 --> 00:08:53.720
be. So the word safety, if you think about where
00:08:53.720 --> 00:08:56.340
it actually came from, right, it actually came
00:08:56.340 --> 00:09:01.080
from ML. When you go back to before generative
00:09:01.080 --> 00:09:04.220
AI, when a model was inputting something which
00:09:04.220 --> 00:09:08.500
was not in line with what they wanted, then that's
00:09:08.500 --> 00:09:10.519
when you would call it a safety. It's a safety
00:09:10.519 --> 00:09:12.899
issue. It's not aligning. It was like an alignment
00:09:12.899 --> 00:09:16.940
problem. But in those cases, the model developer
00:09:16.940 --> 00:09:21.399
and the model user was the same person. But that's
00:09:21.399 --> 00:09:24.320
not the case anymore with generative AI. The
00:09:24.320 --> 00:09:26.639
model creator is completely different and the
00:09:26.639 --> 00:09:30.659
user. obviously uses it in different ways. And
00:09:30.659 --> 00:09:33.440
so that's why this has become a bigger challenge
00:09:33.440 --> 00:09:36.019
right now. It's not so much misalignment because
00:09:36.019 --> 00:09:39.500
the misalignment ultimately ends up with socio
00:09:39.500 --> 00:09:43.759
-technical issues, which just makes the category
00:09:43.759 --> 00:09:49.200
of AI risks just a bigger blob. It's not just
00:09:49.200 --> 00:09:52.259
confidentiality, integrity, availability. It's
00:09:52.259 --> 00:09:55.809
also some other... Some other things, like we've
00:09:55.809 --> 00:09:59.129
actually published that on the AI Bug Bar. I
00:09:59.129 --> 00:10:01.309
can share the link with you, Michael, so you
00:10:01.309 --> 00:10:05.250
can add it to when you publish the podcast. It's
00:10:05.250 --> 00:10:08.990
basically, we have content -related issues, and
00:10:08.990 --> 00:10:13.629
that actually spans a myriad of different types
00:10:13.629 --> 00:10:17.789
of safety issues. Some of them are information
00:10:17.789 --> 00:10:21.230
integrity, where AI has generated content which
00:10:21.230 --> 00:10:26.539
is false or misleading. Then it could be inappropriate
00:10:26.539 --> 00:10:32.440
language or information integrity. It could be
00:10:32.440 --> 00:10:36.100
things about malicious users. It could also be
00:10:36.100 --> 00:10:40.879
CBRN, a harmful use of CBRN to demonstrate information
00:10:40.879 --> 00:10:44.240
which shows information which is not readily
00:10:44.240 --> 00:10:48.559
available. By the way, CBRN actually stands for
00:10:48.559 --> 00:10:51.960
chemical, biological, radiological, or nuclear.
00:10:52.320 --> 00:10:58.440
So this is a category of when the model produces
00:10:58.440 --> 00:11:03.019
outputs, which are in these categories of chemical,
00:11:03.220 --> 00:11:07.240
biological, radiological, nuclear related. category
00:11:07.240 --> 00:11:12.379
then we say hey it's it's the category cbrn but
00:11:12.379 --> 00:11:16.559
there is more nuance to specifically categorizing
00:11:16.559 --> 00:11:19.120
at cbrn that it should not be readily available
00:11:19.120 --> 00:11:22.460
publicly like if someone can get the same information
00:11:22.460 --> 00:11:25.580
in a google search or a bing search then that
00:11:25.580 --> 00:11:29.399
is not brand new we are actually looking for
00:11:29.399 --> 00:11:32.100
uplift what uplift means is that what is the
00:11:32.100 --> 00:11:35.929
model providing completely new which the user
00:11:35.929 --> 00:11:39.710
will not be able to get it otherwise. Yeah. But
00:11:39.710 --> 00:11:40.990
then you've got other topics, right? You've got
00:11:40.990 --> 00:11:45.509
like sexual content, physical harm. Yeah, self
00:11:45.509 --> 00:11:49.450
-harm. Yeah. So these are all like the new categories,
00:11:49.509 --> 00:11:53.769
what we have seen with AI safety. This is new
00:11:53.769 --> 00:11:58.429
overall in the industry, right? And we are looking
00:11:58.429 --> 00:12:01.610
into this because to ensure that we are thinking
00:12:01.610 --> 00:12:04.049
about risks more holistically when it comes to
00:12:04.049 --> 00:12:06.590
AI. So what are some other examples? I mean,
00:12:06.610 --> 00:12:13.549
that's image manipulation. What about any other
00:12:13.549 --> 00:12:16.450
examples? I think this is an area where a lot
00:12:16.450 --> 00:12:18.070
of people, it's brand new to a lot of people,
00:12:18.129 --> 00:12:21.250
and it's an example of stuff you don't even know
00:12:21.250 --> 00:12:24.490
you don't know. So any more sort of context you
00:12:24.490 --> 00:12:25.970
can give around that I think will be really useful
00:12:25.970 --> 00:12:29.210
to our listeners. Yeah, some of the safety issues
00:12:29.210 --> 00:12:35.399
we have seen is folks sending us Actually, even
00:12:35.399 --> 00:12:39.220
with indirect prompt injection, the way we categorize
00:12:39.220 --> 00:12:42.759
something as a security issue is as long as it
00:12:42.759 --> 00:12:48.460
has a security impact. So if the specific security
00:12:48.460 --> 00:12:51.399
impact could be something like a data exfiltration.
00:12:51.850 --> 00:12:55.549
If you can showcase in your bug, as you're sending
00:12:55.549 --> 00:12:58.850
us a bug, if you showcase that, hey, it has a
00:12:58.850 --> 00:13:01.750
security impact, that means I'm able to exfiltrate
00:13:01.750 --> 00:13:05.830
data, then we categorize it as a security issue.
00:13:06.230 --> 00:13:10.149
And in a safety issue is anything where you can
00:13:10.149 --> 00:13:12.889
show indirect prompt injection, but you're not
00:13:12.889 --> 00:13:18.289
able to demonstrate. data exfiltration, for example,
00:13:18.470 --> 00:13:21.029
then we call that a safety issue. So that's one
00:13:21.029 --> 00:13:23.590
way, one of the things we have seen. The other
00:13:23.590 --> 00:13:27.129
stuff we've also, other kind of issues we have
00:13:27.129 --> 00:13:32.269
seen is people just manipulating pictures and
00:13:32.269 --> 00:13:36.389
sending us you know, like different types of
00:13:36.389 --> 00:13:39.309
images with some political figures and things
00:13:39.309 --> 00:13:42.230
like that. And those have been safety issues
00:13:42.230 --> 00:13:45.269
as well. And most of the time it is because a
00:13:45.269 --> 00:13:47.750
classifier, we need to go update a classifier.
00:13:48.190 --> 00:13:51.110
So you mentioned very early on that there is
00:13:51.110 --> 00:13:54.710
an MSRC AI bug bar. Can you give us examples
00:13:54.710 --> 00:13:58.049
of, actually before we do that, why don't I explain
00:13:58.049 --> 00:14:00.649
what a bug bar is at Microsoft and then you can
00:14:00.649 --> 00:14:03.539
sort of chime in with. what the AI bug bar kind
00:14:03.539 --> 00:14:07.159
of looks like. So the bug bar at Microsoft has
00:14:07.159 --> 00:14:08.799
been around for a long, long, long time, and
00:14:08.799 --> 00:14:11.700
it's a way of categorizing overall risk slash
00:14:11.700 --> 00:14:15.899
severity of a vulnerability. So for example,
00:14:16.039 --> 00:14:20.759
a remote code execution exploit that is easy
00:14:20.759 --> 00:14:23.639
to pull off remotely and gives you system on
00:14:23.639 --> 00:14:26.409
a Windows box. That's more than likely critical.
00:14:26.750 --> 00:14:29.330
Now, there may be a lot of extenuating circumstances
00:14:29.330 --> 00:14:31.529
that reduce the potential damage. Perhaps it's
00:14:31.529 --> 00:14:33.850
only a local -only attack. Perhaps it doesn't
00:14:33.850 --> 00:14:38.789
give you a system. Perhaps it's on a feature
00:14:38.789 --> 00:14:42.149
that's disabled by default. All these other things
00:14:42.149 --> 00:14:43.649
and other defenses come into play, and that may
00:14:43.649 --> 00:14:46.789
be rated a low or a moderate. I'm just making
00:14:46.789 --> 00:14:49.929
that up. So the bug bar has been around at Microsoft
00:14:49.929 --> 00:14:52.429
for a long, long, long time. So what does the
00:14:52.429 --> 00:14:57.809
bug bar look like for MSRC AI? Yeah, it's a great
00:14:57.809 --> 00:15:02.690
question. And it's actually exactly what you
00:15:02.690 --> 00:15:06.370
talked about, Michael. It is basically a way
00:15:06.370 --> 00:15:09.409
of categorizing the different vulnerabilities
00:15:09.409 --> 00:15:12.769
we are seeing in the space of AI, both security
00:15:12.769 --> 00:15:17.649
as well as safety, and then talking about severity,
00:15:18.009 --> 00:15:21.570
specifically when it comes to... safety, there
00:15:21.570 --> 00:15:26.590
is no severity. It's more focused on in -scope,
00:15:26.590 --> 00:15:30.570
out -of -scope, because this is basically, we
00:15:30.570 --> 00:15:33.610
are trying to identify whether it's, what is
00:15:33.610 --> 00:15:37.629
the safety issue? And there is nothing about
00:15:37.629 --> 00:15:40.129
in -scope or out -of -scope. Sorry, nothing about
00:15:40.129 --> 00:15:44.009
specific severity because of this emerging space
00:15:44.009 --> 00:15:47.700
we are talking about. But when it comes to security,
00:15:48.179 --> 00:15:53.620
it is very similar to other MSRC bug bars, specifically
00:15:53.620 --> 00:15:57.759
inference manipulation. We talk here, some of
00:15:57.759 --> 00:16:00.120
the things are prompt injection. The indirect
00:16:00.120 --> 00:16:04.440
prompt injection has been the largest category
00:16:04.440 --> 00:16:08.789
of bugs we have seen. and if it's a zero click
00:16:08.789 --> 00:16:13.230
attack like basically if uh without a user performing
00:16:13.230 --> 00:16:18.450
any any action uh then you know uh and the data
00:16:18.450 --> 00:16:22.029
is exfiltrated through uh through a prompt then
00:16:22.029 --> 00:16:26.389
we call that a critical a severity bug and if
00:16:26.389 --> 00:16:30.269
we need a specific interaction that means somebody
00:16:30.269 --> 00:16:34.169
needs to click a link and then that allows to
00:16:34.169 --> 00:16:37.570
exfiltrate data then we call that an important
00:16:37.570 --> 00:16:42.409
severity bug so that's kind of how we've been
00:16:42.409 --> 00:16:47.379
categorizing the the ai vulnerabilities We've
00:16:47.379 --> 00:16:50.879
also called out input perturbation, which is
00:16:50.879 --> 00:16:55.620
basically any time a specific classification
00:16:55.620 --> 00:17:02.500
model is misclassified or, you know, those are
00:17:02.500 --> 00:17:04.859
what we call input perturbation. But we haven't
00:17:04.859 --> 00:17:09.940
really seen these kind of bugs submitted to us.
00:17:11.579 --> 00:17:14.339
Mostly it is in the space of indirect prompt
00:17:14.339 --> 00:17:19.130
injection. And indirect prompt injection is where
00:17:19.130 --> 00:17:23.210
you embed essentially a prompt injection -like
00:17:23.210 --> 00:17:26.609
attack in a document that is read by the LLM.
00:17:26.730 --> 00:17:30.269
Yes, yes, that is indirect prompt injection,
00:17:30.369 --> 00:17:35.029
or it's basically the top LLM vulnerability,
00:17:35.369 --> 00:17:40.630
even in OWASP. top 10. This is basically where
00:17:40.630 --> 00:17:45.809
the attacker is able to say, you know, summarize
00:17:45.809 --> 00:17:49.569
an email, and then the email actually has the
00:17:49.569 --> 00:17:52.650
prompt injection embedded in it. So when the
00:17:52.650 --> 00:17:56.450
user who's interacting with the LLM says, hey,
00:17:56.569 --> 00:18:01.049
summarize my email, the prompt injection automatically
00:18:01.049 --> 00:18:03.490
gets triggered. So I just realized while we're
00:18:03.490 --> 00:18:05.809
talking about prompt injection and indirect prompt
00:18:05.809 --> 00:18:08.319
injection, not everyone will probably understand
00:18:08.319 --> 00:18:10.019
what a prompt injection attack is. Do you want
00:18:10.019 --> 00:18:14.099
to just give a quick example? Yeah. So, for example,
00:18:14.339 --> 00:18:18.480
let's assume somebody sends you an email. The
00:18:18.480 --> 00:18:22.559
attacker can send you an email with just a normal
00:18:22.559 --> 00:18:25.299
email in your inbox. And as part of the email,
00:18:25.480 --> 00:18:31.779
there is a line that says, send this data, send
00:18:31.779 --> 00:18:38.309
sensitive data to attacker. website or attacker
00:18:38.309 --> 00:18:43.130
email right and the user who is just on their
00:18:43.130 --> 00:18:46.730
emails uh using copilot might say hey summarize
00:18:46.730 --> 00:18:51.069
my email and when they do summarize my email
00:18:51.069 --> 00:18:58.329
in our llm it behaves differently right it interprets
00:18:59.789 --> 00:19:04.910
a prompt or an instruction inside an email like
00:19:04.910 --> 00:19:08.730
a prompt, and LLM, as useful as it wants it to
00:19:08.730 --> 00:19:12.630
be, basically takes that prompt and summarizes
00:19:12.630 --> 00:19:17.289
it and basically sets forward that action. What
00:19:17.289 --> 00:19:22.329
it does is that the user is basically just summarizing
00:19:22.329 --> 00:19:25.970
the email, but in the background, the LLM is
00:19:25.970 --> 00:19:29.200
already sent. some sensitive information to the
00:19:29.200 --> 00:19:33.900
attacker email or to whoever is basically sent
00:19:33.900 --> 00:19:35.960
that prompt injection to them so that's why it's
00:19:35.960 --> 00:19:39.259
indirect prompt injection the llm doesn't differentiate
00:19:39.259 --> 00:19:42.519
between the instructions coming from the user
00:19:42.519 --> 00:19:46.740
directly interacting with the llm or or instructions
00:19:46.740 --> 00:19:52.119
embedded in documents it is leveraging to summarize
00:19:52.119 --> 00:19:55.940
your instructions hopefully that's helpful I
00:19:55.940 --> 00:19:58.140
think if people listening to this podcast don't
00:19:58.140 --> 00:19:59.880
know what prompt injection is and indirect prompt
00:19:59.880 --> 00:20:03.819
injection is, this is as classic. It will end
00:20:03.819 --> 00:20:05.859
up being as classic as SQL injection, memory
00:20:05.859 --> 00:20:08.460
safety issues, cross -site scripting. Absolutely.
00:20:08.980 --> 00:20:11.619
So everyone needs to be aware of what they are,
00:20:11.700 --> 00:20:13.000
especially if you're building your own tools
00:20:13.000 --> 00:20:17.920
on top of perhaps your own LLMs or third -party
00:20:17.920 --> 00:20:19.799
LLMs or something. I think it's just critically
00:20:19.799 --> 00:20:22.079
important that you understand what these... these
00:20:22.079 --> 00:20:23.900
vulnerability classes are. In fact, most importantly,
00:20:24.319 --> 00:20:26.339
what the mitigations are. And I think it's fair
00:20:26.339 --> 00:20:29.259
to say that this whole area is constantly evolving.
00:20:30.599 --> 00:20:33.380
The thing that sort of concerns me is if we take
00:20:33.380 --> 00:20:35.720
SQL injection as an example, we know how to fix
00:20:35.720 --> 00:20:38.460
that. We know precisely how to fix that. You
00:20:38.460 --> 00:20:41.019
just use parameterized queries, basically, for
00:20:41.019 --> 00:20:45.119
any untrusted data. The problem solved. The problem
00:20:45.119 --> 00:20:49.340
with LLM prompt injection is there's no... single
00:20:49.340 --> 00:20:52.740
mitigation that fixes the problem. Because as
00:20:52.740 --> 00:20:56.480
you noted, we're sort of mixing control constructs
00:20:56.480 --> 00:20:59.480
with data constructs, and we don't know which
00:20:59.480 --> 00:21:01.940
is which. So it becomes very, very difficult.
00:21:02.119 --> 00:21:08.140
Yeah, absolutely. And I think ultimately, I think
00:21:08.140 --> 00:21:12.130
security basics... apply whether it is ai or
00:21:12.130 --> 00:21:15.069
non -ai so i would say like continue to really
00:21:15.069 --> 00:21:18.410
think about defense and depth and defense and
00:21:18.410 --> 00:21:21.250
depth is no longer optional it's actually really
00:21:21.250 --> 00:21:26.309
important and crucial for uh specifically mitigating
00:21:26.309 --> 00:21:28.589
something like indirect prompt injection you
00:21:28.589 --> 00:21:31.630
need to be thinking about hey what is my content
00:21:31.630 --> 00:21:35.539
security policy And you need to have multiple
00:21:35.539 --> 00:21:38.160
layers of defense. So it's almost like the Swiss
00:21:38.160 --> 00:21:40.859
cheese analogy of like one, okay, it doesn't
00:21:40.859 --> 00:21:44.180
catch here, does it catch the next one? And that
00:21:44.180 --> 00:21:46.740
is the best way to mitigate these. Yeah, don't
00:21:46.740 --> 00:21:48.160
forget least privilege. Like if you're running
00:21:48.160 --> 00:21:51.420
AI agents, if you have an agent that runs with
00:21:51.420 --> 00:21:56.029
godlike permissions over your environment. and
00:21:56.029 --> 00:21:58.210
the LLM decides to start doing its own thing,
00:21:58.349 --> 00:22:00.750
then you can start doing all sorts of damage.
00:22:01.210 --> 00:22:05.250
So yeah, least privilege is storm. It's just
00:22:05.250 --> 00:22:07.450
interesting to me that we talked about least
00:22:07.450 --> 00:22:10.750
privilege since the 60s and 70s, and here we
00:22:10.750 --> 00:22:12.970
are again talking about least privilege with
00:22:12.970 --> 00:22:17.029
a brand new type of technology, which is generative
00:22:17.029 --> 00:22:20.259
AI. Yes. And I think there's another aspect of
00:22:20.259 --> 00:22:23.900
this as well, which is basically, you know, we
00:22:23.900 --> 00:22:27.099
have to be prepared for what can go wrong. I
00:22:27.099 --> 00:22:30.180
mean, and this is like classic security mindset
00:22:30.180 --> 00:22:33.039
of threat modeling, right? Like, well, what can
00:22:33.039 --> 00:22:34.779
go wrong? I mean, like, this is how we should
00:22:34.779 --> 00:22:37.880
be thinking about our systems. So really, really
00:22:37.880 --> 00:22:42.059
digging deep into secure by design and threat
00:22:42.059 --> 00:22:45.319
modeling right in the beginning, right, as you're
00:22:45.319 --> 00:22:47.359
getting this. as you're thinking about ideas
00:22:47.359 --> 00:22:51.660
and kind of coming together as a security team,
00:22:51.779 --> 00:22:55.960
product team, you know, like and working together
00:22:55.960 --> 00:22:57.960
to ensure that we are meeting customer needs
00:22:57.960 --> 00:23:00.240
from a customer perspective, but also really
00:23:00.240 --> 00:23:02.839
thinking about what is the how can we continue
00:23:02.839 --> 00:23:06.240
to keep provide that safety and security aspect
00:23:06.240 --> 00:23:10.079
of things as well as, you know, really focusing
00:23:10.079 --> 00:23:14.319
on or like almost like. assuming things will
00:23:14.319 --> 00:23:18.279
go wrong and uh preparing for what should you
00:23:18.279 --> 00:23:20.420
do when things go wrong because it's too late
00:23:20.420 --> 00:23:23.380
to figure out things at that point so at least
00:23:23.380 --> 00:23:26.720
if you have like um here is my team i will be
00:23:26.720 --> 00:23:30.039
who will be looking into these or um you know
00:23:30.039 --> 00:23:33.519
how do you pull into incident response or what
00:23:33.519 --> 00:23:35.859
are the new terminologies in incident response
00:23:37.579 --> 00:23:40.500
Some of those things is kind of keeping in mind
00:23:40.500 --> 00:23:46.359
as you're thinking about AI systems. Yeah, and
00:23:46.359 --> 00:23:49.339
that's why I think it's so cool that Microsoft
00:23:49.339 --> 00:23:52.180
has now set up a Microsoft Security Response
00:23:52.180 --> 00:23:55.099
Center division or part of the MSRC just to focus
00:23:55.099 --> 00:23:58.660
on AI. This is a constantly evolving area and
00:23:58.660 --> 00:24:01.680
response is a critical part of responding to
00:24:01.680 --> 00:24:03.500
AI vulnerability. So this is really good to see.
00:24:05.130 --> 00:24:10.130
Yeah, absolutely. And it also helps to continue
00:24:10.130 --> 00:24:13.039
to think about it from a... you know we have
00:24:13.039 --> 00:24:15.759
a lot of processes internally when it comes to
00:24:15.759 --> 00:24:18.440
pre -deployment right like before deployment
00:24:18.440 --> 00:24:21.559
like a specific uh you know there are a lot of
00:24:21.559 --> 00:24:24.240
different controls including a deployment safety
00:24:24.240 --> 00:24:28.200
board which actually stamps the approval before
00:24:28.200 --> 00:24:30.579
something actually goes out the door from microsoft
00:24:30.579 --> 00:24:34.599
from an ai perspective and then on the other
00:24:34.599 --> 00:24:38.359
side msrc is working with these researchers who
00:24:38.359 --> 00:24:42.900
are seeing the bugs in production right and that
00:24:42.900 --> 00:24:45.779
kind of provides that feedback loop to continue
00:24:45.779 --> 00:24:50.339
to better our defenses and as well as think about
00:24:50.339 --> 00:24:52.900
things in angles which we had not thought about
00:24:52.900 --> 00:24:54.839
ourselves so different perspectives bringing
00:24:54.839 --> 00:24:57.200
in completely new different perspectives yeah
00:24:57.200 --> 00:24:58.660
and i think that's something that's really important
00:24:58.660 --> 00:25:01.339
as well as you know serious issues, critical
00:25:01.339 --> 00:25:03.460
issues go through a complete root cause analysis
00:25:03.460 --> 00:25:06.599
to find out what happened and what we can learn
00:25:06.599 --> 00:25:09.319
from this. And I think that virtuous cycle is
00:25:09.319 --> 00:25:11.420
critically important. So again, it's good to
00:25:11.420 --> 00:25:13.359
see that we're doing this in the AI space as
00:25:13.359 --> 00:25:16.660
well. All right, Raji. So one question we always
00:25:16.660 --> 00:25:19.759
ask our guests is, what does a typical day in
00:25:19.759 --> 00:25:23.299
the life look like for Raji? I feel like the
00:25:23.299 --> 00:25:25.700
answer is... for most of your guests must be
00:25:25.700 --> 00:25:29.740
there is no typical day in security that doesn't,
00:25:29.740 --> 00:25:33.180
it doesn't really, it's very different every
00:25:33.180 --> 00:25:35.549
day. And I think that keeps me kind of. excited
00:25:35.549 --> 00:25:39.009
and motivated but if i have to really think about
00:25:39.009 --> 00:25:41.150
hey what does my typical day look like it looks
00:25:41.150 --> 00:25:45.630
uh it basically is working with uh both external
00:25:45.630 --> 00:25:49.309
folks as well as internal teams and continuing
00:25:49.309 --> 00:25:52.950
to uh kind of bring together some of these perspectives
00:25:52.950 --> 00:25:57.190
of new vulnerabilities coming in and helping
00:25:57.190 --> 00:26:00.170
our internal teams really think about it from
00:26:00.170 --> 00:26:02.890
uh you know how does that fit into their threat
00:26:02.890 --> 00:26:05.869
model and how do they mitigate that, as well
00:26:05.869 --> 00:26:13.809
as how do they take the feedback from these different
00:26:13.809 --> 00:26:19.309
research to basically better their mitigations
00:26:19.309 --> 00:26:23.289
in the future, as well as working with our...
00:26:23.480 --> 00:26:26.839
bug bounty teams uh because we also you know
00:26:26.839 --> 00:26:29.319
some of the some of the all of the work we are
00:26:29.319 --> 00:26:33.339
working here uh in ai security specifically have
00:26:33.339 --> 00:26:37.940
uh have uh rewards attached to it so we've uh
00:26:37.940 --> 00:26:43.029
and recently in black hat eu Microsoft basically,
00:26:43.130 --> 00:26:46.369
MSRC basically announced a standard award policy,
00:26:46.549 --> 00:26:49.109
which basically says that you don't have to be
00:26:49.109 --> 00:26:52.990
in scope anymore. There is no in scope. But as
00:26:52.990 --> 00:26:55.990
long as you have a critical or an important severity
00:26:55.990 --> 00:26:59.009
bug, then you get paid a standard award, which
00:26:59.009 --> 00:27:01.609
is really cool. Yeah, that is cool. So those
00:27:01.609 --> 00:27:04.190
are some of the things I do work with. Very cool.
00:27:04.670 --> 00:27:07.049
All right. So let's start to bring this episode
00:27:07.049 --> 00:27:09.730
to an end. So again, another question for you.
00:27:10.349 --> 00:27:13.190
If you had one piece of advice to leave our listeners
00:27:13.190 --> 00:27:15.849
with, what would it be? One final thought. Yeah,
00:27:15.910 --> 00:27:19.950
one final thought is really, you know, the Microsoft
00:27:19.950 --> 00:27:23.549
AI security and safety team is always looking
00:27:23.549 --> 00:27:26.910
for AI security researchers or safety researchers,
00:27:27.190 --> 00:27:33.289
you know, to work with. And if you find any bugs
00:27:33.289 --> 00:27:37.529
in Microsoft AI security products or services,
00:27:37.609 --> 00:27:40.769
please reach out to the MSRC at the researcher
00:27:40.769 --> 00:27:45.049
portal. And you can submit the box to us. And
00:27:45.049 --> 00:27:47.849
we would love to continue to engage with you
00:27:47.849 --> 00:27:51.849
and collaborate with you to ensure that we can
00:27:51.849 --> 00:27:55.710
make Microsoft AI products and services better
00:27:55.710 --> 00:27:59.750
for our customers. Very cool. All right. So,
00:27:59.750 --> 00:28:02.569
Raji, thank you for joining us this week. I know
00:28:02.569 --> 00:28:03.849
you're very busy, and that's a very interesting
00:28:03.849 --> 00:28:06.190
area of research. So it's great to see we have
00:28:06.190 --> 00:28:09.210
people like you working in this area. And I always
00:28:09.210 --> 00:28:12.480
learn something new. Again, I'm a classic security
00:28:12.480 --> 00:28:15.420
person, but over the last couple of years, I've
00:28:15.420 --> 00:28:17.880
also learned a great deal about AI security and
00:28:17.880 --> 00:28:20.859
safety. And again, it's amazing. The things you
00:28:20.859 --> 00:28:23.019
don't know, you don't know. So again, thank you
00:28:23.019 --> 00:28:25.019
for joining us this week. And to all our listeners
00:28:25.019 --> 00:28:26.880
out there, we hope you found this episode useful.
00:28:27.220 --> 00:28:29.400
If you're in the AI space, you really need to
00:28:29.400 --> 00:28:32.019
understand AI safety and security, especially
00:28:32.019 --> 00:28:36.079
if you're building products. on top of AI. So
00:28:36.079 --> 00:28:38.039
stay safe, and we'll see you next time.
00:00:03.699 --> 00:00:06.240
Welcome to the Azure Security Podcast, where
00:00:06.240 --> 00:00:08.779
we discuss topics relating to security, privacy,
00:00:09.039 --> 00:00:11.480
reliability, and compliance on the Microsoft
00:00:11.480 --> 00:00:15.839
Cloud Platform. Hey everybody, welcome to episode
00:00:15.839 --> 00:00:20.239
124. This week it's just me, Michael. Everyone
00:00:20.239 --> 00:00:23.260
else is busy this week. And our guest this week
00:00:23.260 --> 00:00:26.100
is Raji, who's here to talk to us about the Microsoft
00:00:26.100 --> 00:00:29.219
Security Response Center AI, safety and security.
00:00:29.940 --> 00:00:31.420
This is certainly an area that's going to be
00:00:31.420 --> 00:00:34.799
very interesting. But before we get to Raji,
00:00:34.820 --> 00:00:37.259
I want to just talk about one little news item
00:00:37.259 --> 00:00:39.200
that I have, and I literally only have one news
00:00:39.200 --> 00:00:42.039
item. And that is, in fact, I really do wish
00:00:42.039 --> 00:00:45.200
Sarah was here because this is right up in her
00:00:45.200 --> 00:00:48.600
wheelhouse. That is, we now have in general availability
00:00:48.600 --> 00:00:53.439
deployment safeguards for pod security in AKS,
00:00:53.439 --> 00:00:55.179
or more accurately, the pod security security
00:00:55.179 --> 00:00:58.119
standard. So it turns out, I didn't know this,
00:00:58.219 --> 00:01:01.250
but now I do. There are some pod security standards
00:01:01.250 --> 00:01:03.710
upon the Kubernetes website that have different
00:01:03.710 --> 00:01:06.010
profiles. There's privileged baseline and restricted.
00:01:06.569 --> 00:01:10.730
And there's now abilities built into AKS that
00:01:10.730 --> 00:01:13.709
allow you to deploy these in a consistent way
00:01:13.709 --> 00:01:16.349
and to enforce pod security configurations across
00:01:16.349 --> 00:01:19.510
clusters. I understand the words. I kind of understand
00:01:19.510 --> 00:01:21.790
the security implications of it. But again, I
00:01:21.790 --> 00:01:23.530
really wish Sarah was here to go into a bit more
00:01:23.530 --> 00:01:26.230
detail because it's her thing. All right, that's
00:01:26.230 --> 00:01:29.120
all I have in the news. So now let's turn our
00:01:29.120 --> 00:01:30.719
attention to our guest. As I mentioned, our guest
00:01:30.719 --> 00:01:32.859
this week is Raji, who's here to talk to us about
00:01:32.859 --> 00:01:36.040
MSRC AI safety and security. So Raji, welcome
00:01:36.040 --> 00:01:39.340
to the episode. We'd like to take a moment and
00:01:39.340 --> 00:01:42.099
introduce yourself to our listeners. Absolutely.
00:01:42.180 --> 00:01:45.340
Thank you so much, Michael, for having me. Exciting
00:01:45.340 --> 00:01:48.400
to be here. I've always been a big fan of the
00:01:48.400 --> 00:01:51.519
podcast, so really excited to be here, specifically
00:01:51.519 --> 00:01:54.060
talking about something really near and dear
00:01:54.060 --> 00:01:57.439
to my heart, the MSRC AI safety and security
00:01:57.439 --> 00:02:01.319
team. So basically, my role at MSRC is that I'm
00:02:01.319 --> 00:02:04.680
a principal engineering manager, and I have been
00:02:04.680 --> 00:02:08.860
with Microsoft completing three years in Feb
00:02:08.860 --> 00:02:12.400
of this year. It feels like a long time, but
00:02:12.400 --> 00:02:16.039
I've been here for three years. My team mainly
00:02:16.039 --> 00:02:18.900
is responsible for driving the end -to -end lifecycle
00:02:18.900 --> 00:02:22.680
for vulnerabilities response for Microsoft AI
00:02:22.680 --> 00:02:26.139
services and products. So this is anything to
00:02:26.139 --> 00:02:30.979
do with Copilot, Copilot Studio, Consumer Copilot,
00:02:31.060 --> 00:02:34.500
M365 Copilot, all the different AI interfaces
00:02:34.500 --> 00:02:38.080
you deal with as you are working with Microsoft
00:02:38.080 --> 00:02:42.639
products. So my team actually, we also lead the
00:02:42.639 --> 00:02:45.979
engagement with a global AI research community.
00:02:46.419 --> 00:02:51.659
This spans both academia and industry. And prior
00:02:51.659 --> 00:02:55.740
to Microsoft, I have held other security leadership
00:02:55.740 --> 00:02:58.659
roles and worked in multiple domains of security
00:02:58.659 --> 00:03:04.879
at Apple and Adobe. So yeah, it's fascinating
00:03:04.879 --> 00:03:06.860
for me because I've seen the transition to the
00:03:06.860 --> 00:03:10.719
cloud, multi -cloud, and now AI and AI agents.
00:03:11.099 --> 00:03:14.719
I just want to end by saying with great connectivity
00:03:14.719 --> 00:03:19.460
comes great risk. I would say with the ambiguity
00:03:19.460 --> 00:03:22.240
of an LLM comes great risk, but let's just leave
00:03:22.240 --> 00:03:24.990
it at that for the moment. I think it's great
00:03:24.990 --> 00:03:27.530
to see a specialized branch of the Microsoft
00:03:27.530 --> 00:03:30.870
Security Response Center coming into focus. Obviously,
00:03:30.889 --> 00:03:33.370
it is an area that needs to be treated in its
00:03:33.370 --> 00:03:36.189
own way. So maybe worthwhile just spending a
00:03:36.189 --> 00:03:37.330
couple of minutes and just talking about the
00:03:37.330 --> 00:03:39.310
difference between the Microsoft Security Response
00:03:39.310 --> 00:03:44.219
Center, the classic MSRC, and MSRC AI. Has anybody
00:03:44.219 --> 00:03:45.900
been here three years? I've been here just a
00:03:45.900 --> 00:03:48.199
little bit longer than that. Yeah, 33 years.
00:03:48.340 --> 00:03:51.400
But anyway, that's another discussion. But yeah,
00:03:51.479 --> 00:03:53.840
so I was there at the beginning with the MSRC.
00:03:53.979 --> 00:03:57.620
I remember the MSRC first forming. People like
00:03:57.620 --> 00:03:59.500
Steve Lipner and Scott Culp will remember some
00:03:59.500 --> 00:04:01.620
of those names who started the MSRC back in the
00:04:01.620 --> 00:04:04.719
day. And obviously the big focus back then was
00:04:04.719 --> 00:04:06.539
on literally security vulnerabilities in products
00:04:06.539 --> 00:04:09.879
like Windows and Office and SQL Server and Exchange
00:04:09.879 --> 00:04:13.430
and Visual Studio and so on and so forth. So,
00:04:13.430 --> 00:04:17.149
I mean, is this like an adjunct to the MSRC?
00:04:17.370 --> 00:04:21.029
Is it part of the MSRC? How does that all fit?
00:04:21.410 --> 00:04:24.829
Yeah, great question. And it always fascinates
00:04:24.829 --> 00:04:28.470
me, you know, people I meet either at Microsoft
00:04:28.470 --> 00:04:30.730
or across the industry. There's like so many
00:04:30.730 --> 00:04:33.490
people just across different companies right
00:04:33.490 --> 00:04:36.050
now who all started their careers at MSRC. So
00:04:36.050 --> 00:04:39.750
it's really fascinating to hear the stories and
00:04:39.750 --> 00:04:43.100
kind of, you know, the... period for which this
00:04:43.100 --> 00:04:47.519
specific part of Microsoft has been, I would
00:04:47.519 --> 00:04:52.360
say, continuing to grow and bring together the
00:04:52.360 --> 00:04:54.740
nuance of coordinated vulnerability disclosure,
00:04:55.120 --> 00:04:57.519
specifically when it comes to external researchers.
00:04:59.149 --> 00:05:01.269
Just fascinated by that. Specifically to your
00:05:01.269 --> 00:05:05.389
question about, you know, AI versus MSRC AI versus
00:05:05.389 --> 00:05:09.790
MSRC. Yes, MSRC AI is part of the MSRC. We have
00:05:09.790 --> 00:05:15.449
two... big groups I would say. One is the security
00:05:15.449 --> 00:05:19.110
assurance engineers who are basically working
00:05:19.110 --> 00:05:23.649
on triaging the bugs which come in. We also look
00:05:23.649 --> 00:05:28.069
at ensuring that it is transferred to the right
00:05:28.069 --> 00:05:31.449
teams within Microsoft and get the right level
00:05:31.449 --> 00:05:35.860
of visibility. Then we also obviously go through
00:05:35.860 --> 00:05:39.779
and ensure that this is in line with our AI bug
00:05:39.779 --> 00:05:42.360
bar. We can talk a little bit about that in just
00:05:42.360 --> 00:05:47.459
a little bit. But then we assess those bugs to
00:05:47.459 --> 00:05:53.920
ensure what kind of severity it is in the space.
00:05:54.180 --> 00:05:56.199
And so that's basically what this team has been
00:05:56.199 --> 00:05:59.060
doing. And we've also built a relationship with
00:05:59.060 --> 00:06:02.680
external researchers, specifically who are doing
00:06:02.680 --> 00:06:06.810
research. in AI safety and security. Some of
00:06:06.810 --> 00:06:10.550
the names are Johan Rehberger. He has published
00:06:10.550 --> 00:06:13.550
the article on Embrace the Red. He has a great
00:06:13.550 --> 00:06:18.110
blog on that. And Zenity, we worked quite closely
00:06:18.110 --> 00:06:21.410
with Michael Bargeri and his team. So those are
00:06:21.410 --> 00:06:23.730
some of the well -known researchers, but we also
00:06:23.730 --> 00:06:28.250
have others we've worked on with them. Yeah,
00:06:28.310 --> 00:06:29.990
I think the thing that makes it interesting is
00:06:29.990 --> 00:06:32.509
that it's such a... brand new area, relatively
00:06:32.509 --> 00:06:34.670
speaking. I mean, security vulnerabilities, SQL
00:06:34.670 --> 00:06:37.769
injection, memory corruption, et cetera, et cetera,
00:06:37.910 --> 00:06:41.470
have been around forever. AI -specific vulnerabilities
00:06:41.470 --> 00:06:44.610
are relatively new in the overall scheme of things.
00:06:44.689 --> 00:06:47.870
However, in many cases, what is old is new again.
00:06:48.029 --> 00:06:49.370
I mean, if you look at prompt injection, you
00:06:49.370 --> 00:06:53.600
could argue it's just input validation. Just
00:06:53.600 --> 00:06:56.740
on steroids, though. Yeah, exactly. Well, you
00:06:56.740 --> 00:06:58.540
know, as you and I were talking about earlier,
00:06:58.660 --> 00:07:01.060
you know, there's a security mantra, which is,
00:07:01.120 --> 00:07:03.459
you know, thou shalt not mix the data plane with
00:07:03.459 --> 00:07:06.199
the control plane. And yet with prompts, that
00:07:06.199 --> 00:07:10.100
is precisely what we're doing, right? Yeah. Absolutely.
00:07:10.360 --> 00:07:14.220
And that leads to the different types of attack
00:07:14.220 --> 00:07:18.279
surfaces, as well as almost sneaky attacks in
00:07:18.279 --> 00:07:21.980
some ways. I mean, we all know, everybody who's
00:07:21.980 --> 00:07:25.120
listening knows about AI and generative AI and
00:07:25.120 --> 00:07:29.120
the impact this has had on our lives from 2023,
00:07:29.519 --> 00:07:33.660
at least, since ChatGPT has been released. There's
00:07:33.660 --> 00:07:40.180
a complete era booming. Everybody, businesses
00:07:40.180 --> 00:07:45.120
are wanting to adapt and obviously it's the right
00:07:45.120 --> 00:07:48.360
thing to do because it brings so much additional
00:07:48.360 --> 00:07:54.819
goodness and efficiencies. But like I said before,
00:07:55.100 --> 00:07:58.279
as the goodness comes along, there's always some
00:07:58.279 --> 00:08:02.420
risk involved. And here we have expanded the
00:08:02.420 --> 00:08:06.720
threat surface and the threat landscape in completely
00:08:06.720 --> 00:08:09.600
new areas. Yeah. I mean, at the end of the day,
00:08:09.620 --> 00:08:10.980
I mean, let's be brutally honest. At the end
00:08:10.980 --> 00:08:12.939
of the day, you know, for every use case, there's
00:08:12.939 --> 00:08:16.000
probably N abuse cases where something can be
00:08:16.000 --> 00:08:17.720
used for good. And then all of a sudden, you
00:08:17.720 --> 00:08:19.740
know, well, if you do it this way, it can actually
00:08:19.740 --> 00:08:22.939
be used for bad. Yeah. So just going back to
00:08:22.939 --> 00:08:27.939
the MSRC versus MSRC AI. I mean, so the MSRC
00:08:27.939 --> 00:08:30.899
handles classic air quotes, classic security
00:08:30.899 --> 00:08:35.360
vulnerabilities and the MSRC AI handles. safety
00:08:35.360 --> 00:08:39.259
-related vulnerabilities. And security, AI security
00:08:39.259 --> 00:08:42.200
vulnerabilities as well, yeah. Yeah. So a classic
00:08:42.200 --> 00:08:46.120
bug is a security bug, but it might not be a
00:08:46.120 --> 00:08:50.720
safety bug necessarily, right? Yes, it may not
00:08:50.720 --> 00:08:53.720
be. So the word safety, if you think about where
00:08:53.720 --> 00:08:56.340
it actually came from, right, it actually came
00:08:56.340 --> 00:09:01.080
from ML. When you go back to before generative
00:09:01.080 --> 00:09:04.220
AI, when a model was inputting something which
00:09:04.220 --> 00:09:08.500
was not in line with what they wanted, then that's
00:09:08.500 --> 00:09:10.519
when you would call it a safety. It's a safety
00:09:10.519 --> 00:09:12.899
issue. It's not aligning. It was like an alignment
00:09:12.899 --> 00:09:16.940
problem. But in those cases, the model developer
00:09:16.940 --> 00:09:21.399
and the model user was the same person. But that's
00:09:21.399 --> 00:09:24.320
not the case anymore with generative AI. The
00:09:24.320 --> 00:09:26.639
model creator is completely different and the
00:09:26.639 --> 00:09:30.659
user. obviously uses it in different ways. And
00:09:30.659 --> 00:09:33.440
so that's why this has become a bigger challenge
00:09:33.440 --> 00:09:36.019
right now. It's not so much misalignment because
00:09:36.019 --> 00:09:39.500
the misalignment ultimately ends up with socio
00:09:39.500 --> 00:09:43.759
-technical issues, which just makes the category
00:09:43.759 --> 00:09:49.200
of AI risks just a bigger blob. It's not just
00:09:49.200 --> 00:09:52.259
confidentiality, integrity, availability. It's
00:09:52.259 --> 00:09:55.809
also some other... Some other things, like we've
00:09:55.809 --> 00:09:59.129
actually published that on the AI Bug Bar. I
00:09:59.129 --> 00:10:01.309
can share the link with you, Michael, so you
00:10:01.309 --> 00:10:05.250
can add it to when you publish the podcast. It's
00:10:05.250 --> 00:10:08.990
basically, we have content -related issues, and
00:10:08.990 --> 00:10:13.629
that actually spans a myriad of different types
00:10:13.629 --> 00:10:17.789
of safety issues. Some of them are information
00:10:17.789 --> 00:10:21.230
integrity, where AI has generated content which
00:10:21.230 --> 00:10:26.539
is false or misleading. Then it could be inappropriate
00:10:26.539 --> 00:10:32.440
language or information integrity. It could be
00:10:32.440 --> 00:10:36.100
things about malicious users. It could also be
00:10:36.100 --> 00:10:40.879
CBRN, a harmful use of CBRN to demonstrate information
00:10:40.879 --> 00:10:44.240
which shows information which is not readily
00:10:44.240 --> 00:10:48.559
available. By the way, CBRN actually stands for
00:10:48.559 --> 00:10:51.960
chemical, biological, radiological, or nuclear.
00:10:52.320 --> 00:10:58.440
So this is a category of when the model produces
00:10:58.440 --> 00:11:03.019
outputs, which are in these categories of chemical,
00:11:03.220 --> 00:11:07.240
biological, radiological, nuclear related. category
00:11:07.240 --> 00:11:12.379
then we say hey it's it's the category cbrn but
00:11:12.379 --> 00:11:16.559
there is more nuance to specifically categorizing
00:11:16.559 --> 00:11:19.120
at cbrn that it should not be readily available
00:11:19.120 --> 00:11:22.460
publicly like if someone can get the same information
00:11:22.460 --> 00:11:25.580
in a google search or a bing search then that
00:11:25.580 --> 00:11:29.399
is not brand new we are actually looking for
00:11:29.399 --> 00:11:32.100
uplift what uplift means is that what is the
00:11:32.100 --> 00:11:35.929
model providing completely new which the user
00:11:35.929 --> 00:11:39.710
will not be able to get it otherwise. Yeah. But
00:11:39.710 --> 00:11:40.990
then you've got other topics, right? You've got
00:11:40.990 --> 00:11:45.509
like sexual content, physical harm. Yeah, self
00:11:45.509 --> 00:11:49.450
-harm. Yeah. So these are all like the new categories,
00:11:49.509 --> 00:11:53.769
what we have seen with AI safety. This is new
00:11:53.769 --> 00:11:58.429
overall in the industry, right? And we are looking
00:11:58.429 --> 00:12:01.610
into this because to ensure that we are thinking
00:12:01.610 --> 00:12:04.049
about risks more holistically when it comes to
00:12:04.049 --> 00:12:06.590
AI. So what are some other examples? I mean,
00:12:06.610 --> 00:12:13.549
that's image manipulation. What about any other
00:12:13.549 --> 00:12:16.450
examples? I think this is an area where a lot
00:12:16.450 --> 00:12:18.070
of people, it's brand new to a lot of people,
00:12:18.129 --> 00:12:21.250
and it's an example of stuff you don't even know
00:12:21.250 --> 00:12:24.490
you don't know. So any more sort of context you
00:12:24.490 --> 00:12:25.970
can give around that I think will be really useful
00:12:25.970 --> 00:12:29.210
to our listeners. Yeah, some of the safety issues
00:12:29.210 --> 00:12:35.399
we have seen is folks sending us Actually, even
00:12:35.399 --> 00:12:39.220
with indirect prompt injection, the way we categorize
00:12:39.220 --> 00:12:42.759
something as a security issue is as long as it
00:12:42.759 --> 00:12:48.460
has a security impact. So if the specific security
00:12:48.460 --> 00:12:51.399
impact could be something like a data exfiltration.
00:12:51.850 --> 00:12:55.549
If you can showcase in your bug, as you're sending
00:12:55.549 --> 00:12:58.850
us a bug, if you showcase that, hey, it has a
00:12:58.850 --> 00:13:01.750
security impact, that means I'm able to exfiltrate
00:13:01.750 --> 00:13:05.830
data, then we categorize it as a security issue.
00:13:06.230 --> 00:13:10.149
And in a safety issue is anything where you can
00:13:10.149 --> 00:13:12.889
show indirect prompt injection, but you're not
00:13:12.889 --> 00:13:18.289
able to demonstrate. data exfiltration, for example,
00:13:18.470 --> 00:13:21.029
then we call that a safety issue. So that's one
00:13:21.029 --> 00:13:23.590
way, one of the things we have seen. The other
00:13:23.590 --> 00:13:27.129
stuff we've also, other kind of issues we have
00:13:27.129 --> 00:13:32.269
seen is people just manipulating pictures and
00:13:32.269 --> 00:13:36.389
sending us you know, like different types of
00:13:36.389 --> 00:13:39.309
images with some political figures and things
00:13:39.309 --> 00:13:42.230
like that. And those have been safety issues
00:13:42.230 --> 00:13:45.269
as well. And most of the time it is because a
00:13:45.269 --> 00:13:47.750
classifier, we need to go update a classifier.
00:13:48.190 --> 00:13:51.110
So you mentioned very early on that there is
00:13:51.110 --> 00:13:54.710
an MSRC AI bug bar. Can you give us examples
00:13:54.710 --> 00:13:58.049
of, actually before we do that, why don't I explain
00:13:58.049 --> 00:14:00.649
what a bug bar is at Microsoft and then you can
00:14:00.649 --> 00:14:03.539
sort of chime in with. what the AI bug bar kind
00:14:03.539 --> 00:14:07.159
of looks like. So the bug bar at Microsoft has
00:14:07.159 --> 00:14:08.799
been around for a long, long, long time, and
00:14:08.799 --> 00:14:11.700
it's a way of categorizing overall risk slash
00:14:11.700 --> 00:14:15.899
severity of a vulnerability. So for example,
00:14:16.039 --> 00:14:20.759
a remote code execution exploit that is easy
00:14:20.759 --> 00:14:23.639
to pull off remotely and gives you system on
00:14:23.639 --> 00:14:26.409
a Windows box. That's more than likely critical.
00:14:26.750 --> 00:14:29.330
Now, there may be a lot of extenuating circumstances
00:14:29.330 --> 00:14:31.529
that reduce the potential damage. Perhaps it's
00:14:31.529 --> 00:14:33.850
only a local -only attack. Perhaps it doesn't
00:14:33.850 --> 00:14:38.789
give you a system. Perhaps it's on a feature
00:14:38.789 --> 00:14:42.149
that's disabled by default. All these other things
00:14:42.149 --> 00:14:43.649
and other defenses come into play, and that may
00:14:43.649 --> 00:14:46.789
be rated a low or a moderate. I'm just making
00:14:46.789 --> 00:14:49.929
that up. So the bug bar has been around at Microsoft
00:14:49.929 --> 00:14:52.429
for a long, long, long time. So what does the
00:14:52.429 --> 00:14:57.809
bug bar look like for MSRC AI? Yeah, it's a great
00:14:57.809 --> 00:15:02.690
question. And it's actually exactly what you
00:15:02.690 --> 00:15:06.370
talked about, Michael. It is basically a way
00:15:06.370 --> 00:15:09.409
of categorizing the different vulnerabilities
00:15:09.409 --> 00:15:12.769
we are seeing in the space of AI, both security
00:15:12.769 --> 00:15:17.649
as well as safety, and then talking about severity,
00:15:18.009 --> 00:15:21.570
specifically when it comes to... safety, there
00:15:21.570 --> 00:15:26.590
is no severity. It's more focused on in -scope,
00:15:26.590 --> 00:15:30.570
out -of -scope, because this is basically, we
00:15:30.570 --> 00:15:33.610
are trying to identify whether it's, what is
00:15:33.610 --> 00:15:37.629
the safety issue? And there is nothing about
00:15:37.629 --> 00:15:40.129
in -scope or out -of -scope. Sorry, nothing about
00:15:40.129 --> 00:15:44.009
specific severity because of this emerging space
00:15:44.009 --> 00:15:47.700
we are talking about. But when it comes to security,
00:15:48.179 --> 00:15:53.620
it is very similar to other MSRC bug bars, specifically
00:15:53.620 --> 00:15:57.759
inference manipulation. We talk here, some of
00:15:57.759 --> 00:16:00.120
the things are prompt injection. The indirect
00:16:00.120 --> 00:16:04.440
prompt injection has been the largest category
00:16:04.440 --> 00:16:08.789
of bugs we have seen. and if it's a zero click
00:16:08.789 --> 00:16:13.230
attack like basically if uh without a user performing
00:16:13.230 --> 00:16:18.450
any any action uh then you know uh and the data
00:16:18.450 --> 00:16:22.029
is exfiltrated through uh through a prompt then
00:16:22.029 --> 00:16:26.389
we call that a critical a severity bug and if
00:16:26.389 --> 00:16:30.269
we need a specific interaction that means somebody
00:16:30.269 --> 00:16:34.169
needs to click a link and then that allows to
00:16:34.169 --> 00:16:37.570
exfiltrate data then we call that an important
00:16:37.570 --> 00:16:42.409
severity bug so that's kind of how we've been
00:16:42.409 --> 00:16:47.379
categorizing the the ai vulnerabilities We've
00:16:47.379 --> 00:16:50.879
also called out input perturbation, which is
00:16:50.879 --> 00:16:55.620
basically any time a specific classification
00:16:55.620 --> 00:17:02.500
model is misclassified or, you know, those are
00:17:02.500 --> 00:17:04.859
what we call input perturbation. But we haven't
00:17:04.859 --> 00:17:09.940
really seen these kind of bugs submitted to us.
00:17:11.579 --> 00:17:14.339
Mostly it is in the space of indirect prompt
00:17:14.339 --> 00:17:19.130
injection. And indirect prompt injection is where
00:17:19.130 --> 00:17:23.210
you embed essentially a prompt injection -like
00:17:23.210 --> 00:17:26.609
attack in a document that is read by the LLM.
00:17:26.730 --> 00:17:30.269
Yes, yes, that is indirect prompt injection,
00:17:30.369 --> 00:17:35.029
or it's basically the top LLM vulnerability,
00:17:35.369 --> 00:17:40.630
even in OWASP. top 10. This is basically where
00:17:40.630 --> 00:17:45.809
the attacker is able to say, you know, summarize
00:17:45.809 --> 00:17:49.569
an email, and then the email actually has the
00:17:49.569 --> 00:17:52.650
prompt injection embedded in it. So when the
00:17:52.650 --> 00:17:56.450
user who's interacting with the LLM says, hey,
00:17:56.569 --> 00:18:01.049
summarize my email, the prompt injection automatically
00:18:01.049 --> 00:18:03.490
gets triggered. So I just realized while we're
00:18:03.490 --> 00:18:05.809
talking about prompt injection and indirect prompt
00:18:05.809 --> 00:18:08.319
injection, not everyone will probably understand
00:18:08.319 --> 00:18:10.019
what a prompt injection attack is. Do you want
00:18:10.019 --> 00:18:14.099
to just give a quick example? Yeah. So, for example,
00:18:14.339 --> 00:18:18.480
let's assume somebody sends you an email. The
00:18:18.480 --> 00:18:22.559
attacker can send you an email with just a normal
00:18:22.559 --> 00:18:25.299
email in your inbox. And as part of the email,
00:18:25.480 --> 00:18:31.779
there is a line that says, send this data, send
00:18:31.779 --> 00:18:38.309
sensitive data to attacker. website or attacker
00:18:38.309 --> 00:18:43.130
email right and the user who is just on their
00:18:43.130 --> 00:18:46.730
emails uh using copilot might say hey summarize
00:18:46.730 --> 00:18:51.069
my email and when they do summarize my email
00:18:51.069 --> 00:18:58.329
in our llm it behaves differently right it interprets
00:18:59.789 --> 00:19:04.910
a prompt or an instruction inside an email like
00:19:04.910 --> 00:19:08.730
a prompt, and LLM, as useful as it wants it to
00:19:08.730 --> 00:19:12.630
be, basically takes that prompt and summarizes
00:19:12.630 --> 00:19:17.289
it and basically sets forward that action. What
00:19:17.289 --> 00:19:22.329
it does is that the user is basically just summarizing
00:19:22.329 --> 00:19:25.970
the email, but in the background, the LLM is
00:19:25.970 --> 00:19:29.200
already sent. some sensitive information to the
00:19:29.200 --> 00:19:33.900
attacker email or to whoever is basically sent
00:19:33.900 --> 00:19:35.960
that prompt injection to them so that's why it's
00:19:35.960 --> 00:19:39.259
indirect prompt injection the llm doesn't differentiate
00:19:39.259 --> 00:19:42.519
between the instructions coming from the user
00:19:42.519 --> 00:19:46.740
directly interacting with the llm or or instructions
00:19:46.740 --> 00:19:52.119
embedded in documents it is leveraging to summarize
00:19:52.119 --> 00:19:55.940
your instructions hopefully that's helpful I
00:19:55.940 --> 00:19:58.140
think if people listening to this podcast don't
00:19:58.140 --> 00:19:59.880
know what prompt injection is and indirect prompt
00:19:59.880 --> 00:20:03.819
injection is, this is as classic. It will end
00:20:03.819 --> 00:20:05.859
up being as classic as SQL injection, memory
00:20:05.859 --> 00:20:08.460
safety issues, cross -site scripting. Absolutely.
00:20:08.980 --> 00:20:11.619
So everyone needs to be aware of what they are,
00:20:11.700 --> 00:20:13.000
especially if you're building your own tools
00:20:13.000 --> 00:20:17.920
on top of perhaps your own LLMs or third -party
00:20:17.920 --> 00:20:19.799
LLMs or something. I think it's just critically
00:20:19.799 --> 00:20:22.079
important that you understand what these... these
00:20:22.079 --> 00:20:23.900
vulnerability classes are. In fact, most importantly,
00:20:24.319 --> 00:20:26.339
what the mitigations are. And I think it's fair
00:20:26.339 --> 00:20:29.259
to say that this whole area is constantly evolving.
00:20:30.599 --> 00:20:33.380
The thing that sort of concerns me is if we take
00:20:33.380 --> 00:20:35.720
SQL injection as an example, we know how to fix
00:20:35.720 --> 00:20:38.460
that. We know precisely how to fix that. You
00:20:38.460 --> 00:20:41.019
just use parameterized queries, basically, for
00:20:41.019 --> 00:20:45.119
any untrusted data. The problem solved. The problem
00:20:45.119 --> 00:20:49.340
with LLM prompt injection is there's no... single
00:20:49.340 --> 00:20:52.740
mitigation that fixes the problem. Because as
00:20:52.740 --> 00:20:56.480
you noted, we're sort of mixing control constructs
00:20:56.480 --> 00:20:59.480
with data constructs, and we don't know which
00:20:59.480 --> 00:21:01.940
is which. So it becomes very, very difficult.
00:21:02.119 --> 00:21:08.140
Yeah, absolutely. And I think ultimately, I think
00:21:08.140 --> 00:21:12.130
security basics... apply whether it is ai or
00:21:12.130 --> 00:21:15.069
non -ai so i would say like continue to really
00:21:15.069 --> 00:21:18.410
think about defense and depth and defense and
00:21:18.410 --> 00:21:21.250
depth is no longer optional it's actually really
00:21:21.250 --> 00:21:26.309
important and crucial for uh specifically mitigating
00:21:26.309 --> 00:21:28.589
something like indirect prompt injection you
00:21:28.589 --> 00:21:31.630
need to be thinking about hey what is my content
00:21:31.630 --> 00:21:35.539
security policy And you need to have multiple
00:21:35.539 --> 00:21:38.160
layers of defense. So it's almost like the Swiss
00:21:38.160 --> 00:21:40.859
cheese analogy of like one, okay, it doesn't
00:21:40.859 --> 00:21:44.180
catch here, does it catch the next one? And that
00:21:44.180 --> 00:21:46.740
is the best way to mitigate these. Yeah, don't
00:21:46.740 --> 00:21:48.160
forget least privilege. Like if you're running
00:21:48.160 --> 00:21:51.420
AI agents, if you have an agent that runs with
00:21:51.420 --> 00:21:56.029
godlike permissions over your environment. and
00:21:56.029 --> 00:21:58.210
the LLM decides to start doing its own thing,
00:21:58.349 --> 00:22:00.750
then you can start doing all sorts of damage.
00:22:01.210 --> 00:22:05.250
So yeah, least privilege is storm. It's just
00:22:05.250 --> 00:22:07.450
interesting to me that we talked about least
00:22:07.450 --> 00:22:10.750
privilege since the 60s and 70s, and here we
00:22:10.750 --> 00:22:12.970
are again talking about least privilege with
00:22:12.970 --> 00:22:17.029
a brand new type of technology, which is generative
00:22:17.029 --> 00:22:20.259
AI. Yes. And I think there's another aspect of
00:22:20.259 --> 00:22:23.900
this as well, which is basically, you know, we
00:22:23.900 --> 00:22:27.099
have to be prepared for what can go wrong. I
00:22:27.099 --> 00:22:30.180
mean, and this is like classic security mindset
00:22:30.180 --> 00:22:33.039
of threat modeling, right? Like, well, what can
00:22:33.039 --> 00:22:34.779
go wrong? I mean, like, this is how we should
00:22:34.779 --> 00:22:37.880
be thinking about our systems. So really, really
00:22:37.880 --> 00:22:42.059
digging deep into secure by design and threat
00:22:42.059 --> 00:22:45.319
modeling right in the beginning, right, as you're
00:22:45.319 --> 00:22:47.359
getting this. as you're thinking about ideas
00:22:47.359 --> 00:22:51.660
and kind of coming together as a security team,
00:22:51.779 --> 00:22:55.960
product team, you know, like and working together
00:22:55.960 --> 00:22:57.960
to ensure that we are meeting customer needs
00:22:57.960 --> 00:23:00.240
from a customer perspective, but also really
00:23:00.240 --> 00:23:02.839
thinking about what is the how can we continue
00:23:02.839 --> 00:23:06.240
to keep provide that safety and security aspect
00:23:06.240 --> 00:23:10.079
of things as well as, you know, really focusing
00:23:10.079 --> 00:23:14.319
on or like almost like. assuming things will
00:23:14.319 --> 00:23:18.279
go wrong and uh preparing for what should you
00:23:18.279 --> 00:23:20.420
do when things go wrong because it's too late
00:23:20.420 --> 00:23:23.380
to figure out things at that point so at least
00:23:23.380 --> 00:23:26.720
if you have like um here is my team i will be
00:23:26.720 --> 00:23:30.039
who will be looking into these or um you know
00:23:30.039 --> 00:23:33.519
how do you pull into incident response or what
00:23:33.519 --> 00:23:35.859
are the new terminologies in incident response
00:23:37.579 --> 00:23:40.500
Some of those things is kind of keeping in mind
00:23:40.500 --> 00:23:46.359
as you're thinking about AI systems. Yeah, and
00:23:46.359 --> 00:23:49.339
that's why I think it's so cool that Microsoft
00:23:49.339 --> 00:23:52.180
has now set up a Microsoft Security Response
00:23:52.180 --> 00:23:55.099
Center division or part of the MSRC just to focus
00:23:55.099 --> 00:23:58.660
on AI. This is a constantly evolving area and
00:23:58.660 --> 00:24:01.680
response is a critical part of responding to
00:24:01.680 --> 00:24:03.500
AI vulnerability. So this is really good to see.
00:24:05.130 --> 00:24:10.130
Yeah, absolutely. And it also helps to continue
00:24:10.130 --> 00:24:13.039
to think about it from a... you know we have
00:24:13.039 --> 00:24:15.759
a lot of processes internally when it comes to
00:24:15.759 --> 00:24:18.440
pre -deployment right like before deployment
00:24:18.440 --> 00:24:21.559
like a specific uh you know there are a lot of
00:24:21.559 --> 00:24:24.240
different controls including a deployment safety
00:24:24.240 --> 00:24:28.200
board which actually stamps the approval before
00:24:28.200 --> 00:24:30.579
something actually goes out the door from microsoft
00:24:30.579 --> 00:24:34.599
from an ai perspective and then on the other
00:24:34.599 --> 00:24:38.359
side msrc is working with these researchers who
00:24:38.359 --> 00:24:42.900
are seeing the bugs in production right and that
00:24:42.900 --> 00:24:45.779
kind of provides that feedback loop to continue
00:24:45.779 --> 00:24:50.339
to better our defenses and as well as think about
00:24:50.339 --> 00:24:52.900
things in angles which we had not thought about
00:24:52.900 --> 00:24:54.839
ourselves so different perspectives bringing
00:24:54.839 --> 00:24:57.200
in completely new different perspectives yeah
00:24:57.200 --> 00:24:58.660
and i think that's something that's really important
00:24:58.660 --> 00:25:01.339
as well as you know serious issues, critical
00:25:01.339 --> 00:25:03.460
issues go through a complete root cause analysis
00:25:03.460 --> 00:25:06.599
to find out what happened and what we can learn
00:25:06.599 --> 00:25:09.319
from this. And I think that virtuous cycle is
00:25:09.319 --> 00:25:11.420
critically important. So again, it's good to
00:25:11.420 --> 00:25:13.359
see that we're doing this in the AI space as
00:25:13.359 --> 00:25:16.660
well. All right, Raji. So one question we always
00:25:16.660 --> 00:25:19.759
ask our guests is, what does a typical day in
00:25:19.759 --> 00:25:23.299
the life look like for Raji? I feel like the
00:25:23.299 --> 00:25:25.700
answer is... for most of your guests must be
00:25:25.700 --> 00:25:29.740
there is no typical day in security that doesn't,
00:25:29.740 --> 00:25:33.180
it doesn't really, it's very different every
00:25:33.180 --> 00:25:35.549
day. And I think that keeps me kind of. excited
00:25:35.549 --> 00:25:39.009
and motivated but if i have to really think about
00:25:39.009 --> 00:25:41.150
hey what does my typical day look like it looks
00:25:41.150 --> 00:25:45.630
uh it basically is working with uh both external
00:25:45.630 --> 00:25:49.309
folks as well as internal teams and continuing
00:25:49.309 --> 00:25:52.950
to uh kind of bring together some of these perspectives
00:25:52.950 --> 00:25:57.190
of new vulnerabilities coming in and helping
00:25:57.190 --> 00:26:00.170
our internal teams really think about it from
00:26:00.170 --> 00:26:02.890
uh you know how does that fit into their threat
00:26:02.890 --> 00:26:05.869
model and how do they mitigate that, as well
00:26:05.869 --> 00:26:13.809
as how do they take the feedback from these different
00:26:13.809 --> 00:26:19.309
research to basically better their mitigations
00:26:19.309 --> 00:26:23.289
in the future, as well as working with our...
00:26:23.480 --> 00:26:26.839
bug bounty teams uh because we also you know
00:26:26.839 --> 00:26:29.319
some of the some of the all of the work we are
00:26:29.319 --> 00:26:33.339
working here uh in ai security specifically have
00:26:33.339 --> 00:26:37.940
uh have uh rewards attached to it so we've uh
00:26:37.940 --> 00:26:43.029
and recently in black hat eu Microsoft basically,
00:26:43.130 --> 00:26:46.369
MSRC basically announced a standard award policy,
00:26:46.549 --> 00:26:49.109
which basically says that you don't have to be
00:26:49.109 --> 00:26:52.990
in scope anymore. There is no in scope. But as
00:26:52.990 --> 00:26:55.990
long as you have a critical or an important severity
00:26:55.990 --> 00:26:59.009
bug, then you get paid a standard award, which
00:26:59.009 --> 00:27:01.609
is really cool. Yeah, that is cool. So those
00:27:01.609 --> 00:27:04.190
are some of the things I do work with. Very cool.
00:27:04.670 --> 00:27:07.049
All right. So let's start to bring this episode
00:27:07.049 --> 00:27:09.730
to an end. So again, another question for you.
00:27:10.349 --> 00:27:13.190
If you had one piece of advice to leave our listeners
00:27:13.190 --> 00:27:15.849
with, what would it be? One final thought. Yeah,
00:27:15.910 --> 00:27:19.950
one final thought is really, you know, the Microsoft
00:27:19.950 --> 00:27:23.549
AI security and safety team is always looking
00:27:23.549 --> 00:27:26.910
for AI security researchers or safety researchers,
00:27:27.190 --> 00:27:33.289
you know, to work with. And if you find any bugs
00:27:33.289 --> 00:27:37.529
in Microsoft AI security products or services,
00:27:37.609 --> 00:27:40.769
please reach out to the MSRC at the researcher
00:27:40.769 --> 00:27:45.049
portal. And you can submit the box to us. And
00:27:45.049 --> 00:27:47.849
we would love to continue to engage with you
00:27:47.849 --> 00:27:51.849
and collaborate with you to ensure that we can
00:27:51.849 --> 00:27:55.710
make Microsoft AI products and services better
00:27:55.710 --> 00:27:59.750
for our customers. Very cool. All right. So,
00:27:59.750 --> 00:28:02.569
Raji, thank you for joining us this week. I know
00:28:02.569 --> 00:28:03.849
you're very busy, and that's a very interesting
00:28:03.849 --> 00:28:06.190
area of research. So it's great to see we have
00:28:06.190 --> 00:28:09.210
people like you working in this area. And I always
00:28:09.210 --> 00:28:12.480
learn something new. Again, I'm a classic security
00:28:12.480 --> 00:28:15.420
person, but over the last couple of years, I've
00:28:15.420 --> 00:28:17.880
also learned a great deal about AI security and
00:28:17.880 --> 00:28:20.859
safety. And again, it's amazing. The things you
00:28:20.859 --> 00:28:23.019
don't know, you don't know. So again, thank you
00:28:23.019 --> 00:28:25.019
for joining us this week. And to all our listeners
00:28:25.019 --> 00:28:26.880
out there, we hope you found this episode useful.
00:28:27.220 --> 00:28:29.400
If you're in the AI space, you really need to
00:28:29.400 --> 00:28:32.019
understand AI safety and security, especially
00:28:32.019 --> 00:28:36.079
if you're building products. on top of AI. So
00:28:36.079 --> 00:28:38.039
stay safe, and we'll see you next time.
