WEBVTT
Kind: captions
Language: en-US
00:00:02.405 --> 00:00:03.107
[MUSIC]
00:00:03.107 --> 00:00:06.800
KATHLEEN SULLIVAN: Welcome
to AI Testing and Evaluation:
00:00:06.800 --> 00:00:11.520
Learnings from Science and Industry.
I'm your host, Kathleen Sullivan.
00:00:11.520 --> 00:00:16.640
As generative AI continues to advance, Microsoft
has gathered a range of experts—from genome
00:00:16.640 --> 00:00:21.040
editing to cybersecurity—to share how
their fields approach evaluation and
00:00:21.040 --> 00:00:26.160
risk assessment. Our goal is to learn from
their successes and their stumbles to move
00:00:26.160 --> 00:00:31.360
the science and practice of AI testing
forward. In this series, we'll explore
00:00:31.360 --> 00:00:40.137
how these insights might help guide the future of
AI development, deployment, and responsible use.
00:00:40.126 --> 00:00:40.659
[MUSIC ENDS]
00:00:40.659 --> 00:00:42.800
SULLIVAN: Today, I'm excited to
welcome Dan Carpenter and Timo
00:00:42.800 --> 00:00:46.400
Minssen to the podcast to explore
testing and risk assessment in the
00:00:46.400 --> 00:00:50.160
areas of pharmaceuticals and
medical devices, respectively.
00:00:50.160 --> 00:00:54.640
Dan Carpenter is chair of the Department
of Government at Harvard University. His
00:00:54.640 --> 00:01:00.160
research spans the sphere of social and political
science, from petitioning in democratic society
00:01:00.160 --> 00:01:05.600
to regulation and government organizations.
His recent work includes the FDA Project,
00:01:05.600 --> 00:01:09.440
which examines pharmaceutical
regulation in the United States.
00:01:09.440 --> 00:01:12.720
Timo is a professor of law at
the University of Copenhagen,
00:01:12.720 --> 00:01:18.400
where he is also director of the Center for
Advanced Studies in Bioscience Innovation Law.
00:01:18.400 --> 00:01:22.080
He specializes in legal aspects of
biomedical innovation, including
00:01:22.080 --> 00:01:27.680
intellectual property law and regulatory law.
He's exercised his expertise as an advisor
00:01:27.680 --> 00:01:33.040
to such organizations as the World Health
Organization and the European Commission.
00:01:33.040 --> 00:01:38.000
And after our conversations, we'll talk to
Microsoft's Chad Atalla, an applied scientist
00:01:38.000 --> 00:01:43.520
in responsible AI, about how we should think
about these insights in the context of AI.
00:01:43.520 --> 00:01:46.160
Daniel, it's a pleasure to
welcome you to the podcast.
00:01:46.160 --> 00:01:49.662
I'm just so appreciative of you being
here. Thanks for joining us today.
00:01:49.662 --> 00:01:50.720
DANIEL CARPENTER: Thanks for having me.
00:01:50.720 --> 00:01:56.480
SULLIVAN: Dan, before we dissect policy,
let's rewind the tape to your origin story.
00:01:56.480 --> 00:02:00.960
Can you take us to the moment that you first
became fascinated with regulators rather than,
00:02:00.960 --> 00:02:05.840
say, politicians? Was there a spark
that pulled you toward the FDA story?
00:02:05.840 --> 00:02:08.720
CARPENTER: At one point during graduate school,
00:02:08.720 --> 00:02:16.960
I was studying a combination of American
politics and political theory, and I did a
00:02:16.960 --> 00:02:24.720
summer interning at the Department of Housing
and Urban Development. And I began to think,
00:02:24.720 --> 00:02:30.960
why don't people study these administrators
more and the rules they make, the, you know,
00:02:30.960 --> 00:02:37.360
inefficiencies, the efficiencies? Really more
from, kind of, a descriptive standpoint, less from
00:02:37.360 --> 00:02:45.520
a normative standpoint. And I was reading a lot
that summer about the Food and Drug Administration
00:02:45.520 --> 00:02:51.590
and some of the decisions it was making on
AIDS drugs. That was a, sort of, a major, …
00:02:51.590 --> 00:02:52.800
SULLIVAN: Right.
00:02:52.800 --> 00:02:57.680
CARPENTER: … sort of, you know, moment in the
news, in the global news as well as the national
00:02:57.680 --> 00:03:06.880
news during, I would say, what? The late ’80s,
early ’90s? And so I began to look into that.
00:03:06.880 --> 00:03:09.520
SULLIVAN: So now that we know what pulled you in,
00:03:09.520 --> 00:03:13.120
let’s zoom out for our listeners.
Give us the whirlwind tour. I think
00:03:13.120 --> 00:03:17.480
most of us know pharma involves years of
trials, but what’s the part we don’t know?
00:03:17.480 --> 00:03:24.480
CARPENTER: So I think when most businesses develop
a product, they all go through some phases of
00:03:24.480 --> 00:03:30.320
research and development and testing. And
I think what's different about the FDA is,
00:03:30.960 --> 00:03:32.880
sort of, two- or three-fold.
00:03:32.880 --> 00:03:38.560
First, a lot of those tests are much more
stringently specified and regulated by the
00:03:38.560 --> 00:03:46.960
government, and second, one of the reasons
for that is that the FDA imposes not simply
00:03:46.960 --> 00:03:56.000
safety requirements upon drugs in particular but
also efficacy requirements. The FDA wants you to
00:03:56.000 --> 00:04:02.720
prove not simply that it's safe and non-toxic but
also that it's effective. And the final thing,
00:04:02.720 --> 00:04:07.280
I think, that makes the FDA different is
that it stands as what I would call the “veto
00:04:07.280 --> 00:04:13.200
player” over R&D [research and development]
to the marketplace. The FDA basically has,
00:04:13.200 --> 00:04:16.880
sort of, this control over
entry to the marketplace.
00:04:16.880 --> 00:04:23.600
And so what that involves is usually first,
a set of human trials where people who have
00:04:23.600 --> 00:04:29.120
no disease take it. And you're only looking
for toxicity generally. Then there's a set
00:04:29.120 --> 00:04:34.160
of Phase 2 trials, where they look more
at safety and a little bit at efficacy,
00:04:34.160 --> 00:04:41.440
and you're now examining people who have the
disease that the drug claims to treat. And
00:04:41.440 --> 00:04:47.440
you're also basically comparing people who
get the drug, often with those who do not.
00:04:47.440 --> 00:04:53.520
And then finally, Phase 3 involves a much more
direct and large-scale attack, if you will,
00:04:53.520 --> 00:04:59.760
or assessment of efficacy, and that's where you
get the sort of large randomized clinical trials
00:04:59.760 --> 00:05:08.000
that are very expensive for pharmaceutical
companies, biomedical companies to launch,
00:05:08.000 --> 00:05:16.640
to execute, to analyze. And those are often the
sort of core evidence base for the decisions that
00:05:16.640 --> 00:05:22.240
the FDA makes about whether or not to approve
a new drug for marketing in the United States.
00:05:22.240 --> 00:05:27.600
SULLIVAN: Are there differences
in how that process has, you know,
00:05:27.600 --> 00:05:32.400
changed through other countries and maybe just
how that's evolved as you've seen it play out?
00:05:32.400 --> 00:05:38.320
CARPENTER: Yeah, for a long time, I would say that
the United States had probably the most stringent
00:05:38.320 --> 00:05:44.960
regime of regulation for biopharmaceutical
products until, I would say, about the 1990s
00:05:44.960 --> 00:05:52.080
and early 2000s. It used to be the case that a
number of other countries, especially in Europe
00:05:52.080 --> 00:05:58.720
but around the world, basically waited for the
FDA to mandate tests on a drug and only after
00:05:58.720 --> 00:06:05.760
the drug was approved in the United States would
they deem it approvable and marketable in their
00:06:05.760 --> 00:06:12.880
own countries. And then after the formation of the
European Union and the creation of the European
00:06:12.880 --> 00:06:20.080
Medicines Agency, gradually the European Medicines
Agency began to get a bit more stringent.
00:06:20.080 --> 00:06:24.720
But, you know, over the long run, there's
been a lot of, sort of, heterogeneity,
00:06:24.720 --> 00:06:29.760
a lot of variation over time and space,
in the way that the FDA has approached
00:06:29.760 --> 00:06:35.920
these problems. And I'd say in the last 20
years, it's begun to partially deregulate,
00:06:37.200 --> 00:06:45.440
namely, you know, trying to find all sorts of
mechanisms or pathways for really innovative
00:06:45.440 --> 00:06:52.800
drugs for deadly diseases without a lot
of treatments to basically get through the
00:06:52.800 --> 00:06:58.880
process at lower cost. For many people, that
has not been sufficient. They're concerned
00:06:58.880 --> 00:07:06.400
about the cost of the system. Of course, then
the agency also gets criticized by those who
00:07:06.400 --> 00:07:13.080
believe it's too lax. It is potentially letting
ineffective and unsafe therapies on the market.
00:07:13.080 --> 00:07:16.960
SULLIVAN: In your view, when does the
structured model genuinely safeguard
00:07:16.960 --> 00:07:21.200
patients and where do you think it
maybe slows or limits innovation?
00:07:21.200 --> 00:07:30.320
CARPENTER: So I think the worry is that if you
approach pharmaceutical approval as a world
00:07:30.320 --> 00:07:36.480
where only things can go wrong, then you're
really at a risk of limiting innovation. And
00:07:36.480 --> 00:07:44.000
even if you end up letting a lot of things
through, if by your regulations you end up
00:07:44.000 --> 00:07:48.480
basically slowing down the development
process or making it very, very costly,
00:07:48.480 --> 00:07:54.000
then there's just a whole bunch of drugs that
either come to market too slowly or they come
00:07:54.000 --> 00:08:00.000
to market not at all because they just aren't
worth the kind of cost-benefit or, sort of,
00:08:00.000 --> 00:08:05.760
profit analysis of the firm. You know, so that's
been a concern. And I think it's been one of the
00:08:05.760 --> 00:08:11.120
reasons that the Food and Drug Administration
as well as other world regulators have begun to
00:08:11.120 --> 00:08:18.640
basically try to smooth the process and
accelerate the process at the margins.
00:08:18.640 --> 00:08:24.000
The other thing is that they've started to
basically make approvals on the basis of what
00:08:24.000 --> 00:08:29.600
are called surrogate endpoints. So the idea
is that a cancer drug, we really want to know
00:08:29.600 --> 00:08:36.720
whether that drug saves lives, but if we wait
to see whose lives are saved or prolonged by
00:08:36.720 --> 00:08:41.360
that drug, we might miss the opportunity
to make judgments on the basis of, well,
00:08:41.360 --> 00:08:47.360
are we detecting tumors in the bloodstream? Or
can we measure the size of those tumors in, say,
00:08:47.360 --> 00:08:53.920
a solid cancer? And then the further question
is, is the size of the tumor basically a really
00:08:53.920 --> 00:08:59.040
good correlate or predictor of whether
people will die or not, right? Generally,
00:08:59.040 --> 00:09:07.760
the FDA tends to be less stringent when you've
got, you know, a remarkably innovative new therapy
00:09:07.760 --> 00:09:14.400
and the disease being treated is one that just
doesn't have a lot of available treatments, right.
00:09:14.960 --> 00:09:18.880
The one thing that people often think
about when they're thinking about
00:09:18.880 --> 00:09:23.510
pharmaceutical regulation is they often
contrast, kind of, speed versus safety, …
00:09:23.510 --> 00:09:24.065
SULLIVAN: Right.
00:09:24.065 --> 00:09:28.400
CARPENTER: … right. And that's useful as a
tradeoff, but I often try to remind people
00:09:28.400 --> 00:09:34.400
that it's not simply about whether the drug
gets out there and it's unsafe. You know,
00:09:34.400 --> 00:09:39.520
you and I as patients and even doctors have
a hard time knowing whether something works
00:09:39.520 --> 00:09:44.960
and whether it should be prescribed.
And the evidence for knowing whether
00:09:44.960 --> 00:09:49.440
something works isn't just, well, you
know, Sally took it, or Dan took it,
00:09:49.440 --> 00:09:54.000
or Kathleen took it, and they seem to get
better or they didn't seem to get better.
00:09:54.000 --> 00:09:58.240
The really rigorous evidence comes from
randomized clinical trials. And I think
00:09:58.240 --> 00:10:03.280
it's fair to say that if you didn't
have the FDA there as a veto player,
00:10:03.280 --> 00:10:07.840
you wouldn't get as many randomized clinical
trials and the evidence probably wouldn't be
00:10:07.840 --> 00:10:12.800
as rigorous for whether these things work. And
as I like to put it, basically there's a whole
00:10:12.800 --> 00:10:18.800
ecology of expectations and beliefs around
the biopharmaceutical industry in the United
00:10:18.800 --> 00:10:24.960
States and globally, and to some extent, it's
undergirded by all of these tests that happen.
00:10:24.960 --> 00:10:25.810
SULLIVAN: Right.
00:10:25.810 --> 00:10:31.040
CARPENTER: And in part, that means it's
undergirded by regulation. Would there still be
00:10:31.040 --> 00:10:36.640
a market without regulation? Yes. But it would be
a market in which people had far less information
00:10:36.640 --> 00:10:43.840
in and confidence about the drugs that are being
taken. And so I think it's important to recognize
00:10:43.840 --> 00:10:51.760
that kind of confidence-boosting potential
of, kind of, a scientific regulation base.
00:10:51.760 --> 00:10:55.040
SULLIVAN: Actually, if we could double-click
on that for a minute, I'd love to hear your
00:10:55.040 --> 00:10:59.600
perspective on, testing has been completed;
there's results. Can you walk us through how
00:10:59.600 --> 00:11:05.360
those results actually shape the next steps and
decisions of a particular drug and just, like, how
00:11:05.360 --> 00:11:11.920
regulators actually think about using that data
to influence really what happens next with it?
00:11:11.920 --> 00:11:18.320
CARPENTER: Right. So it's important to
understand that every drug is approved
00:11:18.320 --> 00:11:23.840
for what's called an indication. It
can have a first primary indication,
00:11:23.840 --> 00:11:28.160
which is the main disease that it treats, and
then others can be added as more evidence is
00:11:28.160 --> 00:11:34.640
shown. But a drug is not something that just
kind of exists out there in the ether. It
00:11:34.640 --> 00:11:39.120
has to have the right form of administration.
Maybe it should be injected. Maybe it should be
00:11:39.120 --> 00:11:44.720
ingested. Maybe it should be administered only
at a clinic because it needs to be kind of
00:11:44.720 --> 00:11:50.720
administered in just the right way. As doctors
will tell you, dosage is everything, right.
00:11:50.720 --> 00:11:56.000
And so one of the reasons that you want
those trials is not simply a, you know,
00:11:56.000 --> 00:12:02.560
yes or no answer about whether the drug
works, right. It's not simply if-then. It's
00:12:02.560 --> 00:12:06.320
literally what goes into what you might
call the dose response curve. You know,
00:12:06.320 --> 00:12:11.600
how much of this drug do we need to basically,
you know, get the benefit? At what point does
00:12:11.600 --> 00:12:17.680
that fall off significantly that we can basically
say, we can stop there? All that evidence comes
00:12:17.680 --> 00:12:25.600
from trials. And that's the kind of evidence
that is required on the basis of regulation.
00:12:25.600 --> 00:12:30.080
Because it's not simply a drug
that's approved. It's a drug and
00:12:30.080 --> 00:12:36.000
a frequency of administration. It's a method
of administration. And so the drug isn't just,
00:12:36.000 --> 00:12:39.120
there's something to be taken off the
shelf and popped into your mouth. I mean,
00:12:39.120 --> 00:12:43.360
sometimes that's what happens, but even then,
we want to know what the dosage is, right.
00:12:44.160 --> 00:12:47.760
We want to know what to look for in
terms of side effects, things like that.
00:12:47.760 --> 00:12:49.440
SULLIVAN: Going back to that point, I mean,
00:12:49.440 --> 00:12:55.920
it sounds like we're making a lot of progress
from a regulation perspective in, you know,
00:12:55.920 --> 00:13:01.200
sort of speed and getting things approved but
doing it in a really balanced way. I mean,
00:13:01.200 --> 00:13:05.840
any other kind of closing thoughts on the
tradeoffs there or where you're seeing that going?
00:13:05.840 --> 00:13:10.560
CARPENTER: I think you're going to see some move
in the coming years—there's already been some of
00:13:10.560 --> 00:13:16.400
it—to say, do we always need a really large
Phase 3 clinical trial? And to what degree do
00:13:16.400 --> 00:13:21.040
we need the, like, you know, all the i's dotted
and the t's crossed or a really, really large
00:13:21.040 --> 00:13:29.920
sample size? And I'm open to innovation there.
I'm also open to the idea that we consider, again,
00:13:29.920 --> 00:13:37.520
things like accelerated approvals or pathways
for looking at different kinds of surrogate
00:13:37.520 --> 00:13:42.360
endpoints. I do think, once we do that, then
we also have to have some degree of follow-up.
00:13:42.360 --> 00:13:44.480
SULLIVAN: So I know we're
getting close to out of time,
00:13:44.480 --> 00:13:50.800
but maybe just a quick rapid fire if you’re
open to it. Biggest myth about clinical trials?
00:13:50.800 --> 00:13:54.640
CARPENTER: Well, some people tend to think
that the FDA performs them. You know,
00:13:54.640 --> 00:14:00.160
it's companies that do it. And the only other
thing I would say is the company that does a
00:14:00.160 --> 00:14:05.600
lot of the testing and even the innovating is not
always the company that takes the drug to market,
00:14:05.600 --> 00:14:12.240
and it tells you something about how powerful
regulation is in our system, in our world,
00:14:12.240 --> 00:14:18.640
that you often need a company that has dealt with
the FDA quite a bit and knows all the regulations
00:14:18.640 --> 00:14:23.760
and knows how to dot the i's and cross the t's
in order to get a drug across the finish line.
00:14:23.760 --> 00:14:27.920
SULLIVAN: If you had a magic wand, what's the
one thing you'd change in regulation today?
00:14:27.920 --> 00:14:32.960
CARPENTER: I would like people to think a little
bit less about just speed versus safety and,
00:14:32.960 --> 00:14:37.280
again, more about this basic issue of
confidence. I think it's fundamental
00:14:37.280 --> 00:14:41.200
to everything that happens in markets
but especially in biopharmaceuticals.
00:14:41.200 --> 00:14:45.821
SULLIVAN: Such a great point. This has been really
fun. Just thanks so much for being here today.
00:14:45.821 --> 00:14:46.358
[TRANSITION MUSIC]
00:14:46.358 --> 00:14:49.920
We're really excited to share your
thoughts out to our listeners. Thanks.
00:14:49.920 --> 00:14:54.160
CARPENTER: Likewise.
00:14:58.120 --> 00:14:59.120
SULLIVAN:
00:14:59.120 --> 00:15:04.480
Now to the world of medical devices, I'm joined
by Professor Timo Minssen. Professor Minssen,
00:15:04.480 --> 00:15:07.746
it's great to have you here.
Thank you for joining us today.
00:15:07.746 --> 00:15:09.720
TIMO MINSSEN: Yeah, thank you
very much, it's a pleasure.
00:15:09.720 --> 00:15:13.440
SULLIVAN: Before getting into the
regulatory world of medical devices,
00:15:13.440 --> 00:15:17.440
tell our audience a bit about your personal
journey or your origin story, as we're asking
00:15:17.440 --> 00:15:22.480
our guests. How did you land in regulation,
and what's kept you hooked in this space?
00:15:22.480 --> 00:15:28.160
MINSSEN: So I started out as a patent expert in
the biomedical area, starting with my PhD thesis
00:15:28.160 --> 00:15:34.240
on patenting biologics in Europe and in the US.
So during that time, I was mostly interested in
00:15:34.240 --> 00:15:39.200
patent and trade secret questions. But at the
same time, I also developed and taught courses
00:15:39.200 --> 00:15:45.680
in regulatory law and held talks on regulating
advanced medical therapy medicinal products.
00:15:45.680 --> 00:15:52.720
I then started to lead large research projects
on legal challenges in a wide variety of health
00:15:52.720 --> 00:15:57.840
and life science innovation frontiers. I also
started to focus increasingly on AI-enabled
00:15:57.840 --> 00:16:03.760
medical devices and software as a medical device,
resulting in several academic articles in this
00:16:03.760 --> 00:16:09.800
area and also in the regulatory area and a book
on the future of medical device regulation.
00:16:09.800 --> 00:16:12.800
SULLIVAN: Yeah, what's kept
you hooked in the space?
00:16:12.800 --> 00:16:15.600
MINSSEN: It's just incredibly exciting,
00:16:15.600 --> 00:16:22.880
in particular right now with everything that
is going on, you know, in the software arena,
00:16:22.880 --> 00:16:28.000
in the marriage between AI and medical
devices. And this is really challenging
00:16:28.000 --> 00:16:32.360
not only societies but also regulators
and authorities in Europe and in the US.
00:16:32.360 --> 00:16:35.760
SULLIVAN: Yeah, it's a super exciting
time to be in this space. You know,
00:16:35.760 --> 00:16:39.360
we talked to Daniel a little earlier and,
you know, I think similar to pharmaceuticals,
00:16:39.360 --> 00:16:45.280
people have a general sense of what we mean when
we say medical devices, but most listeners may
00:16:45.280 --> 00:16:52.240
picture like a stethoscope or a hip implant.
The word “medical device” reaches much wider.
00:16:52.240 --> 00:16:57.120
Can you give us a quick, kind of, range from
perhaps very simple to even, I don't know,
00:16:57.120 --> 00:17:03.040
sci-fi and then your 90-second tour of how risk
assessment works and why a framework is essential?
00:17:03.040 --> 00:17:06.800
MINSSEN: Let me start out by saying that the WHO
[World Health Organization] estimates that today
00:17:06.800 --> 00:17:11.840
there are approximately 2 million different
kinds of medical devices on the world market,
00:17:11.840 --> 00:17:19.360
and as of the FDA's latest update that I'm
aware of, the FDA has authorized more than 1,000
00:17:19.360 --> 00:17:24.720
AI-, machine learning-enabled medical
devices, and that number is rising rapidly.
00:17:24.720 --> 00:17:29.040
So in that context, I think it is important
to understand that medical devices can be
00:17:29.040 --> 00:17:35.520
any instrument, apparatus, implement, machine,
appliance, implant, reagent for in vitro use,
00:17:35.520 --> 00:17:42.080
software, material, or other similar or related
articles that are intended by the manufacturer to
00:17:42.080 --> 00:17:47.920
be used alone or in combination for a medical
purpose. And the spectrum of what constitutes
00:17:47.920 --> 00:17:52.800
a medical device can thus range from very
simple devices such as tongue depressors,
00:17:52.800 --> 00:17:58.240
contact lenses, and thermometers to more
complex devices such as blood pressure monitors,
00:17:58.240 --> 00:18:04.960
insulin pumps, MRI machines, implantable
pacemakers, and even software as a medical
00:18:04.960 --> 00:18:09.680
device or AI-enabled monitors or
drug device combinations, as well.
00:18:09.680 --> 00:18:13.520
So talking about regulation, I think
it is also very important to stress
00:18:13.520 --> 00:18:17.280
that medical devices are used in
many diverse situations by very
00:18:17.280 --> 00:18:22.480
different stakeholders. And testing has
to take this variety into consideration,
00:18:22.480 --> 00:18:28.560
and it is intrinsically tied to regulatory
requirements across various jurisdictions.
00:18:28.560 --> 00:18:33.120
During the pre-market phase, medical
testing establishes baseline safety
00:18:33.120 --> 00:18:37.120
and effectiveness metrics through
bench testing, performance standards,
00:18:37.120 --> 00:18:43.360
and clinical studies. And post-market testing
ensures that real-world data informs ongoing
00:18:43.360 --> 00:18:49.120
compliance and safety improvements. So testing
is indispensable in translating technological
00:18:49.120 --> 00:18:53.600
innovation into safe and effective
medical devices. And while particular
00:18:53.600 --> 00:18:58.960
details of pre-market and post-market review
procedures may slightly differ among countries,
00:18:58.960 --> 00:19:04.640
most developed jurisdictions regulate medical
devices similarly to the US or European models.
00:19:04.640 --> 00:19:09.840
So most jurisdictions with medical device
regulation classify devices based on their
00:19:09.840 --> 00:19:15.440
risk profile, intended use, indications
for use, technological characteristics,
00:19:15.440 --> 00:19:22.200
and the regulatory controls necessary to provide a
reasonable assurance of safety and effectiveness.
00:19:22.200 --> 00:19:27.280
SULLIVAN: So medical devices face
a pretty prescriptive multi-level
00:19:27.280 --> 00:19:30.320
testing path before they hit the
market. From your vantage point,
00:19:30.320 --> 00:19:34.120
what are some of the downsides of that
system and when does it make the most sense?
00:19:34.120 --> 00:19:39.440
MINSSEN: One primary drawback is, of course,
the lengthy and expensive approval process.
00:19:39.440 --> 00:19:44.240
High-risk devices, for example, often
undergo years of clinical trials,
00:19:44.240 --> 00:19:47.840
which can cost millions of dollars, and
this can create a significant barrier
00:19:47.840 --> 00:19:54.240
for startups and small companies with limited
resources. And even for moderate-risk devices,
00:19:54.240 --> 00:19:59.520
the regulatory burden can slow product
development and time to the market.
00:19:59.520 --> 00:20:05.760
And the approach can also limit flexibility.
Prescriptive requirements may not accommodate
00:20:05.760 --> 00:20:10.800
emerging innovations like digital therapeutics
or AI-based diagnostics in a feasible way. And
00:20:10.800 --> 00:20:16.240
in such cases, the framework can unintentionally
[stiffen] innovation by discouraging creative
00:20:16.240 --> 00:20:20.960
solutions or iterative improvements,
which as matter of fact can also put
00:20:20.960 --> 00:20:27.040
patients at risk when you don't use new
technologies and AI. And additionally,
00:20:27.040 --> 00:20:31.680
the same level of scrutiny may be
applied to low-risk devices, where
00:20:31.680 --> 00:20:37.680
the extensive testing and documentation may also
be disproportionate to the actual patient risk.
00:20:37.680 --> 00:20:42.320
However, the prescriptive model is
highly appropriate where we have high
00:20:42.320 --> 00:20:47.360
testing standards for high-risk medical
devices, in my view, particularly those
00:20:47.360 --> 00:20:52.960
that are life-sustaining, implanted,
or involve new materials or mechanisms.
00:20:52.960 --> 00:20:57.680
I also wanted to say that I think that
these higher compliance thresholds can
00:20:57.680 --> 00:21:03.520
be OK and necessary if you have a system
where authorities and stakeholders also
00:21:03.520 --> 00:21:10.320
have the capacity and funding to enforce,
monitor, and achieve compliance with such
00:21:10.320 --> 00:21:14.240
rules in a feasible, time-effective,
and straightforward manner. And this,
00:21:14.240 --> 00:21:18.480
of course, requires resources,
novel solutions, and investments.
00:21:18.480 --> 00:21:23.600
SULLIVAN: A range of tests are undertaken across
the life cycle of medical devices. How do these
00:21:23.600 --> 00:21:28.480
testing requirements vary across different stages
of development and across various applications?
00:21:28.480 --> 00:21:35.280
MINSSEN: Yes, that's a good question. So
I think first it is important to realize
00:21:35.280 --> 00:21:42.160
that testing is conducted by various entities,
including manufacturers, independent third-party
00:21:42.160 --> 00:21:49.120
laboratories, and regulatory agencies. And
it occurs throughout the device life cycle,
00:21:49.120 --> 00:21:54.000
beginning with iterative testing during the
research and development stage, advancing
00:21:54.000 --> 00:22:01.120
to pre-market evaluations, and continuing into
post-market monitoring. And the outcomes of these
00:22:01.120 --> 00:22:09.280
tests directly impact regulatory approvals, market
access, and device design refinements, as well.
00:22:09.840 --> 00:22:14.800
So the testing results are typically shared
with regulatory authorities and in some cases
00:22:14.800 --> 00:22:20.480
with healthcare providers and the broader
public to enhance transparency and trust.
00:22:20.480 --> 00:22:25.520
So if you talk about the different phases that
play a role here … so let's turn to the pre-market
00:22:25.520 --> 00:22:31.760
phase, where manufacturers must demonstrate
that the device is conformed to safety and
00:22:31.760 --> 00:22:38.560
performance benchmarks defined by regulatory
authorities. Pre-market evaluations include
00:22:38.560 --> 00:22:45.360
functional bench testing, biocompatibility, for
example, assessments and software validation,
00:22:45.360 --> 00:22:50.960
all of which are integral components
of a manufacturer's submission.
00:22:50.960 --> 00:22:55.520
But, yes, but, testing also, and
we touched already up on that,
00:22:55.520 --> 00:23:03.200
extends into the post-market phase, where it
continues to ensure device safety and efficacy,
00:23:03.200 --> 00:23:09.280
and post-market surveillance relies on testing
to monitor real-world performance and identify
00:23:09.280 --> 00:23:15.200
emerging risks on the post-market phase. By
integrating real-world evidence into ongoing
00:23:15.200 --> 00:23:22.560
assessments, manufacturers can address unforeseen
issues, update devices as needed, and maintain
00:23:22.560 --> 00:23:27.840
compliance with evolving regulatory expectations.
And I think this is particularly important in
00:23:27.840 --> 00:23:33.440
this new generation of medical devices that
are AI-enabled or machine-learning enabled.
00:23:33.440 --> 00:23:38.640
I think we have to understand that in this
AI-enabled medical devices field, you know,
00:23:38.640 --> 00:23:45.040
the devices and the algorithms that are working
with them, they can improve in the lifetime of a
00:23:45.040 --> 00:23:51.200
product. So actually, not only you could assess
them and make sure that they maintain safe,
00:23:51.200 --> 00:23:56.880
you could also sometimes lower the risk
category by finding evidence that these
00:23:56.880 --> 00:24:04.720
devices are actually becoming more precise and
safer. So it can both, you know, heighten the risk
00:24:04.720 --> 00:24:11.520
category or lower the risk category, and that's
why this continuous testing is so important.
00:24:11.520 --> 00:24:15.360
SULLIVAN: Given what you just said, how
should regulators handle a device whose
00:24:15.360 --> 00:24:18.480
algorithm keeps updating itself after approval?
00:24:18.480 --> 00:24:27.040
MINSSEN: Well, it has to be an iterative process
that is feasible and straightforward and that is
00:24:27.040 --> 00:24:33.520
based on a very efficient, both time efficient
and performance efficient, communication between
00:24:33.520 --> 00:24:39.760
the regulatory authorities and the medical device
developers, right. We need to have the sensors in
00:24:39.760 --> 00:24:50.240
place that spot potential changes, and we need
to have the mechanisms in place that allow us to
00:24:50.240 --> 00:24:57.440
quickly react to these changes both regulatory
wise and also in the technological way.
00:24:58.320 --> 00:25:04.880
So I think communication is important, and we need
to have the pathways and the feedback loops in the
00:25:04.880 --> 00:25:14.680
regulation that quickly allow us to monitor
these self-learning algorithms and devices.
00:25:14.680 --> 00:25:19.840
SULLIVAN: It sounds like it's just … there's such
a delicate balance between advancing technology
00:25:19.840 --> 00:25:26.000
and really ensuring public safety. You know, if
we clamp down too hard, we stifle that innovation.
00:25:26.000 --> 00:25:31.760
You already touched upon this a bit. But if we're
too lax, we risk unintended consequences. And I'd
00:25:31.760 --> 00:25:37.200
just love to hear how you think the field is
balancing that and any learnings you can share.
00:25:37.200 --> 00:25:41.920
MINSSEN: So this is very true, and you just
touched upon a very central question also
00:25:41.920 --> 00:25:47.440
in our research and our writing. And this is
also the reason why medical device regulation
00:25:47.440 --> 00:25:53.600
is so fascinating and continues to evolve in
response to rapid advancements in technologies,
00:25:53.600 --> 00:25:58.560
particularly dual technologies regarding
digital health, artificial intelligence,
00:25:58.560 --> 00:26:01.040
for example, and personalized medicine.
00:26:01.040 --> 00:26:05.360
And finding the balance is tricky because
also [a] related major future challenge
00:26:06.160 --> 00:26:10.880
relates to the increasing regulatory
jungle and the complex interplay between
00:26:10.880 --> 00:26:16.640
evolving regulatory landscapes
that regulate AI more generally.
00:26:16.640 --> 00:26:20.960
We really need to make sure that the regulatory
authorities that deal with this, that need to
00:26:20.960 --> 00:26:27.200
find the right balance to promote innovation
and mitigate and prevent risks, need to have
00:26:27.200 --> 00:26:32.560
the capacity to do this. So this requires
investments, and it also requires new ways
00:26:32.560 --> 00:26:39.960
to regulate this technology more flexibly, for
example through regulatory sandboxes and so on.
00:26:39.960 --> 00:26:43.600
SULLIVAN: Could you just expand upon
that a bit and double-click on what
00:26:43.600 --> 00:26:47.360
it is you're seeing there? What excites
you about what's happening in that space?
00:26:47.360 --> 00:26:53.600
MINSSEN: Yes, well, the research of my group at
the Center for Advanced Studies in Bioscience
00:26:53.600 --> 00:27:01.680
Innovation Law is very broad. I mean, we are
looking into gene editing technologies. We are
00:27:01.680 --> 00:27:09.520
looking into new biologics. We are looking
into medical devices, as well, obviously,
00:27:09.520 --> 00:27:12.800
but also other technologies
in advanced medical computing.
00:27:12.800 --> 00:27:20.400
And what we see across the line here is that
there is an increasing demand for having
00:27:20.400 --> 00:27:26.000
more adaptive and flexible regulatory
frameworks in these new technologies,
00:27:26.000 --> 00:27:31.520
in particular when they have new uses,
regulations that are focusing more on
00:27:31.520 --> 00:27:37.520
the product rather than the process. And I
have recently written a report, for example,
00:27:37.520 --> 00:27:44.720
for emerging biotechnologies and bio-solutions
for the EU commission. And even in that area,
00:27:44.720 --> 00:27:50.160
regulatory sandboxes are increasingly
important, increasingly considered.
00:27:50.160 --> 00:27:56.480
So this idea of regulatory sandboxes has been
developing originally in the financial sector,
00:27:56.480 --> 00:28:02.560
and it is now penetrating into other
sectors, including synthetic biology,
00:28:03.360 --> 00:28:08.720
emerging biotechnologies, gene
editing, AI, quantum technology,
00:28:08.720 --> 00:28:15.200
as well. This is basically creating an
environment where actors can test new
00:28:15.200 --> 00:28:21.520
ideas in close collaboration and under
the oversight of regulatory authorities.
00:28:21.520 --> 00:28:29.680
But to implement this in the AI sector now also
leads us to a lot of questions and challenges.
00:28:29.680 --> 00:28:37.280
For example, you need to have the capacities of
authorities that are governing and monitoring and
00:28:37.280 --> 00:28:43.360
deciding on these regulatory sandboxes. There are
issues relating to competition law, for example,
00:28:43.360 --> 00:28:49.680
which you call antitrust law in the US, because
the question is, who can enter the sandbox and how
00:28:49.680 --> 00:28:57.200
may they compete after they exit the sandbox?
And there are many questions relating to,
00:28:57.200 --> 00:29:03.411
how should we work with these sandboxes and
how should we implement these sandboxes?
00:29:03.411 --> 00:29:04.055
[TRANSITION MUSIC]
00:29:04.055 --> 00:29:06.520
SULLIVAN: Well, Timo, it has just been
such a pleasure to speak with you today.
00:29:06.520 --> 00:29:08.560
MINSSEN: Yes, thank you very much.
00:29:16.800 --> 00:29:19.600
And now I'm happy to introduce Chad Atalla.
00:29:19.600 --> 00:29:23.760
Chad is senior applied scientist in
Microsoft Research New York City's
00:29:23.760 --> 00:29:28.800
Sociotechnical Alignment Center, where they
contribute to foundational responsible AI
00:29:28.800 --> 00:29:33.280
research and practical responsible AI
solutions for teams across Microsoft.
00:29:33.280 --> 00:29:34.627
Chad, welcome!
00:29:34.627 --> 00:29:35.200
CHAD ATALLA: Thank you.
00:29:35.200 --> 00:29:40.480
SULLIVAN: So we'll kick off with a couple
questions just to dive right in. So tell me a
00:29:40.480 --> 00:29:46.240
little bit more about the Sociotechnical Alignment
Center, or STAC? I know it was founded in 2022.
00:29:46.240 --> 00:29:50.160
I'd love to just learn a little bit more about
what the group does, how you're thinking about
00:29:50.160 --> 00:29:54.320
evaluating AI, and maybe just give us a sense
of some of the projects you're working on.
00:29:54.320 --> 00:29:57.486
ATALLA: Yeah, absolutely.
The name is quite a mouthful.
00:29:57.486 --> 00:29:58.044
SULLIVAN: It is! [LAUGHS]
00:29:58.044 --> 00:30:01.000
ATALLA: So let's start by breaking
that down and seeing what that means.
00:30:01.000 --> 00:30:01.943
SULLIVAN: Great.
00:30:01.943 --> 00:30:05.360
ATALLA: So modern AI systems
are sociotechnical systems,
00:30:05.360 --> 00:30:10.160
meaning that the social and technical
aspects are deeply intertwined. And
00:30:10.160 --> 00:30:16.560
we're interested in aligning the behaviors of
these sociotechnical systems with some values.
00:30:16.560 --> 00:30:21.520
Those could be societal values; they could
be regulatory values, organizational values,
00:30:21.520 --> 00:30:28.560
etc. And to make this alignment happen, we
need the ability to evaluate the systems.
00:30:28.560 --> 00:30:34.640
So my team is broadly working on an evaluation
framework that acknowledges the sociotechnical
00:30:34.640 --> 00:30:40.640
nature of the technology and the often-abstract
nature of the concepts we're actually interested
00:30:40.640 --> 00:30:46.800
in evaluating. As you noted, it's an applied
science team, so we split our time between some
00:30:46.800 --> 00:30:54.320
fundamental research and time to bridge the
work into real products across the company.
00:30:54.320 --> 00:30:59.760
And I also want to note that to power this
sort of work, we have an interdisciplinary team
00:30:59.760 --> 00:31:04.320
drawing upon the social sciences, linguistics,
statistics, and, of course, computer science.
00:31:04.320 --> 00:31:08.560
SULLIVAN: Well, I'm eager to get into our
takeaways from the conversation with both
00:31:08.560 --> 00:31:12.240
Daniel and Timo. But maybe just to
double-click on this for a minute,
00:31:12.240 --> 00:31:16.960
can you talk a bit about some of the overarching
goals of the AI evaluations that you noted?
00:31:16.960 --> 00:31:25.040
ATALLA: So evaluation is really the act of making
valuative judgments based on some evidence,
00:31:25.040 --> 00:31:32.160
and in the case of AI evaluation, that evidence
might be from tests or measurements, right. And
00:31:32.160 --> 00:31:38.240
the goal of why we're doing this in the first
place is to make decisions and claims most often.
00:31:38.240 --> 00:31:44.080
So perhaps I am going to make a claim about
a model that I'm producing, and I want to say
00:31:44.080 --> 00:31:50.560
that it's better than this other model. Or we are
asking whether a certain product is safe to ship.
00:31:50.560 --> 00:31:57.360
All of these decisions need to be informed by
good evaluation and therefore good measurement
00:31:57.360 --> 00:32:05.040
or testing. And I'll also note that in the
regulatory conversation, risk is often what
00:32:05.040 --> 00:32:11.160
we want to evaluate. So that is a goal in and
of itself. And I'll touch more on that later.
00:32:11.160 --> 00:32:15.280
SULLIVAN: I read a recent paper that you
had put out with some of our colleagues
00:32:15.280 --> 00:32:19.040
from Microsoft Research, from the
University of Michigan, and Stanford,
00:32:19.040 --> 00:32:23.360
and you were arguing that evaluating
generative AI is the social-science
00:32:23.360 --> 00:32:27.120
measurement challenge. Maybe for those
who haven't read the paper, what does this
00:32:27.120 --> 00:32:31.200
mean? And can you tell us a little bit more
about what motivated you and your coauthors?
00:32:31.200 --> 00:32:37.280
ATALLA: So the measurement tasks involved in
evaluating generative AI systems are often
00:32:37.280 --> 00:32:42.400
abstract and contested. So that means
they cannot be directly measured and
00:32:42.400 --> 00:32:48.400
must instead [be] indirectly measured via other
observable phenomena. So this is very different
00:32:48.400 --> 00:32:54.080
than the older machine learning paradigm, where,
let's say, for example, I had a system that took a
00:32:54.080 --> 00:32:59.440
picture of a traffic light and told you whether
it was green, yellow, or red at a given time.
00:32:59.440 --> 00:33:05.760
If we wanted to evaluate that system, the task is
much simpler. But with the modern generative AI
00:33:05.760 --> 00:33:15.200
systems that are also general purpose, they have
open-ended output, and language in a whole chat or
00:33:15.200 --> 00:33:20.480
multiple paragraphs being outputted can have a lot
of different properties. And as I noted, these are
00:33:20.480 --> 00:33:25.920
general-purpose systems, so we don't know exactly
what task they're supposed to be carrying out.
00:33:25.920 --> 00:33:32.640
So then the question becomes, if I want to make
some decision or claim—maybe I want to make a
00:33:32.640 --> 00:33:41.200
claim that this system has human-level reasoning
capabilities—well, what does that mean? Do I have
00:33:41.200 --> 00:33:48.320
the same impression of what that means as you
do? And how do we know whether the downstream,
00:33:48.320 --> 00:33:53.520
you know, measurements and tests that I'm
conducting actually will support my notion of
00:33:53.520 --> 00:33:59.920
what it means to have human-level reasoning,
right? Difficult questions. But luckily,
00:33:59.920 --> 00:34:03.840
social scientists have been dealing with these
exact sorts of challenges for multiple decades
00:34:03.840 --> 00:34:09.920
in fields like education, political science, and
psychometrics. So we're really attempting to avoid
00:34:09.920 --> 00:34:16.000
reinventing the wheel here and trying
to learn from their past methodologies.
00:34:16.000 --> 00:34:21.200
And so the rest of the paper goes on to delve into
a four-level framework, a measurement framework,
00:34:21.200 --> 00:34:26.320
that's grounded in the measurement theory from
the quantitative social sciences that takes us
00:34:26.320 --> 00:34:32.960
all the way from these abstract and contested
concepts through processes to get much clearer
00:34:32.960 --> 00:34:37.320
and eventually reach reliable and valid
measurements that can power our evaluations.
00:34:37.320 --> 00:34:40.400
SULLIVAN: I love that. I mean, that's
the whole point of this podcast, too,
00:34:40.400 --> 00:34:44.320
right. Is to really build on those other
learnings and frameworks that we're taking
00:34:44.320 --> 00:34:49.760
from industries that have been thinking about this
for much longer. Maybe from your vantage point,
00:34:49.760 --> 00:34:54.880
what are some of the biggest day-to-day
hurdles in building solid AI evaluations and,
00:34:54.880 --> 00:34:59.040
I don't know, do we need more shared
standards? Are there bespoke methods?
00:34:59.040 --> 00:35:01.680
Are those the way to go? I would love
to just hear your thoughts on that.
00:35:01.680 --> 00:35:06.800
ATALLA: So let's talk about some of those
practical challenges. And I want to briefly
00:35:06.800 --> 00:35:12.400
go back to what I mentioned about risk before,
all right. Oftentimes, some of the regulatory
00:35:12.400 --> 00:35:18.640
environment is requiring practitioners to measure
the risk involved in deploying one of their
00:35:18.640 --> 00:35:27.200
models or AI systems. Now, risk is importantly
a concept that includes both event and impact,
00:35:27.200 --> 00:35:33.280
right. So there's the probability of some event
occurring. For the case of AI evaluation, perhaps
00:35:33.280 --> 00:35:41.520
this is us seeing a certain AI behavior exhibited.
Then there's also the severity of the impacts,
00:35:41.520 --> 00:35:47.840
and this is a complex chain of effects
in the real world that happen to people,
00:35:47.840 --> 00:35:56.640
organizations, systems, etc., and it's a lot
more challenging to observe the impacts, right.
00:35:56.640 --> 00:36:03.280
So if we're saying that we need to measure
risk, we have to measure both the event and
00:36:03.280 --> 00:36:09.280
the impacts. But realistically, right now,
the field is not doing a very good job of
00:36:09.280 --> 00:36:14.320
actually measuring the impacts. This requires
vastly different techniques and methodologies
00:36:14.320 --> 00:36:19.120
where if I just wanted to measure something
about the event itself, I can, you know,
00:36:19.120 --> 00:36:26.160
do that in a technical sandbox environment and
perhaps have some automated methods to detect
00:36:26.160 --> 00:36:30.720
whether a certain AI behavior is being exhibited.
But if I want to measure the impacts? Now,
00:36:30.720 --> 00:36:35.360
we're in the realm of needing to have real people
involved, and perhaps a longitudinal study where
00:36:35.360 --> 00:36:39.520
you have interviews, questionnaires,
and more qualitative evidence-gathering
00:36:39.520 --> 00:36:47.200
techniques to truly understand the long-term
impacts. So that's a significant challenge.
00:36:47.200 --> 00:36:52.560
Another is that, you know, let's say we forget
about the impacts for now and we focus on the
00:36:52.560 --> 00:37:00.560
event side of things. Still, we need datasets, we
need annotations, and we need metrics to make this
00:37:00.560 --> 00:37:08.000
whole thing work. When I say we need datasets,
if I want to test whether my system has good
00:37:08.000 --> 00:37:14.240
mathematical reasoning, what questions should I
ask? What are my set of inputs that are relevant?
00:37:14.240 --> 00:37:18.800
And then when I get the response from the system,
how do I annotate them? How do I know if it was a
00:37:18.800 --> 00:37:23.840
good response that did demonstrate mathematical
reasoning or if it was a mediocre response?
00:37:24.640 --> 00:37:29.680
And then once I have an annotation of
all of these outputs from the AI system,
00:37:29.680 --> 00:37:33.800
how do I aggregate those all up
into a single informative number?
00:37:33.800 --> 00:37:38.320
SULLIVAN: Earlier in this episode, we
heard Daniel and Timo walk through the
00:37:38.320 --> 00:37:42.960
regulatory frameworks in pharma and medical
devices. I'd be curious what pieces of those
00:37:42.960 --> 00:37:47.240
mature systems are already showing up or at
least may be bubbling up in AI governance.
00:37:47.240 --> 00:37:53.200
ATALLA: Great question. You know, Timo was
talking about the pre-market and post-market
00:37:53.200 --> 00:38:00.720
testing difference. Of course, this is similarly
important in the AI evaluation space. But again,
00:38:00.720 --> 00:38:04.720
these have different methodologies
and serve different purposes.
00:38:04.720 --> 00:38:12.400
So within the pre-deployment phase, we
don't have evidence of how people are
00:38:12.400 --> 00:38:16.560
going to use the system. And when we
have these general-purpose AI systems,
00:38:16.560 --> 00:38:22.320
to understand what the risks are, we really
need to have a sense of what might happen
00:38:22.320 --> 00:38:26.640
and how they might be used. So there are
significant challenges there where I think
00:38:26.640 --> 00:38:33.680
we can learn from other fields and how they do
pre-market testing. And the difference in that
00:38:33.680 --> 00:38:38.800
pre- versus post-market testing also ties to
testing at different stages in the life cycle.
00:38:38.800 --> 00:38:45.920
For AI systems, we already see some regulations
saying you need to start with the base model
00:38:45.920 --> 00:38:52.160
and do some evaluation of the base model,
some basic attributes, some core attributes,
00:38:52.160 --> 00:38:57.760
of that base model before you start putting
it into any real products. But once we have
00:38:57.760 --> 00:39:03.520
a product in mind, we have a user base in mind,
we have a specific task—like maybe we're going
00:39:03.520 --> 00:39:07.440
to integrate this model into Outlook and
it's going to help you write emails—now
00:39:07.440 --> 00:39:12.320
we suddenly have a much crisper picture of
how the system will interact with the world
00:39:12.320 --> 00:39:18.400
around it. And again, at that stage, we need
to think about another round of evaluation.
00:39:18.400 --> 00:39:23.680
Another part that jumped out to me in what
they were saying about pharmaceuticals is
00:39:23.680 --> 00:39:28.080
that sometimes approvals can be
based on surrogate endpoints. So
00:39:28.080 --> 00:39:33.120
this is like we're choosing some heuristic.
Instead of measuring the long-term impact,
00:39:33.120 --> 00:39:36.880
which is what we actually care about,
perhaps we have a proxy that we feel
00:39:36.880 --> 00:39:42.960
like is a good enough indicator of what
that long-term impact might look like.
00:39:42.960 --> 00:39:50.480
This is occurring in the AI evaluation space
right now and is often perhaps even the default
00:39:50.480 --> 00:39:56.320
here since we're not seeing that many studies
of the long-term impact itself. We are seeing,
00:39:56.320 --> 00:40:03.600
instead, folks constructing these heuristics or
proxies and saying if I see this behavior happen,
00:40:03.600 --> 00:40:09.440
I'm going to assume that it indicates this
sort of impact will happen downstream. And
00:40:09.440 --> 00:40:16.480
that's great. It's one of the techniques that
was used to speed up and reduce the barrier
00:40:16.480 --> 00:40:22.320
to innovation in the other fields. And I think
it's great that we are applying that in the AI
00:40:22.320 --> 00:40:27.920
evaluation space. But special care is, of course,
needed to ensure that those heuristics and proxies
00:40:27.920 --> 00:40:33.280
you're using are reasonable indicators of
the greater outcome you're looking for.
00:40:33.280 --> 00:40:38.880
SULLIVAN: What are some of the promising ideas
from maybe pharma or med device regulation
00:40:38.880 --> 00:40:43.360
that maybe haven't made it to AI testing
yet and maybe should? And where would you
00:40:43.360 --> 00:40:47.440
urge technologists, policymakers, and
researchers to focus their energy next?
00:40:47.440 --> 00:40:51.200
ATALLA: Well, one of the key things
that jumped out to me in the discussion
00:40:51.200 --> 00:40:58.240
about pharmaceuticals was driving home the
emphasis that there is a holistic focus on
00:40:58.240 --> 00:41:04.800
safety and efficacy. These go hand in hand and
decisions must be made while considering both
00:41:04.800 --> 00:41:11.360
pieces of the picture. I would like to see that
further emphasized in the AI evaluation space.
00:41:11.360 --> 00:41:18.080
Often, we are seeing evaluations of
risk being separated from evaluations
00:41:18.080 --> 00:41:27.440
of performance or quality or efficacy, but
these two pieces of the puzzle really are
00:41:27.440 --> 00:41:33.760
not enough for us to make informed
decisions independently. And that
00:41:33.760 --> 00:41:41.040
ties back into my desire to really
also see us measuring the impacts.
00:41:41.040 --> 00:41:47.920
So we see Phase 3 trials as something that occurs
in the medical devices and pharmaceuticals field.
00:41:47.920 --> 00:41:52.640
That's not something that we are doing an
equivalent of in the AI evaluation space
00:41:52.640 --> 00:41:59.440
at this time. These are really cost intensive.
They can last years and really involve careful
00:41:59.440 --> 00:42:05.840
monitoring of that holistic picture of
safety and efficacy. And realistically,
00:42:05.840 --> 00:42:10.560
we are not going to be able to put that
on the critical path to getting specific
00:42:10.560 --> 00:42:16.400
individual AI models or AI systems vetted
before they go out into the world. However,
00:42:16.400 --> 00:42:24.800
I would love to see a world in which this sort
of work is prioritized and funded or required.
00:42:24.800 --> 00:42:31.360
Think of how, with social media, it took quite
a long time for us to understand that there are
00:42:31.360 --> 00:42:38.720
some long-term negative impacts on mental health,
and we have the opportunity now, while the AI wave
00:42:38.720 --> 00:42:44.880
is still building, to start prioritizing and
funding this sort of work. Let it run in the
00:42:44.880 --> 00:42:52.960
background and as soon as possible develop a good
understanding of the subtle, long-term effects.
00:42:52.960 --> 00:42:59.040
More broadly, I would love to see us focus on
reliability and validity of the evaluations we're
00:42:59.040 --> 00:43:06.240
conducting because trust in these decisions
and claims is important. If we don't focus
00:43:06.240 --> 00:43:11.760
on building reliable, valid, and trustworthy
evaluations, we're just going to continue to be
00:43:11.760 --> 00:43:18.320
flooded by a bunch of competing, conflicting,
and largely meaningless AI evaluations.
00:43:18.320 --> 00:43:22.480
SULLIVAN: In a number of the discussions
we've had on this podcast, we talked about
00:43:22.480 --> 00:43:28.000
how it's not just one entity that really needs
to ensure safety across the board, and I’d just
00:43:28.000 --> 00:43:33.040
love to hear from you how you think about some
of those ecosystem collaborations, and you know,
00:43:33.040 --> 00:43:38.800
from across … where we think about ourselves
as more of a platform company or places that
00:43:38.800 --> 00:43:43.760
these AI models are being deployed more at the
application level. Tell me a little bit about how
00:43:43.760 --> 00:43:49.400
you think about, sort of, stakeholders in that mix
and where responsibility lies across the board.
00:43:49.400 --> 00:43:54.560
ATALLA: It's interesting. In this age
of general-purpose AI technologies,
00:43:54.560 --> 00:44:00.080
we're often seeing one company or
organization being responsible for
00:44:00.080 --> 00:44:05.440
building the foundational model. And then
many, many other people will take that
00:44:05.440 --> 00:44:11.120
model and build it into specific products that
are designed for specific tasks and contexts.
00:44:11.120 --> 00:44:17.440
Of course, in that, we already see that
there is a responsibility of the owners
00:44:17.440 --> 00:44:22.560
of that foundational model to do some
testing of the central model before
00:44:22.560 --> 00:44:27.120
they distribute it broadly. And then
again, there is responsibility of all
00:44:27.120 --> 00:44:34.320
of the downstream individuals digesting that
and turning it into products to consider the
00:44:34.320 --> 00:44:39.520
specific contexts that they are deploying
into and how that may affect the risks we're
00:44:39.520 --> 00:44:47.040
concerned with or the types of quality and
safety and performance we need to evaluate.
00:44:47.040 --> 00:44:52.640
Again, because that field of risks
we may be concerned with is so broad,
00:44:52.640 --> 00:44:59.680
some of them also require an immense amount of
expertise. Let's think about whether AI systems
00:44:59.680 --> 00:45:06.640
can enable people to create dangerous chemicals
or dangerous weapons at home. It's not that every
00:45:06.640 --> 00:45:12.640
AI practitioner is going to have the knowledge
to evaluate this, so in some of those cases,
00:45:12.640 --> 00:45:19.600
we really need third-party experts, people
who are experts in chemistry, biology,
00:45:19.600 --> 00:45:26.800
etc., to come in and evaluate certain systems
and models for those specific risks, as well.
00:45:26.800 --> 00:45:31.680
So I think there are many reasons why
multiple stakeholders need to be involved,
00:45:31.680 --> 00:45:36.400
partly from who owns what and is responsible
for what and partly from the perspective of
00:45:36.400 --> 00:45:42.200
who has the expertise to meaningfully
construct the evaluations that we need.
00:45:42.200 --> 00:45:47.360
SULLIVAN: Well, Chad, this has just been great
to connect, and in a few of our discussions,
00:45:47.360 --> 00:45:53.040
we've done a bit of a lightning round, so I'd
love to just hear your 30-second responses
00:45:53.040 --> 00:45:58.400
to a few of these questions. Perhaps favorite
evaluation you've run so far this year?
00:45:58.400 --> 00:46:04.560
ATALLA: So I've been involved in trying to
evaluate some language models for whether
00:46:04.560 --> 00:46:10.720
they infer sensitive attributes about people.
So perhaps you're chatting with a chatbot,
00:46:10.720 --> 00:46:17.440
and it infers your religion or sexuality based
on things you're saying or how you sound, right.
00:46:17.440 --> 00:46:23.120
And in working to evaluate this, we encounter a
lot of interesting questions. Or, like, what is
00:46:23.120 --> 00:46:28.560
a sensitive attribute? What makes these attributes
sensitive, and what are the differences that make
00:46:28.560 --> 00:46:33.840
it inappropriate for an AI system to infer these
things about a person? Whereas realistically,
00:46:33.840 --> 00:46:39.920
whenever I meet a person on the street, my brain
is immediately forming first impressions and some
00:46:39.920 --> 00:46:45.600
assumptions about these people. So it's a very
interesting and thought-provoking evaluation to
00:46:46.240 --> 00:46:51.840
conduct and think about the norms that
we place upon people interacting with
00:46:51.840 --> 00:46:56.120
other people and the norms we place upon
AI systems interacting with other people.
00:46:56.120 --> 00:47:02.144
SULLIVAN: That’s fascinating! I'd love to hear
the AI buzzword you'd retire tomorrow. [LAUGHTER]
00:47:02.144 --> 00:47:08.320
ATALLA: I would love to see the term
“bias” being used less when referring
00:47:08.320 --> 00:47:15.440
to fairness-related issues and systems.
Bias happens to be a highly overloaded
00:47:15.440 --> 00:47:20.960
term in statistics and machine learning
and has a lot of technical meanings and
00:47:20.960 --> 00:47:26.800
just fails to perfectly capture
what we mean in the AI risk sense.
00:47:26.800 --> 00:47:30.280
SULLIVAN: And last one. One
metric we're not tracking enough.
00:47:30.280 --> 00:47:39.440
ATALLA: I would say over-blocking, and this comes
into that connection between the holistic picture
00:47:39.440 --> 00:47:47.440
of safety and efficacy. It's too easy to produce
systems that throw safety to the wind and focus
00:47:47.440 --> 00:47:52.560
purely on utility or achieving some goal, but
simultaneously, the other side of the picture
00:47:52.560 --> 00:47:59.760
is possible, where we can clamp down too hard
and reduce the utility of our systems and block
00:47:59.760 --> 00:48:07.520
even benign and useful outputs just because they
border on something sensitive. So it's important
00:48:07.520 --> 00:48:13.680
for us to track that over-blocking and actively
track that tradeoff between safety and efficacy.
00:48:13.680 --> 00:48:18.480
SULLIVAN: Yeah, we talk a lot about this on the
podcast, too, of how do you both make things safe
00:48:18.480 --> 00:48:24.320
but also ensure innovation can thrive, and I think
you hit the nail on the head with that last piece.
00:48:24.320 --> 00:48:25.158
[MUSIC]
00:48:25.158 --> 00:48:30.290
Well, Chad, this was really terrific. Thanks
for joining us and thanks for your work and your
00:48:30.290 --> 00:48:33.656
perspectives. And another big thanks to Daniel and
Timo for setting the stage earlier in the podcast.
00:48:33.656 --> 00:48:38.755
And to our listeners, thanks for tuning
in. You can find resources related to
00:48:38.755 --> 00:48:41.440
this podcast in the show notes. And
if you want to learn more about how
00:48:41.440 --> 00:48:47.360
Microsoft approaches AI governance,
you can visit microsoft.com/RAI.
00:48:47.360 --> 00:49:03.440
See you next time!
00:49:03.440 --> 00:49:04.334
[MUSIC FADES]
Kind: captions
Language: en-US
00:00:02.405 --> 00:00:03.107
[MUSIC]
00:00:03.107 --> 00:00:06.800
KATHLEEN SULLIVAN: Welcome
to AI Testing and Evaluation:
00:00:06.800 --> 00:00:11.520
Learnings from Science and Industry.
I'm your host, Kathleen Sullivan.
00:00:11.520 --> 00:00:16.640
As generative AI continues to advance, Microsoft
has gathered a range of experts—from genome
00:00:16.640 --> 00:00:21.040
editing to cybersecurity—to share how
their fields approach evaluation and
00:00:21.040 --> 00:00:26.160
risk assessment. Our goal is to learn from
their successes and their stumbles to move
00:00:26.160 --> 00:00:31.360
the science and practice of AI testing
forward. In this series, we'll explore
00:00:31.360 --> 00:00:40.137
how these insights might help guide the future of
AI development, deployment, and responsible use.
00:00:40.126 --> 00:00:40.659
[MUSIC ENDS]
00:00:40.659 --> 00:00:42.800
SULLIVAN: Today, I'm excited to
welcome Dan Carpenter and Timo
00:00:42.800 --> 00:00:46.400
Minssen to the podcast to explore
testing and risk assessment in the
00:00:46.400 --> 00:00:50.160
areas of pharmaceuticals and
medical devices, respectively.
00:00:50.160 --> 00:00:54.640
Dan Carpenter is chair of the Department
of Government at Harvard University. His
00:00:54.640 --> 00:01:00.160
research spans the sphere of social and political
science, from petitioning in democratic society
00:01:00.160 --> 00:01:05.600
to regulation and government organizations.
His recent work includes the FDA Project,
00:01:05.600 --> 00:01:09.440
which examines pharmaceutical
regulation in the United States.
00:01:09.440 --> 00:01:12.720
Timo is a professor of law at
the University of Copenhagen,
00:01:12.720 --> 00:01:18.400
where he is also director of the Center for
Advanced Studies in Bioscience Innovation Law.
00:01:18.400 --> 00:01:22.080
He specializes in legal aspects of
biomedical innovation, including
00:01:22.080 --> 00:01:27.680
intellectual property law and regulatory law.
He's exercised his expertise as an advisor
00:01:27.680 --> 00:01:33.040
to such organizations as the World Health
Organization and the European Commission.
00:01:33.040 --> 00:01:38.000
And after our conversations, we'll talk to
Microsoft's Chad Atalla, an applied scientist
00:01:38.000 --> 00:01:43.520
in responsible AI, about how we should think
about these insights in the context of AI.
00:01:43.520 --> 00:01:46.160
Daniel, it's a pleasure to
welcome you to the podcast.
00:01:46.160 --> 00:01:49.662
I'm just so appreciative of you being
here. Thanks for joining us today.
00:01:49.662 --> 00:01:50.720
DANIEL CARPENTER: Thanks for having me.
00:01:50.720 --> 00:01:56.480
SULLIVAN: Dan, before we dissect policy,
let's rewind the tape to your origin story.
00:01:56.480 --> 00:02:00.960
Can you take us to the moment that you first
became fascinated with regulators rather than,
00:02:00.960 --> 00:02:05.840
say, politicians? Was there a spark
that pulled you toward the FDA story?
00:02:05.840 --> 00:02:08.720
CARPENTER: At one point during graduate school,
00:02:08.720 --> 00:02:16.960
I was studying a combination of American
politics and political theory, and I did a
00:02:16.960 --> 00:02:24.720
summer interning at the Department of Housing
and Urban Development. And I began to think,
00:02:24.720 --> 00:02:30.960
why don't people study these administrators
more and the rules they make, the, you know,
00:02:30.960 --> 00:02:37.360
inefficiencies, the efficiencies? Really more
from, kind of, a descriptive standpoint, less from
00:02:37.360 --> 00:02:45.520
a normative standpoint. And I was reading a lot
that summer about the Food and Drug Administration
00:02:45.520 --> 00:02:51.590
and some of the decisions it was making on
AIDS drugs. That was a, sort of, a major, …
00:02:51.590 --> 00:02:52.800
SULLIVAN: Right.
00:02:52.800 --> 00:02:57.680
CARPENTER: … sort of, you know, moment in the
news, in the global news as well as the national
00:02:57.680 --> 00:03:06.880
news during, I would say, what? The late ’80s,
early ’90s? And so I began to look into that.
00:03:06.880 --> 00:03:09.520
SULLIVAN: So now that we know what pulled you in,
00:03:09.520 --> 00:03:13.120
let’s zoom out for our listeners.
Give us the whirlwind tour. I think
00:03:13.120 --> 00:03:17.480
most of us know pharma involves years of
trials, but what’s the part we don’t know?
00:03:17.480 --> 00:03:24.480
CARPENTER: So I think when most businesses develop
a product, they all go through some phases of
00:03:24.480 --> 00:03:30.320
research and development and testing. And
I think what's different about the FDA is,
00:03:30.960 --> 00:03:32.880
sort of, two- or three-fold.
00:03:32.880 --> 00:03:38.560
First, a lot of those tests are much more
stringently specified and regulated by the
00:03:38.560 --> 00:03:46.960
government, and second, one of the reasons
for that is that the FDA imposes not simply
00:03:46.960 --> 00:03:56.000
safety requirements upon drugs in particular but
also efficacy requirements. The FDA wants you to
00:03:56.000 --> 00:04:02.720
prove not simply that it's safe and non-toxic but
also that it's effective. And the final thing,
00:04:02.720 --> 00:04:07.280
I think, that makes the FDA different is
that it stands as what I would call the “veto
00:04:07.280 --> 00:04:13.200
player” over R&D [research and development]
to the marketplace. The FDA basically has,
00:04:13.200 --> 00:04:16.880
sort of, this control over
entry to the marketplace.
00:04:16.880 --> 00:04:23.600
And so what that involves is usually first,
a set of human trials where people who have
00:04:23.600 --> 00:04:29.120
no disease take it. And you're only looking
for toxicity generally. Then there's a set
00:04:29.120 --> 00:04:34.160
of Phase 2 trials, where they look more
at safety and a little bit at efficacy,
00:04:34.160 --> 00:04:41.440
and you're now examining people who have the
disease that the drug claims to treat. And
00:04:41.440 --> 00:04:47.440
you're also basically comparing people who
get the drug, often with those who do not.
00:04:47.440 --> 00:04:53.520
And then finally, Phase 3 involves a much more
direct and large-scale attack, if you will,
00:04:53.520 --> 00:04:59.760
or assessment of efficacy, and that's where you
get the sort of large randomized clinical trials
00:04:59.760 --> 00:05:08.000
that are very expensive for pharmaceutical
companies, biomedical companies to launch,
00:05:08.000 --> 00:05:16.640
to execute, to analyze. And those are often the
sort of core evidence base for the decisions that
00:05:16.640 --> 00:05:22.240
the FDA makes about whether or not to approve
a new drug for marketing in the United States.
00:05:22.240 --> 00:05:27.600
SULLIVAN: Are there differences
in how that process has, you know,
00:05:27.600 --> 00:05:32.400
changed through other countries and maybe just
how that's evolved as you've seen it play out?
00:05:32.400 --> 00:05:38.320
CARPENTER: Yeah, for a long time, I would say that
the United States had probably the most stringent
00:05:38.320 --> 00:05:44.960
regime of regulation for biopharmaceutical
products until, I would say, about the 1990s
00:05:44.960 --> 00:05:52.080
and early 2000s. It used to be the case that a
number of other countries, especially in Europe
00:05:52.080 --> 00:05:58.720
but around the world, basically waited for the
FDA to mandate tests on a drug and only after
00:05:58.720 --> 00:06:05.760
the drug was approved in the United States would
they deem it approvable and marketable in their
00:06:05.760 --> 00:06:12.880
own countries. And then after the formation of the
European Union and the creation of the European
00:06:12.880 --> 00:06:20.080
Medicines Agency, gradually the European Medicines
Agency began to get a bit more stringent.
00:06:20.080 --> 00:06:24.720
But, you know, over the long run, there's
been a lot of, sort of, heterogeneity,
00:06:24.720 --> 00:06:29.760
a lot of variation over time and space,
in the way that the FDA has approached
00:06:29.760 --> 00:06:35.920
these problems. And I'd say in the last 20
years, it's begun to partially deregulate,
00:06:37.200 --> 00:06:45.440
namely, you know, trying to find all sorts of
mechanisms or pathways for really innovative
00:06:45.440 --> 00:06:52.800
drugs for deadly diseases without a lot
of treatments to basically get through the
00:06:52.800 --> 00:06:58.880
process at lower cost. For many people, that
has not been sufficient. They're concerned
00:06:58.880 --> 00:07:06.400
about the cost of the system. Of course, then
the agency also gets criticized by those who
00:07:06.400 --> 00:07:13.080
believe it's too lax. It is potentially letting
ineffective and unsafe therapies on the market.
00:07:13.080 --> 00:07:16.960
SULLIVAN: In your view, when does the
structured model genuinely safeguard
00:07:16.960 --> 00:07:21.200
patients and where do you think it
maybe slows or limits innovation?
00:07:21.200 --> 00:07:30.320
CARPENTER: So I think the worry is that if you
approach pharmaceutical approval as a world
00:07:30.320 --> 00:07:36.480
where only things can go wrong, then you're
really at a risk of limiting innovation. And
00:07:36.480 --> 00:07:44.000
even if you end up letting a lot of things
through, if by your regulations you end up
00:07:44.000 --> 00:07:48.480
basically slowing down the development
process or making it very, very costly,
00:07:48.480 --> 00:07:54.000
then there's just a whole bunch of drugs that
either come to market too slowly or they come
00:07:54.000 --> 00:08:00.000
to market not at all because they just aren't
worth the kind of cost-benefit or, sort of,
00:08:00.000 --> 00:08:05.760
profit analysis of the firm. You know, so that's
been a concern. And I think it's been one of the
00:08:05.760 --> 00:08:11.120
reasons that the Food and Drug Administration
as well as other world regulators have begun to
00:08:11.120 --> 00:08:18.640
basically try to smooth the process and
accelerate the process at the margins.
00:08:18.640 --> 00:08:24.000
The other thing is that they've started to
basically make approvals on the basis of what
00:08:24.000 --> 00:08:29.600
are called surrogate endpoints. So the idea
is that a cancer drug, we really want to know
00:08:29.600 --> 00:08:36.720
whether that drug saves lives, but if we wait
to see whose lives are saved or prolonged by
00:08:36.720 --> 00:08:41.360
that drug, we might miss the opportunity
to make judgments on the basis of, well,
00:08:41.360 --> 00:08:47.360
are we detecting tumors in the bloodstream? Or
can we measure the size of those tumors in, say,
00:08:47.360 --> 00:08:53.920
a solid cancer? And then the further question
is, is the size of the tumor basically a really
00:08:53.920 --> 00:08:59.040
good correlate or predictor of whether
people will die or not, right? Generally,
00:08:59.040 --> 00:09:07.760
the FDA tends to be less stringent when you've
got, you know, a remarkably innovative new therapy
00:09:07.760 --> 00:09:14.400
and the disease being treated is one that just
doesn't have a lot of available treatments, right.
00:09:14.960 --> 00:09:18.880
The one thing that people often think
about when they're thinking about
00:09:18.880 --> 00:09:23.510
pharmaceutical regulation is they often
contrast, kind of, speed versus safety, …
00:09:23.510 --> 00:09:24.065
SULLIVAN: Right.
00:09:24.065 --> 00:09:28.400
CARPENTER: … right. And that's useful as a
tradeoff, but I often try to remind people
00:09:28.400 --> 00:09:34.400
that it's not simply about whether the drug
gets out there and it's unsafe. You know,
00:09:34.400 --> 00:09:39.520
you and I as patients and even doctors have
a hard time knowing whether something works
00:09:39.520 --> 00:09:44.960
and whether it should be prescribed.
And the evidence for knowing whether
00:09:44.960 --> 00:09:49.440
something works isn't just, well, you
know, Sally took it, or Dan took it,
00:09:49.440 --> 00:09:54.000
or Kathleen took it, and they seem to get
better or they didn't seem to get better.
00:09:54.000 --> 00:09:58.240
The really rigorous evidence comes from
randomized clinical trials. And I think
00:09:58.240 --> 00:10:03.280
it's fair to say that if you didn't
have the FDA there as a veto player,
00:10:03.280 --> 00:10:07.840
you wouldn't get as many randomized clinical
trials and the evidence probably wouldn't be
00:10:07.840 --> 00:10:12.800
as rigorous for whether these things work. And
as I like to put it, basically there's a whole
00:10:12.800 --> 00:10:18.800
ecology of expectations and beliefs around
the biopharmaceutical industry in the United
00:10:18.800 --> 00:10:24.960
States and globally, and to some extent, it's
undergirded by all of these tests that happen.
00:10:24.960 --> 00:10:25.810
SULLIVAN: Right.
00:10:25.810 --> 00:10:31.040
CARPENTER: And in part, that means it's
undergirded by regulation. Would there still be
00:10:31.040 --> 00:10:36.640
a market without regulation? Yes. But it would be
a market in which people had far less information
00:10:36.640 --> 00:10:43.840
in and confidence about the drugs that are being
taken. And so I think it's important to recognize
00:10:43.840 --> 00:10:51.760
that kind of confidence-boosting potential
of, kind of, a scientific regulation base.
00:10:51.760 --> 00:10:55.040
SULLIVAN: Actually, if we could double-click
on that for a minute, I'd love to hear your
00:10:55.040 --> 00:10:59.600
perspective on, testing has been completed;
there's results. Can you walk us through how
00:10:59.600 --> 00:11:05.360
those results actually shape the next steps and
decisions of a particular drug and just, like, how
00:11:05.360 --> 00:11:11.920
regulators actually think about using that data
to influence really what happens next with it?
00:11:11.920 --> 00:11:18.320
CARPENTER: Right. So it's important to
understand that every drug is approved
00:11:18.320 --> 00:11:23.840
for what's called an indication. It
can have a first primary indication,
00:11:23.840 --> 00:11:28.160
which is the main disease that it treats, and
then others can be added as more evidence is
00:11:28.160 --> 00:11:34.640
shown. But a drug is not something that just
kind of exists out there in the ether. It
00:11:34.640 --> 00:11:39.120
has to have the right form of administration.
Maybe it should be injected. Maybe it should be
00:11:39.120 --> 00:11:44.720
ingested. Maybe it should be administered only
at a clinic because it needs to be kind of
00:11:44.720 --> 00:11:50.720
administered in just the right way. As doctors
will tell you, dosage is everything, right.
00:11:50.720 --> 00:11:56.000
And so one of the reasons that you want
those trials is not simply a, you know,
00:11:56.000 --> 00:12:02.560
yes or no answer about whether the drug
works, right. It's not simply if-then. It's
00:12:02.560 --> 00:12:06.320
literally what goes into what you might
call the dose response curve. You know,
00:12:06.320 --> 00:12:11.600
how much of this drug do we need to basically,
you know, get the benefit? At what point does
00:12:11.600 --> 00:12:17.680
that fall off significantly that we can basically
say, we can stop there? All that evidence comes
00:12:17.680 --> 00:12:25.600
from trials. And that's the kind of evidence
that is required on the basis of regulation.
00:12:25.600 --> 00:12:30.080
Because it's not simply a drug
that's approved. It's a drug and
00:12:30.080 --> 00:12:36.000
a frequency of administration. It's a method
of administration. And so the drug isn't just,
00:12:36.000 --> 00:12:39.120
there's something to be taken off the
shelf and popped into your mouth. I mean,
00:12:39.120 --> 00:12:43.360
sometimes that's what happens, but even then,
we want to know what the dosage is, right.
00:12:44.160 --> 00:12:47.760
We want to know what to look for in
terms of side effects, things like that.
00:12:47.760 --> 00:12:49.440
SULLIVAN: Going back to that point, I mean,
00:12:49.440 --> 00:12:55.920
it sounds like we're making a lot of progress
from a regulation perspective in, you know,
00:12:55.920 --> 00:13:01.200
sort of speed and getting things approved but
doing it in a really balanced way. I mean,
00:13:01.200 --> 00:13:05.840
any other kind of closing thoughts on the
tradeoffs there or where you're seeing that going?
00:13:05.840 --> 00:13:10.560
CARPENTER: I think you're going to see some move
in the coming years—there's already been some of
00:13:10.560 --> 00:13:16.400
it—to say, do we always need a really large
Phase 3 clinical trial? And to what degree do
00:13:16.400 --> 00:13:21.040
we need the, like, you know, all the i's dotted
and the t's crossed or a really, really large
00:13:21.040 --> 00:13:29.920
sample size? And I'm open to innovation there.
I'm also open to the idea that we consider, again,
00:13:29.920 --> 00:13:37.520
things like accelerated approvals or pathways
for looking at different kinds of surrogate
00:13:37.520 --> 00:13:42.360
endpoints. I do think, once we do that, then
we also have to have some degree of follow-up.
00:13:42.360 --> 00:13:44.480
SULLIVAN: So I know we're
getting close to out of time,
00:13:44.480 --> 00:13:50.800
but maybe just a quick rapid fire if you’re
open to it. Biggest myth about clinical trials?
00:13:50.800 --> 00:13:54.640
CARPENTER: Well, some people tend to think
that the FDA performs them. You know,
00:13:54.640 --> 00:14:00.160
it's companies that do it. And the only other
thing I would say is the company that does a
00:14:00.160 --> 00:14:05.600
lot of the testing and even the innovating is not
always the company that takes the drug to market,
00:14:05.600 --> 00:14:12.240
and it tells you something about how powerful
regulation is in our system, in our world,
00:14:12.240 --> 00:14:18.640
that you often need a company that has dealt with
the FDA quite a bit and knows all the regulations
00:14:18.640 --> 00:14:23.760
and knows how to dot the i's and cross the t's
in order to get a drug across the finish line.
00:14:23.760 --> 00:14:27.920
SULLIVAN: If you had a magic wand, what's the
one thing you'd change in regulation today?
00:14:27.920 --> 00:14:32.960
CARPENTER: I would like people to think a little
bit less about just speed versus safety and,
00:14:32.960 --> 00:14:37.280
again, more about this basic issue of
confidence. I think it's fundamental
00:14:37.280 --> 00:14:41.200
to everything that happens in markets
but especially in biopharmaceuticals.
00:14:41.200 --> 00:14:45.821
SULLIVAN: Such a great point. This has been really
fun. Just thanks so much for being here today.
00:14:45.821 --> 00:14:46.358
[TRANSITION MUSIC]
00:14:46.358 --> 00:14:49.920
We're really excited to share your
thoughts out to our listeners. Thanks.
00:14:49.920 --> 00:14:54.160
CARPENTER: Likewise.
00:14:58.120 --> 00:14:59.120
SULLIVAN:
00:14:59.120 --> 00:15:04.480
Now to the world of medical devices, I'm joined
by Professor Timo Minssen. Professor Minssen,
00:15:04.480 --> 00:15:07.746
it's great to have you here.
Thank you for joining us today.
00:15:07.746 --> 00:15:09.720
TIMO MINSSEN: Yeah, thank you
very much, it's a pleasure.
00:15:09.720 --> 00:15:13.440
SULLIVAN: Before getting into the
regulatory world of medical devices,
00:15:13.440 --> 00:15:17.440
tell our audience a bit about your personal
journey or your origin story, as we're asking
00:15:17.440 --> 00:15:22.480
our guests. How did you land in regulation,
and what's kept you hooked in this space?
00:15:22.480 --> 00:15:28.160
MINSSEN: So I started out as a patent expert in
the biomedical area, starting with my PhD thesis
00:15:28.160 --> 00:15:34.240
on patenting biologics in Europe and in the US.
So during that time, I was mostly interested in
00:15:34.240 --> 00:15:39.200
patent and trade secret questions. But at the
same time, I also developed and taught courses
00:15:39.200 --> 00:15:45.680
in regulatory law and held talks on regulating
advanced medical therapy medicinal products.
00:15:45.680 --> 00:15:52.720
I then started to lead large research projects
on legal challenges in a wide variety of health
00:15:52.720 --> 00:15:57.840
and life science innovation frontiers. I also
started to focus increasingly on AI-enabled
00:15:57.840 --> 00:16:03.760
medical devices and software as a medical device,
resulting in several academic articles in this
00:16:03.760 --> 00:16:09.800
area and also in the regulatory area and a book
on the future of medical device regulation.
00:16:09.800 --> 00:16:12.800
SULLIVAN: Yeah, what's kept
you hooked in the space?
00:16:12.800 --> 00:16:15.600
MINSSEN: It's just incredibly exciting,
00:16:15.600 --> 00:16:22.880
in particular right now with everything that
is going on, you know, in the software arena,
00:16:22.880 --> 00:16:28.000
in the marriage between AI and medical
devices. And this is really challenging
00:16:28.000 --> 00:16:32.360
not only societies but also regulators
and authorities in Europe and in the US.
00:16:32.360 --> 00:16:35.760
SULLIVAN: Yeah, it's a super exciting
time to be in this space. You know,
00:16:35.760 --> 00:16:39.360
we talked to Daniel a little earlier and,
you know, I think similar to pharmaceuticals,
00:16:39.360 --> 00:16:45.280
people have a general sense of what we mean when
we say medical devices, but most listeners may
00:16:45.280 --> 00:16:52.240
picture like a stethoscope or a hip implant.
The word “medical device” reaches much wider.
00:16:52.240 --> 00:16:57.120
Can you give us a quick, kind of, range from
perhaps very simple to even, I don't know,
00:16:57.120 --> 00:17:03.040
sci-fi and then your 90-second tour of how risk
assessment works and why a framework is essential?
00:17:03.040 --> 00:17:06.800
MINSSEN: Let me start out by saying that the WHO
[World Health Organization] estimates that today
00:17:06.800 --> 00:17:11.840
there are approximately 2 million different
kinds of medical devices on the world market,
00:17:11.840 --> 00:17:19.360
and as of the FDA's latest update that I'm
aware of, the FDA has authorized more than 1,000
00:17:19.360 --> 00:17:24.720
AI-, machine learning-enabled medical
devices, and that number is rising rapidly.
00:17:24.720 --> 00:17:29.040
So in that context, I think it is important
to understand that medical devices can be
00:17:29.040 --> 00:17:35.520
any instrument, apparatus, implement, machine,
appliance, implant, reagent for in vitro use,
00:17:35.520 --> 00:17:42.080
software, material, or other similar or related
articles that are intended by the manufacturer to
00:17:42.080 --> 00:17:47.920
be used alone or in combination for a medical
purpose. And the spectrum of what constitutes
00:17:47.920 --> 00:17:52.800
a medical device can thus range from very
simple devices such as tongue depressors,
00:17:52.800 --> 00:17:58.240
contact lenses, and thermometers to more
complex devices such as blood pressure monitors,
00:17:58.240 --> 00:18:04.960
insulin pumps, MRI machines, implantable
pacemakers, and even software as a medical
00:18:04.960 --> 00:18:09.680
device or AI-enabled monitors or
drug device combinations, as well.
00:18:09.680 --> 00:18:13.520
So talking about regulation, I think
it is also very important to stress
00:18:13.520 --> 00:18:17.280
that medical devices are used in
many diverse situations by very
00:18:17.280 --> 00:18:22.480
different stakeholders. And testing has
to take this variety into consideration,
00:18:22.480 --> 00:18:28.560
and it is intrinsically tied to regulatory
requirements across various jurisdictions.
00:18:28.560 --> 00:18:33.120
During the pre-market phase, medical
testing establishes baseline safety
00:18:33.120 --> 00:18:37.120
and effectiveness metrics through
bench testing, performance standards,
00:18:37.120 --> 00:18:43.360
and clinical studies. And post-market testing
ensures that real-world data informs ongoing
00:18:43.360 --> 00:18:49.120
compliance and safety improvements. So testing
is indispensable in translating technological
00:18:49.120 --> 00:18:53.600
innovation into safe and effective
medical devices. And while particular
00:18:53.600 --> 00:18:58.960
details of pre-market and post-market review
procedures may slightly differ among countries,
00:18:58.960 --> 00:19:04.640
most developed jurisdictions regulate medical
devices similarly to the US or European models.
00:19:04.640 --> 00:19:09.840
So most jurisdictions with medical device
regulation classify devices based on their
00:19:09.840 --> 00:19:15.440
risk profile, intended use, indications
for use, technological characteristics,
00:19:15.440 --> 00:19:22.200
and the regulatory controls necessary to provide a
reasonable assurance of safety and effectiveness.
00:19:22.200 --> 00:19:27.280
SULLIVAN: So medical devices face
a pretty prescriptive multi-level
00:19:27.280 --> 00:19:30.320
testing path before they hit the
market. From your vantage point,
00:19:30.320 --> 00:19:34.120
what are some of the downsides of that
system and when does it make the most sense?
00:19:34.120 --> 00:19:39.440
MINSSEN: One primary drawback is, of course,
the lengthy and expensive approval process.
00:19:39.440 --> 00:19:44.240
High-risk devices, for example, often
undergo years of clinical trials,
00:19:44.240 --> 00:19:47.840
which can cost millions of dollars, and
this can create a significant barrier
00:19:47.840 --> 00:19:54.240
for startups and small companies with limited
resources. And even for moderate-risk devices,
00:19:54.240 --> 00:19:59.520
the regulatory burden can slow product
development and time to the market.
00:19:59.520 --> 00:20:05.760
And the approach can also limit flexibility.
Prescriptive requirements may not accommodate
00:20:05.760 --> 00:20:10.800
emerging innovations like digital therapeutics
or AI-based diagnostics in a feasible way. And
00:20:10.800 --> 00:20:16.240
in such cases, the framework can unintentionally
[stiffen] innovation by discouraging creative
00:20:16.240 --> 00:20:20.960
solutions or iterative improvements,
which as matter of fact can also put
00:20:20.960 --> 00:20:27.040
patients at risk when you don't use new
technologies and AI. And additionally,
00:20:27.040 --> 00:20:31.680
the same level of scrutiny may be
applied to low-risk devices, where
00:20:31.680 --> 00:20:37.680
the extensive testing and documentation may also
be disproportionate to the actual patient risk.
00:20:37.680 --> 00:20:42.320
However, the prescriptive model is
highly appropriate where we have high
00:20:42.320 --> 00:20:47.360
testing standards for high-risk medical
devices, in my view, particularly those
00:20:47.360 --> 00:20:52.960
that are life-sustaining, implanted,
or involve new materials or mechanisms.
00:20:52.960 --> 00:20:57.680
I also wanted to say that I think that
these higher compliance thresholds can
00:20:57.680 --> 00:21:03.520
be OK and necessary if you have a system
where authorities and stakeholders also
00:21:03.520 --> 00:21:10.320
have the capacity and funding to enforce,
monitor, and achieve compliance with such
00:21:10.320 --> 00:21:14.240
rules in a feasible, time-effective,
and straightforward manner. And this,
00:21:14.240 --> 00:21:18.480
of course, requires resources,
novel solutions, and investments.
00:21:18.480 --> 00:21:23.600
SULLIVAN: A range of tests are undertaken across
the life cycle of medical devices. How do these
00:21:23.600 --> 00:21:28.480
testing requirements vary across different stages
of development and across various applications?
00:21:28.480 --> 00:21:35.280
MINSSEN: Yes, that's a good question. So
I think first it is important to realize
00:21:35.280 --> 00:21:42.160
that testing is conducted by various entities,
including manufacturers, independent third-party
00:21:42.160 --> 00:21:49.120
laboratories, and regulatory agencies. And
it occurs throughout the device life cycle,
00:21:49.120 --> 00:21:54.000
beginning with iterative testing during the
research and development stage, advancing
00:21:54.000 --> 00:22:01.120
to pre-market evaluations, and continuing into
post-market monitoring. And the outcomes of these
00:22:01.120 --> 00:22:09.280
tests directly impact regulatory approvals, market
access, and device design refinements, as well.
00:22:09.840 --> 00:22:14.800
So the testing results are typically shared
with regulatory authorities and in some cases
00:22:14.800 --> 00:22:20.480
with healthcare providers and the broader
public to enhance transparency and trust.
00:22:20.480 --> 00:22:25.520
So if you talk about the different phases that
play a role here … so let's turn to the pre-market
00:22:25.520 --> 00:22:31.760
phase, where manufacturers must demonstrate
that the device is conformed to safety and
00:22:31.760 --> 00:22:38.560
performance benchmarks defined by regulatory
authorities. Pre-market evaluations include
00:22:38.560 --> 00:22:45.360
functional bench testing, biocompatibility, for
example, assessments and software validation,
00:22:45.360 --> 00:22:50.960
all of which are integral components
of a manufacturer's submission.
00:22:50.960 --> 00:22:55.520
But, yes, but, testing also, and
we touched already up on that,
00:22:55.520 --> 00:23:03.200
extends into the post-market phase, where it
continues to ensure device safety and efficacy,
00:23:03.200 --> 00:23:09.280
and post-market surveillance relies on testing
to monitor real-world performance and identify
00:23:09.280 --> 00:23:15.200
emerging risks on the post-market phase. By
integrating real-world evidence into ongoing
00:23:15.200 --> 00:23:22.560
assessments, manufacturers can address unforeseen
issues, update devices as needed, and maintain
00:23:22.560 --> 00:23:27.840
compliance with evolving regulatory expectations.
And I think this is particularly important in
00:23:27.840 --> 00:23:33.440
this new generation of medical devices that
are AI-enabled or machine-learning enabled.
00:23:33.440 --> 00:23:38.640
I think we have to understand that in this
AI-enabled medical devices field, you know,
00:23:38.640 --> 00:23:45.040
the devices and the algorithms that are working
with them, they can improve in the lifetime of a
00:23:45.040 --> 00:23:51.200
product. So actually, not only you could assess
them and make sure that they maintain safe,
00:23:51.200 --> 00:23:56.880
you could also sometimes lower the risk
category by finding evidence that these
00:23:56.880 --> 00:24:04.720
devices are actually becoming more precise and
safer. So it can both, you know, heighten the risk
00:24:04.720 --> 00:24:11.520
category or lower the risk category, and that's
why this continuous testing is so important.
00:24:11.520 --> 00:24:15.360
SULLIVAN: Given what you just said, how
should regulators handle a device whose
00:24:15.360 --> 00:24:18.480
algorithm keeps updating itself after approval?
00:24:18.480 --> 00:24:27.040
MINSSEN: Well, it has to be an iterative process
that is feasible and straightforward and that is
00:24:27.040 --> 00:24:33.520
based on a very efficient, both time efficient
and performance efficient, communication between
00:24:33.520 --> 00:24:39.760
the regulatory authorities and the medical device
developers, right. We need to have the sensors in
00:24:39.760 --> 00:24:50.240
place that spot potential changes, and we need
to have the mechanisms in place that allow us to
00:24:50.240 --> 00:24:57.440
quickly react to these changes both regulatory
wise and also in the technological way.
00:24:58.320 --> 00:25:04.880
So I think communication is important, and we need
to have the pathways and the feedback loops in the
00:25:04.880 --> 00:25:14.680
regulation that quickly allow us to monitor
these self-learning algorithms and devices.
00:25:14.680 --> 00:25:19.840
SULLIVAN: It sounds like it's just … there's such
a delicate balance between advancing technology
00:25:19.840 --> 00:25:26.000
and really ensuring public safety. You know, if
we clamp down too hard, we stifle that innovation.
00:25:26.000 --> 00:25:31.760
You already touched upon this a bit. But if we're
too lax, we risk unintended consequences. And I'd
00:25:31.760 --> 00:25:37.200
just love to hear how you think the field is
balancing that and any learnings you can share.
00:25:37.200 --> 00:25:41.920
MINSSEN: So this is very true, and you just
touched upon a very central question also
00:25:41.920 --> 00:25:47.440
in our research and our writing. And this is
also the reason why medical device regulation
00:25:47.440 --> 00:25:53.600
is so fascinating and continues to evolve in
response to rapid advancements in technologies,
00:25:53.600 --> 00:25:58.560
particularly dual technologies regarding
digital health, artificial intelligence,
00:25:58.560 --> 00:26:01.040
for example, and personalized medicine.
00:26:01.040 --> 00:26:05.360
And finding the balance is tricky because
also [a] related major future challenge
00:26:06.160 --> 00:26:10.880
relates to the increasing regulatory
jungle and the complex interplay between
00:26:10.880 --> 00:26:16.640
evolving regulatory landscapes
that regulate AI more generally.
00:26:16.640 --> 00:26:20.960
We really need to make sure that the regulatory
authorities that deal with this, that need to
00:26:20.960 --> 00:26:27.200
find the right balance to promote innovation
and mitigate and prevent risks, need to have
00:26:27.200 --> 00:26:32.560
the capacity to do this. So this requires
investments, and it also requires new ways
00:26:32.560 --> 00:26:39.960
to regulate this technology more flexibly, for
example through regulatory sandboxes and so on.
00:26:39.960 --> 00:26:43.600
SULLIVAN: Could you just expand upon
that a bit and double-click on what
00:26:43.600 --> 00:26:47.360
it is you're seeing there? What excites
you about what's happening in that space?
00:26:47.360 --> 00:26:53.600
MINSSEN: Yes, well, the research of my group at
the Center for Advanced Studies in Bioscience
00:26:53.600 --> 00:27:01.680
Innovation Law is very broad. I mean, we are
looking into gene editing technologies. We are
00:27:01.680 --> 00:27:09.520
looking into new biologics. We are looking
into medical devices, as well, obviously,
00:27:09.520 --> 00:27:12.800
but also other technologies
in advanced medical computing.
00:27:12.800 --> 00:27:20.400
And what we see across the line here is that
there is an increasing demand for having
00:27:20.400 --> 00:27:26.000
more adaptive and flexible regulatory
frameworks in these new technologies,
00:27:26.000 --> 00:27:31.520
in particular when they have new uses,
regulations that are focusing more on
00:27:31.520 --> 00:27:37.520
the product rather than the process. And I
have recently written a report, for example,
00:27:37.520 --> 00:27:44.720
for emerging biotechnologies and bio-solutions
for the EU commission. And even in that area,
00:27:44.720 --> 00:27:50.160
regulatory sandboxes are increasingly
important, increasingly considered.
00:27:50.160 --> 00:27:56.480
So this idea of regulatory sandboxes has been
developing originally in the financial sector,
00:27:56.480 --> 00:28:02.560
and it is now penetrating into other
sectors, including synthetic biology,
00:28:03.360 --> 00:28:08.720
emerging biotechnologies, gene
editing, AI, quantum technology,
00:28:08.720 --> 00:28:15.200
as well. This is basically creating an
environment where actors can test new
00:28:15.200 --> 00:28:21.520
ideas in close collaboration and under
the oversight of regulatory authorities.
00:28:21.520 --> 00:28:29.680
But to implement this in the AI sector now also
leads us to a lot of questions and challenges.
00:28:29.680 --> 00:28:37.280
For example, you need to have the capacities of
authorities that are governing and monitoring and
00:28:37.280 --> 00:28:43.360
deciding on these regulatory sandboxes. There are
issues relating to competition law, for example,
00:28:43.360 --> 00:28:49.680
which you call antitrust law in the US, because
the question is, who can enter the sandbox and how
00:28:49.680 --> 00:28:57.200
may they compete after they exit the sandbox?
And there are many questions relating to,
00:28:57.200 --> 00:29:03.411
how should we work with these sandboxes and
how should we implement these sandboxes?
00:29:03.411 --> 00:29:04.055
[TRANSITION MUSIC]
00:29:04.055 --> 00:29:06.520
SULLIVAN: Well, Timo, it has just been
such a pleasure to speak with you today.
00:29:06.520 --> 00:29:08.560
MINSSEN: Yes, thank you very much.
00:29:16.800 --> 00:29:19.600
And now I'm happy to introduce Chad Atalla.
00:29:19.600 --> 00:29:23.760
Chad is senior applied scientist in
Microsoft Research New York City's
00:29:23.760 --> 00:29:28.800
Sociotechnical Alignment Center, where they
contribute to foundational responsible AI
00:29:28.800 --> 00:29:33.280
research and practical responsible AI
solutions for teams across Microsoft.
00:29:33.280 --> 00:29:34.627
Chad, welcome!
00:29:34.627 --> 00:29:35.200
CHAD ATALLA: Thank you.
00:29:35.200 --> 00:29:40.480
SULLIVAN: So we'll kick off with a couple
questions just to dive right in. So tell me a
00:29:40.480 --> 00:29:46.240
little bit more about the Sociotechnical Alignment
Center, or STAC? I know it was founded in 2022.
00:29:46.240 --> 00:29:50.160
I'd love to just learn a little bit more about
what the group does, how you're thinking about
00:29:50.160 --> 00:29:54.320
evaluating AI, and maybe just give us a sense
of some of the projects you're working on.
00:29:54.320 --> 00:29:57.486
ATALLA: Yeah, absolutely.
The name is quite a mouthful.
00:29:57.486 --> 00:29:58.044
SULLIVAN: It is! [LAUGHS]
00:29:58.044 --> 00:30:01.000
ATALLA: So let's start by breaking
that down and seeing what that means.
00:30:01.000 --> 00:30:01.943
SULLIVAN: Great.
00:30:01.943 --> 00:30:05.360
ATALLA: So modern AI systems
are sociotechnical systems,
00:30:05.360 --> 00:30:10.160
meaning that the social and technical
aspects are deeply intertwined. And
00:30:10.160 --> 00:30:16.560
we're interested in aligning the behaviors of
these sociotechnical systems with some values.
00:30:16.560 --> 00:30:21.520
Those could be societal values; they could
be regulatory values, organizational values,
00:30:21.520 --> 00:30:28.560
etc. And to make this alignment happen, we
need the ability to evaluate the systems.
00:30:28.560 --> 00:30:34.640
So my team is broadly working on an evaluation
framework that acknowledges the sociotechnical
00:30:34.640 --> 00:30:40.640
nature of the technology and the often-abstract
nature of the concepts we're actually interested
00:30:40.640 --> 00:30:46.800
in evaluating. As you noted, it's an applied
science team, so we split our time between some
00:30:46.800 --> 00:30:54.320
fundamental research and time to bridge the
work into real products across the company.
00:30:54.320 --> 00:30:59.760
And I also want to note that to power this
sort of work, we have an interdisciplinary team
00:30:59.760 --> 00:31:04.320
drawing upon the social sciences, linguistics,
statistics, and, of course, computer science.
00:31:04.320 --> 00:31:08.560
SULLIVAN: Well, I'm eager to get into our
takeaways from the conversation with both
00:31:08.560 --> 00:31:12.240
Daniel and Timo. But maybe just to
double-click on this for a minute,
00:31:12.240 --> 00:31:16.960
can you talk a bit about some of the overarching
goals of the AI evaluations that you noted?
00:31:16.960 --> 00:31:25.040
ATALLA: So evaluation is really the act of making
valuative judgments based on some evidence,
00:31:25.040 --> 00:31:32.160
and in the case of AI evaluation, that evidence
might be from tests or measurements, right. And
00:31:32.160 --> 00:31:38.240
the goal of why we're doing this in the first
place is to make decisions and claims most often.
00:31:38.240 --> 00:31:44.080
So perhaps I am going to make a claim about
a model that I'm producing, and I want to say
00:31:44.080 --> 00:31:50.560
that it's better than this other model. Or we are
asking whether a certain product is safe to ship.
00:31:50.560 --> 00:31:57.360
All of these decisions need to be informed by
good evaluation and therefore good measurement
00:31:57.360 --> 00:32:05.040
or testing. And I'll also note that in the
regulatory conversation, risk is often what
00:32:05.040 --> 00:32:11.160
we want to evaluate. So that is a goal in and
of itself. And I'll touch more on that later.
00:32:11.160 --> 00:32:15.280
SULLIVAN: I read a recent paper that you
had put out with some of our colleagues
00:32:15.280 --> 00:32:19.040
from Microsoft Research, from the
University of Michigan, and Stanford,
00:32:19.040 --> 00:32:23.360
and you were arguing that evaluating
generative AI is the social-science
00:32:23.360 --> 00:32:27.120
measurement challenge. Maybe for those
who haven't read the paper, what does this
00:32:27.120 --> 00:32:31.200
mean? And can you tell us a little bit more
about what motivated you and your coauthors?
00:32:31.200 --> 00:32:37.280
ATALLA: So the measurement tasks involved in
evaluating generative AI systems are often
00:32:37.280 --> 00:32:42.400
abstract and contested. So that means
they cannot be directly measured and
00:32:42.400 --> 00:32:48.400
must instead [be] indirectly measured via other
observable phenomena. So this is very different
00:32:48.400 --> 00:32:54.080
than the older machine learning paradigm, where,
let's say, for example, I had a system that took a
00:32:54.080 --> 00:32:59.440
picture of a traffic light and told you whether
it was green, yellow, or red at a given time.
00:32:59.440 --> 00:33:05.760
If we wanted to evaluate that system, the task is
much simpler. But with the modern generative AI
00:33:05.760 --> 00:33:15.200
systems that are also general purpose, they have
open-ended output, and language in a whole chat or
00:33:15.200 --> 00:33:20.480
multiple paragraphs being outputted can have a lot
of different properties. And as I noted, these are
00:33:20.480 --> 00:33:25.920
general-purpose systems, so we don't know exactly
what task they're supposed to be carrying out.
00:33:25.920 --> 00:33:32.640
So then the question becomes, if I want to make
some decision or claim—maybe I want to make a
00:33:32.640 --> 00:33:41.200
claim that this system has human-level reasoning
capabilities—well, what does that mean? Do I have
00:33:41.200 --> 00:33:48.320
the same impression of what that means as you
do? And how do we know whether the downstream,
00:33:48.320 --> 00:33:53.520
you know, measurements and tests that I'm
conducting actually will support my notion of
00:33:53.520 --> 00:33:59.920
what it means to have human-level reasoning,
right? Difficult questions. But luckily,
00:33:59.920 --> 00:34:03.840
social scientists have been dealing with these
exact sorts of challenges for multiple decades
00:34:03.840 --> 00:34:09.920
in fields like education, political science, and
psychometrics. So we're really attempting to avoid
00:34:09.920 --> 00:34:16.000
reinventing the wheel here and trying
to learn from their past methodologies.
00:34:16.000 --> 00:34:21.200
And so the rest of the paper goes on to delve into
a four-level framework, a measurement framework,
00:34:21.200 --> 00:34:26.320
that's grounded in the measurement theory from
the quantitative social sciences that takes us
00:34:26.320 --> 00:34:32.960
all the way from these abstract and contested
concepts through processes to get much clearer
00:34:32.960 --> 00:34:37.320
and eventually reach reliable and valid
measurements that can power our evaluations.
00:34:37.320 --> 00:34:40.400
SULLIVAN: I love that. I mean, that's
the whole point of this podcast, too,
00:34:40.400 --> 00:34:44.320
right. Is to really build on those other
learnings and frameworks that we're taking
00:34:44.320 --> 00:34:49.760
from industries that have been thinking about this
for much longer. Maybe from your vantage point,
00:34:49.760 --> 00:34:54.880
what are some of the biggest day-to-day
hurdles in building solid AI evaluations and,
00:34:54.880 --> 00:34:59.040
I don't know, do we need more shared
standards? Are there bespoke methods?
00:34:59.040 --> 00:35:01.680
Are those the way to go? I would love
to just hear your thoughts on that.
00:35:01.680 --> 00:35:06.800
ATALLA: So let's talk about some of those
practical challenges. And I want to briefly
00:35:06.800 --> 00:35:12.400
go back to what I mentioned about risk before,
all right. Oftentimes, some of the regulatory
00:35:12.400 --> 00:35:18.640
environment is requiring practitioners to measure
the risk involved in deploying one of their
00:35:18.640 --> 00:35:27.200
models or AI systems. Now, risk is importantly
a concept that includes both event and impact,
00:35:27.200 --> 00:35:33.280
right. So there's the probability of some event
occurring. For the case of AI evaluation, perhaps
00:35:33.280 --> 00:35:41.520
this is us seeing a certain AI behavior exhibited.
Then there's also the severity of the impacts,
00:35:41.520 --> 00:35:47.840
and this is a complex chain of effects
in the real world that happen to people,
00:35:47.840 --> 00:35:56.640
organizations, systems, etc., and it's a lot
more challenging to observe the impacts, right.
00:35:56.640 --> 00:36:03.280
So if we're saying that we need to measure
risk, we have to measure both the event and
00:36:03.280 --> 00:36:09.280
the impacts. But realistically, right now,
the field is not doing a very good job of
00:36:09.280 --> 00:36:14.320
actually measuring the impacts. This requires
vastly different techniques and methodologies
00:36:14.320 --> 00:36:19.120
where if I just wanted to measure something
about the event itself, I can, you know,
00:36:19.120 --> 00:36:26.160
do that in a technical sandbox environment and
perhaps have some automated methods to detect
00:36:26.160 --> 00:36:30.720
whether a certain AI behavior is being exhibited.
But if I want to measure the impacts? Now,
00:36:30.720 --> 00:36:35.360
we're in the realm of needing to have real people
involved, and perhaps a longitudinal study where
00:36:35.360 --> 00:36:39.520
you have interviews, questionnaires,
and more qualitative evidence-gathering
00:36:39.520 --> 00:36:47.200
techniques to truly understand the long-term
impacts. So that's a significant challenge.
00:36:47.200 --> 00:36:52.560
Another is that, you know, let's say we forget
about the impacts for now and we focus on the
00:36:52.560 --> 00:37:00.560
event side of things. Still, we need datasets, we
need annotations, and we need metrics to make this
00:37:00.560 --> 00:37:08.000
whole thing work. When I say we need datasets,
if I want to test whether my system has good
00:37:08.000 --> 00:37:14.240
mathematical reasoning, what questions should I
ask? What are my set of inputs that are relevant?
00:37:14.240 --> 00:37:18.800
And then when I get the response from the system,
how do I annotate them? How do I know if it was a
00:37:18.800 --> 00:37:23.840
good response that did demonstrate mathematical
reasoning or if it was a mediocre response?
00:37:24.640 --> 00:37:29.680
And then once I have an annotation of
all of these outputs from the AI system,
00:37:29.680 --> 00:37:33.800
how do I aggregate those all up
into a single informative number?
00:37:33.800 --> 00:37:38.320
SULLIVAN: Earlier in this episode, we
heard Daniel and Timo walk through the
00:37:38.320 --> 00:37:42.960
regulatory frameworks in pharma and medical
devices. I'd be curious what pieces of those
00:37:42.960 --> 00:37:47.240
mature systems are already showing up or at
least may be bubbling up in AI governance.
00:37:47.240 --> 00:37:53.200
ATALLA: Great question. You know, Timo was
talking about the pre-market and post-market
00:37:53.200 --> 00:38:00.720
testing difference. Of course, this is similarly
important in the AI evaluation space. But again,
00:38:00.720 --> 00:38:04.720
these have different methodologies
and serve different purposes.
00:38:04.720 --> 00:38:12.400
So within the pre-deployment phase, we
don't have evidence of how people are
00:38:12.400 --> 00:38:16.560
going to use the system. And when we
have these general-purpose AI systems,
00:38:16.560 --> 00:38:22.320
to understand what the risks are, we really
need to have a sense of what might happen
00:38:22.320 --> 00:38:26.640
and how they might be used. So there are
significant challenges there where I think
00:38:26.640 --> 00:38:33.680
we can learn from other fields and how they do
pre-market testing. And the difference in that
00:38:33.680 --> 00:38:38.800
pre- versus post-market testing also ties to
testing at different stages in the life cycle.
00:38:38.800 --> 00:38:45.920
For AI systems, we already see some regulations
saying you need to start with the base model
00:38:45.920 --> 00:38:52.160
and do some evaluation of the base model,
some basic attributes, some core attributes,
00:38:52.160 --> 00:38:57.760
of that base model before you start putting
it into any real products. But once we have
00:38:57.760 --> 00:39:03.520
a product in mind, we have a user base in mind,
we have a specific task—like maybe we're going
00:39:03.520 --> 00:39:07.440
to integrate this model into Outlook and
it's going to help you write emails—now
00:39:07.440 --> 00:39:12.320
we suddenly have a much crisper picture of
how the system will interact with the world
00:39:12.320 --> 00:39:18.400
around it. And again, at that stage, we need
to think about another round of evaluation.
00:39:18.400 --> 00:39:23.680
Another part that jumped out to me in what
they were saying about pharmaceuticals is
00:39:23.680 --> 00:39:28.080
that sometimes approvals can be
based on surrogate endpoints. So
00:39:28.080 --> 00:39:33.120
this is like we're choosing some heuristic.
Instead of measuring the long-term impact,
00:39:33.120 --> 00:39:36.880
which is what we actually care about,
perhaps we have a proxy that we feel
00:39:36.880 --> 00:39:42.960
like is a good enough indicator of what
that long-term impact might look like.
00:39:42.960 --> 00:39:50.480
This is occurring in the AI evaluation space
right now and is often perhaps even the default
00:39:50.480 --> 00:39:56.320
here since we're not seeing that many studies
of the long-term impact itself. We are seeing,
00:39:56.320 --> 00:40:03.600
instead, folks constructing these heuristics or
proxies and saying if I see this behavior happen,
00:40:03.600 --> 00:40:09.440
I'm going to assume that it indicates this
sort of impact will happen downstream. And
00:40:09.440 --> 00:40:16.480
that's great. It's one of the techniques that
was used to speed up and reduce the barrier
00:40:16.480 --> 00:40:22.320
to innovation in the other fields. And I think
it's great that we are applying that in the AI
00:40:22.320 --> 00:40:27.920
evaluation space. But special care is, of course,
needed to ensure that those heuristics and proxies
00:40:27.920 --> 00:40:33.280
you're using are reasonable indicators of
the greater outcome you're looking for.
00:40:33.280 --> 00:40:38.880
SULLIVAN: What are some of the promising ideas
from maybe pharma or med device regulation
00:40:38.880 --> 00:40:43.360
that maybe haven't made it to AI testing
yet and maybe should? And where would you
00:40:43.360 --> 00:40:47.440
urge technologists, policymakers, and
researchers to focus their energy next?
00:40:47.440 --> 00:40:51.200
ATALLA: Well, one of the key things
that jumped out to me in the discussion
00:40:51.200 --> 00:40:58.240
about pharmaceuticals was driving home the
emphasis that there is a holistic focus on
00:40:58.240 --> 00:41:04.800
safety and efficacy. These go hand in hand and
decisions must be made while considering both
00:41:04.800 --> 00:41:11.360
pieces of the picture. I would like to see that
further emphasized in the AI evaluation space.
00:41:11.360 --> 00:41:18.080
Often, we are seeing evaluations of
risk being separated from evaluations
00:41:18.080 --> 00:41:27.440
of performance or quality or efficacy, but
these two pieces of the puzzle really are
00:41:27.440 --> 00:41:33.760
not enough for us to make informed
decisions independently. And that
00:41:33.760 --> 00:41:41.040
ties back into my desire to really
also see us measuring the impacts.
00:41:41.040 --> 00:41:47.920
So we see Phase 3 trials as something that occurs
in the medical devices and pharmaceuticals field.
00:41:47.920 --> 00:41:52.640
That's not something that we are doing an
equivalent of in the AI evaluation space
00:41:52.640 --> 00:41:59.440
at this time. These are really cost intensive.
They can last years and really involve careful
00:41:59.440 --> 00:42:05.840
monitoring of that holistic picture of
safety and efficacy. And realistically,
00:42:05.840 --> 00:42:10.560
we are not going to be able to put that
on the critical path to getting specific
00:42:10.560 --> 00:42:16.400
individual AI models or AI systems vetted
before they go out into the world. However,
00:42:16.400 --> 00:42:24.800
I would love to see a world in which this sort
of work is prioritized and funded or required.
00:42:24.800 --> 00:42:31.360
Think of how, with social media, it took quite
a long time for us to understand that there are
00:42:31.360 --> 00:42:38.720
some long-term negative impacts on mental health,
and we have the opportunity now, while the AI wave
00:42:38.720 --> 00:42:44.880
is still building, to start prioritizing and
funding this sort of work. Let it run in the
00:42:44.880 --> 00:42:52.960
background and as soon as possible develop a good
understanding of the subtle, long-term effects.
00:42:52.960 --> 00:42:59.040
More broadly, I would love to see us focus on
reliability and validity of the evaluations we're
00:42:59.040 --> 00:43:06.240
conducting because trust in these decisions
and claims is important. If we don't focus
00:43:06.240 --> 00:43:11.760
on building reliable, valid, and trustworthy
evaluations, we're just going to continue to be
00:43:11.760 --> 00:43:18.320
flooded by a bunch of competing, conflicting,
and largely meaningless AI evaluations.
00:43:18.320 --> 00:43:22.480
SULLIVAN: In a number of the discussions
we've had on this podcast, we talked about
00:43:22.480 --> 00:43:28.000
how it's not just one entity that really needs
to ensure safety across the board, and I’d just
00:43:28.000 --> 00:43:33.040
love to hear from you how you think about some
of those ecosystem collaborations, and you know,
00:43:33.040 --> 00:43:38.800
from across … where we think about ourselves
as more of a platform company or places that
00:43:38.800 --> 00:43:43.760
these AI models are being deployed more at the
application level. Tell me a little bit about how
00:43:43.760 --> 00:43:49.400
you think about, sort of, stakeholders in that mix
and where responsibility lies across the board.
00:43:49.400 --> 00:43:54.560
ATALLA: It's interesting. In this age
of general-purpose AI technologies,
00:43:54.560 --> 00:44:00.080
we're often seeing one company or
organization being responsible for
00:44:00.080 --> 00:44:05.440
building the foundational model. And then
many, many other people will take that
00:44:05.440 --> 00:44:11.120
model and build it into specific products that
are designed for specific tasks and contexts.
00:44:11.120 --> 00:44:17.440
Of course, in that, we already see that
there is a responsibility of the owners
00:44:17.440 --> 00:44:22.560
of that foundational model to do some
testing of the central model before
00:44:22.560 --> 00:44:27.120
they distribute it broadly. And then
again, there is responsibility of all
00:44:27.120 --> 00:44:34.320
of the downstream individuals digesting that
and turning it into products to consider the
00:44:34.320 --> 00:44:39.520
specific contexts that they are deploying
into and how that may affect the risks we're
00:44:39.520 --> 00:44:47.040
concerned with or the types of quality and
safety and performance we need to evaluate.
00:44:47.040 --> 00:44:52.640
Again, because that field of risks
we may be concerned with is so broad,
00:44:52.640 --> 00:44:59.680
some of them also require an immense amount of
expertise. Let's think about whether AI systems
00:44:59.680 --> 00:45:06.640
can enable people to create dangerous chemicals
or dangerous weapons at home. It's not that every
00:45:06.640 --> 00:45:12.640
AI practitioner is going to have the knowledge
to evaluate this, so in some of those cases,
00:45:12.640 --> 00:45:19.600
we really need third-party experts, people
who are experts in chemistry, biology,
00:45:19.600 --> 00:45:26.800
etc., to come in and evaluate certain systems
and models for those specific risks, as well.
00:45:26.800 --> 00:45:31.680
So I think there are many reasons why
multiple stakeholders need to be involved,
00:45:31.680 --> 00:45:36.400
partly from who owns what and is responsible
for what and partly from the perspective of
00:45:36.400 --> 00:45:42.200
who has the expertise to meaningfully
construct the evaluations that we need.
00:45:42.200 --> 00:45:47.360
SULLIVAN: Well, Chad, this has just been great
to connect, and in a few of our discussions,
00:45:47.360 --> 00:45:53.040
we've done a bit of a lightning round, so I'd
love to just hear your 30-second responses
00:45:53.040 --> 00:45:58.400
to a few of these questions. Perhaps favorite
evaluation you've run so far this year?
00:45:58.400 --> 00:46:04.560
ATALLA: So I've been involved in trying to
evaluate some language models for whether
00:46:04.560 --> 00:46:10.720
they infer sensitive attributes about people.
So perhaps you're chatting with a chatbot,
00:46:10.720 --> 00:46:17.440
and it infers your religion or sexuality based
on things you're saying or how you sound, right.
00:46:17.440 --> 00:46:23.120
And in working to evaluate this, we encounter a
lot of interesting questions. Or, like, what is
00:46:23.120 --> 00:46:28.560
a sensitive attribute? What makes these attributes
sensitive, and what are the differences that make
00:46:28.560 --> 00:46:33.840
it inappropriate for an AI system to infer these
things about a person? Whereas realistically,
00:46:33.840 --> 00:46:39.920
whenever I meet a person on the street, my brain
is immediately forming first impressions and some
00:46:39.920 --> 00:46:45.600
assumptions about these people. So it's a very
interesting and thought-provoking evaluation to
00:46:46.240 --> 00:46:51.840
conduct and think about the norms that
we place upon people interacting with
00:46:51.840 --> 00:46:56.120
other people and the norms we place upon
AI systems interacting with other people.
00:46:56.120 --> 00:47:02.144
SULLIVAN: That’s fascinating! I'd love to hear
the AI buzzword you'd retire tomorrow. [LAUGHTER]
00:47:02.144 --> 00:47:08.320
ATALLA: I would love to see the term
“bias” being used less when referring
00:47:08.320 --> 00:47:15.440
to fairness-related issues and systems.
Bias happens to be a highly overloaded
00:47:15.440 --> 00:47:20.960
term in statistics and machine learning
and has a lot of technical meanings and
00:47:20.960 --> 00:47:26.800
just fails to perfectly capture
what we mean in the AI risk sense.
00:47:26.800 --> 00:47:30.280
SULLIVAN: And last one. One
metric we're not tracking enough.
00:47:30.280 --> 00:47:39.440
ATALLA: I would say over-blocking, and this comes
into that connection between the holistic picture
00:47:39.440 --> 00:47:47.440
of safety and efficacy. It's too easy to produce
systems that throw safety to the wind and focus
00:47:47.440 --> 00:47:52.560
purely on utility or achieving some goal, but
simultaneously, the other side of the picture
00:47:52.560 --> 00:47:59.760
is possible, where we can clamp down too hard
and reduce the utility of our systems and block
00:47:59.760 --> 00:48:07.520
even benign and useful outputs just because they
border on something sensitive. So it's important
00:48:07.520 --> 00:48:13.680
for us to track that over-blocking and actively
track that tradeoff between safety and efficacy.
00:48:13.680 --> 00:48:18.480
SULLIVAN: Yeah, we talk a lot about this on the
podcast, too, of how do you both make things safe
00:48:18.480 --> 00:48:24.320
but also ensure innovation can thrive, and I think
you hit the nail on the head with that last piece.
00:48:24.320 --> 00:48:25.158
[MUSIC]
00:48:25.158 --> 00:48:30.290
Well, Chad, this was really terrific. Thanks
for joining us and thanks for your work and your
00:48:30.290 --> 00:48:33.656
perspectives. And another big thanks to Daniel and
Timo for setting the stage earlier in the podcast.
00:48:33.656 --> 00:48:38.755
And to our listeners, thanks for tuning
in. You can find resources related to
00:48:38.755 --> 00:48:41.440
this podcast in the show notes. And
if you want to learn more about how
00:48:41.440 --> 00:48:47.360
Microsoft approaches AI governance,
you can visit microsoft.com/RAI.
00:48:47.360 --> 00:49:03.440
See you next time!
00:49:03.440 --> 00:49:04.334
[MUSIC FADES]
