#62 - What Is Drift? (And Why We Need To Keep A Close Eye)

WEBVTT
00:00:00.160 --> 00:00:03.520
Your clinical AI model past validation is great.
00:00:03.759 --> 00:00:09.599
But what happens when the patients change, or the workflows change, or the clinical practice changes?
00:00:09.839 --> 00:00:16.079
That's drift, and monitoring it may be the real work of AI safety.
00:00:24.800 --> 00:00:30.320
Hello and welcome back to Coding Cure, the podcast where we discuss decoding health and the age of AI.
00:00:30.480 --> 00:00:32.000
My name is Vasant Sarati.
00:00:32.240 --> 00:00:38.880
I'm an AI researcher and cognitive scientist, and I'm here as always with Laura Hagopian, an emergency medicine physician.
00:00:39.359 --> 00:00:39.759
All right.
00:00:39.840 --> 00:00:47.280
So we've talked a lot in all of our episodes about making sure AI is safe, having humans in the loop.
00:00:47.359 --> 00:00:51.359
And a lot of that has been about building better AI models, right?
00:00:51.520 --> 00:00:56.399
Making them better, more accurate, making them more trustworthy.
00:00:56.560 --> 00:01:00.640
And that has more to do with the actual building of it and the training of it.
00:01:01.119 --> 00:01:03.600
And sometimes even the interaction and the use of it.
00:01:03.840 --> 00:01:12.319
But today we're going to talk about something which is okay, so you've built this great model and you put it out there in some real world setting.
00:01:12.480 --> 00:01:13.040
What then?
00:01:13.120 --> 00:01:13.840
What happens then?
00:01:14.079 --> 00:01:15.120
Yeah, you've launched it.
00:01:15.200 --> 00:01:16.239
How exciting, right?
00:01:16.400 --> 00:01:16.719
Yeah.
00:01:16.959 --> 00:01:19.359
And I think the point is that the work's not over.
00:01:19.760 --> 00:01:21.760
Yeah, and work's just beginning, really.
00:01:22.239 --> 00:01:24.319
I I think engineers understand that.
00:01:24.400 --> 00:01:33.599
And people who make product understand that because for them, they want to make sure that the customer experience is as good as it was when they first sold the idea to the customer.
00:01:33.840 --> 00:01:42.239
They want to make sure that they can track and log the interactions if there are any such interactions between the AI and any other customer and so on.
00:01:42.480 --> 00:01:46.879
But that's in a kind of a narrow setting where you have the sort of user-facing AI.
00:01:46.959 --> 00:01:59.359
But a lot of other AI tools are kind of sitting quietly working away, doing things, classifying things, especially in the medical setting where it's like embedded an AI medical device might be embedded somewhere, and that's it's not like a chat bot necessarily.
00:01:59.840 --> 00:02:02.079
So the user experience isn't the thing to be tracked.
00:02:02.239 --> 00:02:08.080
And so logging its outputs and making sure that the user's happy with it is not a relevant metric anymore.
00:02:08.319 --> 00:02:11.039
But in those cases, too, we might need to be monitoring, right?
00:02:11.120 --> 00:02:13.280
I mean, the point is the work isn't done.
00:02:13.599 --> 00:02:14.479
The work isn't done.
00:02:14.719 --> 00:02:19.360
Like these things need maintenance and they need people looking at them ahead of time, right?
00:02:19.520 --> 00:02:25.919
It's not like you, you know, improvise a plan after something went wrong.
00:02:26.080 --> 00:02:33.199
Like you should know, hey, this is this could go wrong in these ways, and here's how we plan to monitor it so we can intervene, right?
00:02:33.759 --> 00:02:37.120
And that's not, I think it's it's difficult.
00:02:37.439 --> 00:02:42.400
It's easy to say that, like, oh, we should monitor everything and make sure it's working according to plan.
00:02:42.639 --> 00:02:49.919
It's way harder to implement, but if you have like a framework that you can structure the thinking around, it gets a little bit easier.
00:02:50.159 --> 00:02:50.319
Yes.
00:02:50.719 --> 00:02:51.599
No, exactly, exactly.
00:02:51.680 --> 00:02:55.919
And I think that's kind of where this paper comes in, is that it that provides that kind of framework.
00:02:56.159 --> 00:02:56.400
Yeah.
00:02:56.479 --> 00:02:59.840
So they talk about like three different kinds of drift.
00:02:59.919 --> 00:03:09.680
Well, actually, before we before we like go into this whole drift conversation from a from like an AI technical standpoint, how would you define drift?
00:03:10.159 --> 00:03:10.560
Yes.
00:03:10.800 --> 00:03:31.439
So I I'd like to think about this in a very simple setting, which is supposing you have an AI model, which is not even an AI model, you have a machine learning model, which is given a bunch of data about your height and maybe your uh whether or not you smoke and all those things, and it's meant to predict what you're the likelihood of you getting diabetes five years from now, right?
00:03:31.599 --> 00:03:33.520
Let's just assume that's a simple, straightforward model.
00:03:33.599 --> 00:03:35.599
I mean when I say simple, it's not actually simple.
00:03:35.680 --> 00:03:38.879
I I get that there is a a host of complexity to even predicting that.
00:03:39.199 --> 00:03:40.479
But we're just doing like an A to B.
00:03:40.719 --> 00:03:43.120
Yeah, A to B, but the A to B is key, it's X to Y.
00:03:43.360 --> 00:03:49.599
You have some inputs, X, Y, X, X, X, you know, the the the age and the and the height and the you know the smoke.
00:03:51.199 --> 00:03:55.360
The inputs the covariates, how the fancy words covariates of features.
00:03:56.879 --> 00:03:57.919
I just felt so fancy.
00:03:58.240 --> 00:04:01.520
Yeah, you know, the other fancy word from machine learning perspective is features.
00:04:01.599 --> 00:04:09.120
That's the other thing they use, is is the things that we care about as we think as being relevant as part of the input in order to predict the output.
00:04:09.280 --> 00:04:12.240
And the output is things like the diabetes prediction, which is the y.
00:04:12.560 --> 00:04:14.479
So you have the X and you have the Y.
00:04:15.039 --> 00:04:18.000
And machine learning models are data driven.
00:04:18.160 --> 00:04:34.160
So if you went back in time a little bit, and I'm going to do a little bit of a history lesson, um, which is that if you went back in time to the 1980s, uh before the machine learning revolution, the X to Y problem was solved by writing computer programs to go from X to Y.
00:04:34.319 --> 00:04:45.120
And an expert user would provide knowledge about what it means to have a person of a certain height and a certain smoking pattern to also be diabetic.
00:04:45.199 --> 00:04:50.079
Maybe they have some rules about it, and they would hard code or put those rules into a computer program.
00:04:50.160 --> 00:04:59.199
And then every time the data came in, you just run it through the computer program that the human drafted, and out comes the answer, whether or not something, you know, whatever you cared about, right?
00:04:59.279 --> 00:04:59.920
Whatever the output was.
00:05:01.439 --> 00:05:01.680
Or whatever.
00:05:02.319 --> 00:05:02.399
Yep.
00:05:02.560 --> 00:05:07.839
So but the machine learning model came around and said, hey, you know, we we in a lot of cases can't do that.
00:05:08.000 --> 00:05:11.439
It's really hard to have rules for everything because there's exceptions to everything.
00:05:11.680 --> 00:05:18.639
And because you have exceptions to everything, you also don't know um how these features all our covariates all kind of work together.
00:05:18.800 --> 00:05:22.079
There's a lot of interdependencies, there's a lot of things you're not seeing.
00:05:22.399 --> 00:05:24.639
But what we do have is data.
00:05:24.879 --> 00:05:32.079
We have lots of patients who we've recorded their heights and smoking habits of all their inputs, other X's, and all their covariates.
00:05:32.240 --> 00:05:33.279
All their covariates.
00:05:33.360 --> 00:05:33.920
That's right.
00:05:34.079 --> 00:05:37.600
And all their Ys, the outputs, and we have data for this.
00:05:37.759 --> 00:05:40.399
You know, uh, why don't we do something different?
00:05:40.480 --> 00:05:46.399
And so it's the machine learning revolution, and you know, the timelines are a little bit off because there's a lot of machine learning work even in the 80s.
00:05:46.480 --> 00:05:53.439
But really, a lot of it took off in the 90s, and um they decided that, hey, why don't we just train machine learning models?
00:05:53.519 --> 00:05:57.360
We'll have a model model that says that that's simplified in some fashion.
00:05:57.439 --> 00:06:00.800
But then if you give it enough data, you can figure out uh what that looks like.
00:06:00.959 --> 00:06:06.639
So a simple example of that is imagine you have a uh a picture of X and Y axis with a bunch of dots in there, right?
00:06:06.720 --> 00:06:08.399
You have a bunch of dots you're trying to make sense of.
00:06:08.560 --> 00:06:09.360
That's the data.
00:06:09.600 --> 00:06:13.600
Maybe the dots are kind of clustered together to look like a line, kind of.
00:06:13.759 --> 00:06:16.639
You you could sort of draw a line right through it, right?
00:06:16.879 --> 00:06:18.639
Um, and they're kind of close to one another.
00:06:18.720 --> 00:06:24.240
So it feels like this like cloud of dots is not really just a random cloud of dots, but it has a pattern to it, right?
00:06:24.480 --> 00:06:26.480
And that pattern might be a line.
00:06:26.639 --> 00:06:31.439
And if it's a line, we know mathematically how to write a line, how to write an equation of a line.
00:06:31.680 --> 00:06:33.120
Y equals mx plus b.
00:06:33.360 --> 00:06:34.000
Look at me.
00:06:34.240 --> 00:06:46.720
I'm coming out with my uh not only did you come out with your knowledge grade algebra, but not only did you do that, you also use the x and the y, which you've implicitly said explicitly said so y equals mx plus b.
00:06:46.879 --> 00:06:52.240
But what we don't know in that line before looking at any of the data is what is m and what is b.
00:06:52.480 --> 00:06:54.480
Like we don't know which way the line is sloped.
00:06:54.639 --> 00:06:57.600
We don't know where the line is, is it up or down?
00:06:57.759 --> 00:06:59.439
We just know there exists a line, right?
00:06:59.600 --> 00:07:00.240
There's a pattern there.
00:07:00.560 --> 00:07:01.519
Or there might exist a line.
00:07:01.600 --> 00:07:04.319
We just don't know for sure if there exists a line, but there's a pattern there.
00:07:04.639 --> 00:07:18.160
And um, if the data kind of looks like that, then maybe we can fit, it's called fitting a line to that, which is we train the data, we train the model so that it learns what the M's and the B's are to draw the best kind of line right through the middle, right?
00:07:18.399 --> 00:07:21.439
To reduce the error that that that model is going to do.
00:07:21.600 --> 00:07:24.240
So that model inherently is going to have some degree of error.
00:07:24.319 --> 00:07:25.519
That's how that's what models are.
00:07:25.680 --> 00:07:29.600
Mathematical models are abstractions that are meant to be simplified in some fashion.
00:07:29.680 --> 00:07:30.879
They're not meant to be perfect.
00:07:31.040 --> 00:07:35.600
But at the same time, they want to be as close to the true to the data as much as possible.
00:07:35.920 --> 00:07:40.240
Right, especially from a clinical angle, like you you can only use it when it's like a pretty good model, right?
00:07:40.560 --> 00:07:40.879
That's right.
00:07:41.120 --> 00:07:41.519
That's right.
00:07:41.680 --> 00:07:46.079
So you have machine learning models that can do this and learn the line.
00:07:46.319 --> 00:07:49.839
Now, neural networks came along and said, what if that's not a line?
00:07:49.920 --> 00:07:53.279
What if the data is kind of weird and you can't actually draw a line through it?
00:07:53.439 --> 00:07:55.680
Then you might need something more sophisticated.
00:07:55.839 --> 00:08:02.240
And without going into the mathematics of it, there's other kinds of things you can add, non-linearities or things that make it not line like.
00:08:02.480 --> 00:08:03.680
Yeah, I don't know any of those equations.
00:08:03.759 --> 00:08:07.120
I done my mathematical knowledge strapped at y equals mx plus b.
00:08:07.360 --> 00:08:08.399
Which is fine, which is totally fine.
00:08:08.480 --> 00:08:16.399
But that's, you know, a lot of times your data can fit a line, or a lot of times you can like adjust your data so that it then becomes a line, and then you can fit it to that adjusted line.
00:08:16.560 --> 00:08:26.800
But regardless, the point I'm trying to make is that you had machine learning models that learned those M's and Bs or those knobs that you can turn to make the model, whatever equation you have.
00:08:26.959 --> 00:08:30.399
So now you only have um two things, M and B, you have to care about.
00:08:30.480 --> 00:08:33.519
You might have started with a bunch of X's, but now you only have two things to manage.
00:08:33.600 --> 00:08:38.000
Um so regardless of whatever the X was, you have, you know, you can manage it much more easily.
00:08:38.240 --> 00:08:48.080
Um but so uh the going back to the the drift case here, the whole principle of this is that you have a bunch of data that kind of looks like a line that has a pattern to it, right?
00:08:48.320 --> 00:08:48.639
Right.
00:08:49.279 --> 00:08:53.279
What if that when you trained the model, that was the pattern?
00:08:53.440 --> 00:08:58.799
But then when it's out there in the real world and you're still collecting data, the pattern changes.
00:08:59.200 --> 00:09:04.399
The pattern could change but because maybe the um the population changes.
00:09:04.559 --> 00:09:07.279
Maybe you're not looking at the same age group anymore.
00:09:07.440 --> 00:09:09.279
You're looking at different age groups.
00:09:09.519 --> 00:09:12.480
Or maybe the output variable changes.
00:09:12.639 --> 00:09:15.279
Maybe you start to care more about diabetes sooner or something.
00:09:15.440 --> 00:09:17.759
Or maybe the relationship between them changes, right?
00:09:17.919 --> 00:09:26.080
Maybe it is not just uh you could have more variables that you might have to take into account, or some things are no longer relevant, as relevant as they were.
00:09:26.159 --> 00:09:31.600
Maybe in your original data, somehow for some reason making this up, smoking was like the most important factor.
00:09:31.759 --> 00:09:35.360
But really, what ended up happening later on was that that wasn't the important factor.
00:09:35.519 --> 00:09:40.320
People stopped smoking, but that you still need a model for diabetes and you no longer cared about smoking.
00:09:40.480 --> 00:09:44.399
So things can change uh along the way, and the relationships can change.
00:09:44.480 --> 00:09:48.000
Like, how does smoking and weight work to predict diabetes?
00:09:48.080 --> 00:09:51.600
That might itself the core mechanism by that might might change as well.
00:09:51.919 --> 00:09:55.600
You might also have changes in people defining what it means to have a certain disease.
00:09:55.679 --> 00:10:07.840
I mean, diabetes is diabetes, but sometimes it's more there's guidelines and they're gonna be like, oh, there's a different threshold at which we define the hemoglobin A1C for pre-diabetes versus diabetes, or there's a different threshold for what we define high blood pressure.
00:10:08.080 --> 00:10:09.039
Yes, that happens.
00:10:09.120 --> 00:10:15.919
Yes, that happens where we in the clinical literature redefine something or redefine what the treatment algorithm for something is.
00:10:16.159 --> 00:10:16.399
Yes.
00:10:16.480 --> 00:10:22.399
So your X could change, your Y could change, and your dots on that picture could change completely to make it a different line.
00:10:22.720 --> 00:10:25.440
So no longer is your so your model is no longer good.
00:10:25.600 --> 00:10:29.120
In a sense, your model has an expiry date on it, right?
00:10:29.279 --> 00:10:31.120
Except we don't know what that expiry date is.
00:10:31.360 --> 00:10:33.360
And we don't know how it will expire either, right?
00:10:33.519 --> 00:10:37.519
Because there are so many different ways that it that it could be altered.
00:10:37.679 --> 00:10:40.399
And a few of them could happen at the same time, too.
00:10:40.720 --> 00:10:41.200
Exactly.
00:10:41.279 --> 00:10:42.879
So that's at the core of drift.
00:10:43.039 --> 00:10:47.279
Drift just means that the model's distribution has shifted.
00:10:47.440 --> 00:10:58.399
Um, just when I say the word distribution, I want people I want you to imagine a bell curve that you might have been familiar with, like a little bell-shaped mathematical curve that people usually use to represent grades, right?
00:10:58.879 --> 00:11:10.960
You get it's not maybe not fondest of memories from a lot of people, but but regardless, that that's what a distribution is, is a list of things that it and it most generally kind of averages out in the middle, but then there are some other numbers in the side.
00:11:11.200 --> 00:11:18.240
Drift just means that that moved to a different point now, and so you whatever model you had before uh might not work.
00:11:19.120 --> 00:11:19.440
Okay.
00:11:19.600 --> 00:11:20.080
All right.
00:11:20.320 --> 00:11:36.000
So that this all makes sense, and I think we can um you gave some really good examples, but I'd love to dive into those examples of like here are the kind of subtypes of drift, um, and and why it matters from like the clinical angle.
00:11:36.240 --> 00:11:36.879
Yes, let's do that.
00:11:37.039 --> 00:11:43.919
And the first one you talked about was my fancy word, which was covariate drift, um, which is like the inputs, right?
00:11:44.000 --> 00:11:44.720
The X's.
00:11:44.799 --> 00:11:48.159
It's like, hey, maybe our population has changed.
00:11:48.399 --> 00:12:00.080
So, you know, we have different ages or different sexes or different ethnicities, um, or maybe like the people who are included are just different than they were before.
00:12:00.240 --> 00:12:00.639
Right.
00:12:00.879 --> 00:12:04.159
Or maybe something else, like the measurement method changed.
00:12:04.240 --> 00:12:04.639
Who knows?
00:12:04.720 --> 00:12:08.000
But I thought there were some really great examples in this article.
00:12:08.639 --> 00:12:14.799
One, which should surprise no one who's listened to our podcast, is that they trained uh a speech rec recognition model.
00:12:15.679 --> 00:12:23.600
Um and I I think they were looking at Alzheimer's for this one, screening for Alzheimer's or detecting early signs of Alzheimer's disease.
00:12:23.759 --> 00:12:30.320
And they trained it with um, you know, native English speakers, and that's that's who they used the recordings from.
00:12:30.399 --> 00:12:35.360
They were from a specific region um geographically where the speech patterns were all the same.
00:12:35.840 --> 00:12:45.039
And guess what happened when they deployed it in the real world, which included non-native English speakers who were from different geographic regions.
00:12:45.360 --> 00:12:51.679
Wow, different dialects, different dialects, different uh different accents, different words they used.
00:12:52.159 --> 00:12:59.200
And so it wasn't able to perform very well with a diff very different population.
00:12:59.519 --> 00:12:59.759
Right.
00:13:00.639 --> 00:13:19.519
And so I don't think that would surprise anyone um who's listened to the podcast that when you you you've given it X tasked and trained it on, well, I shouldn't use X, you've given it one task and you've trained it on a specific set of data, now you're trying to use it for a task that it's not really been trained for.
00:13:19.840 --> 00:13:23.360
I mean, this goes back to our cat-lung example that we that we keep repeating on this podcast.
00:13:23.440 --> 00:13:34.559
And for those of those who haven't heard that example, it's the example where a machine was trained on uh detecting whether an x-ray picture of a co of a lung had had COVID or not.
00:13:34.799 --> 00:13:44.000
And um and so the the the funny part of that story is that after training the model, they gave it a picture of a cat and it was 100% confident that this was a COVID lung.
00:13:44.159 --> 00:13:47.039
And that's just an example of completely different distribution.
00:13:47.200 --> 00:13:53.279
And I get it, that's a crazy example, but it just shows that these models are not the input matters and the distribution of the input matters.
00:13:53.440 --> 00:13:54.879
And if that changes, then everything changes.
00:13:55.120 --> 00:13:55.279
Right.
00:13:55.360 --> 00:14:01.679
And we have to train people like, hey, this is this this is tasked with this in this subpopulation.
00:14:01.919 --> 00:14:06.480
It can't be used on people with who who don't speak English as a first language, for example.
00:14:06.639 --> 00:14:08.000
It should never have been used for that.
00:14:08.399 --> 00:14:11.519
But there's also ones that are more subtle.
00:14:11.679 --> 00:14:16.799
Um, there's an example in the article about a software update in mammography equipment.
00:14:16.960 --> 00:14:21.919
And what happened was it affected the images that came out of the software.
00:14:22.080 --> 00:14:37.919
And so the AI was flagging more images as abnormal and then bringing more people back to get recalled for like repeat imaging, which you can imagine if you get a phone call that's like if you get three times the number of phone calls, they're like, oh, come back.
00:14:38.080 --> 00:14:42.320
Now you're making three times the number of people anxious that they have breast cancer, right?
00:14:42.399 --> 00:14:42.559
Right.
00:14:42.799 --> 00:14:43.440
That they could.
00:14:43.600 --> 00:14:51.360
And two, you're like creating all of this operational disaster because now you're you've had you have to do all these extra mammograms.
00:14:51.600 --> 00:14:51.679
Right.
00:14:51.840 --> 00:14:53.600
It's more expensive for the whole system as well.
00:14:53.919 --> 00:14:54.320
Exactly.
00:14:54.720 --> 00:14:58.320
And so that's an example where, hey, the software was recalibrated.
00:14:58.399 --> 00:15:05.840
And so you have to have in in that kind of a case, you're like, whoa, geez, like we need a mechanism to identify, hey, there's going to be a software update.
00:15:06.000 --> 00:15:08.879
We know that could affect how the images are interpreted.
00:15:09.120 --> 00:15:12.720
Here's the plan ahead of time for how we could fix something like this.
00:15:12.960 --> 00:15:21.360
Because even that sort of minor technical change, it can have really big downstream effects from a clinical standpoint.
00:15:21.600 --> 00:15:22.559
Yeah, that's right.
00:15:22.799 --> 00:15:23.440
That's right.
00:15:23.759 --> 00:15:25.600
And that's just changing the inputs, right?
00:15:25.679 --> 00:15:26.559
The covariates.
00:15:27.039 --> 00:15:27.759
Exactly.
00:15:28.080 --> 00:15:31.759
So that was the first sort sort of example, the X's, like you were talking about.
00:15:31.840 --> 00:15:37.120
And then the second, second type of drift they identified here was the outputs, the whys.
00:15:37.279 --> 00:15:39.440
It's like, hey, what are we trying to predict?
00:15:39.679 --> 00:15:41.919
And those can change over time too.
00:15:42.159 --> 00:15:48.879
And that may be something as simple as, you know, maybe the definition of something changes, right?
00:15:49.360 --> 00:15:52.159
Or maybe the outcome prevalence changes.
00:15:52.320 --> 00:15:52.720
One example.
00:15:52.960 --> 00:15:53.679
That's what I was looking at.
00:15:53.759 --> 00:15:56.000
The prevalence was an interesting one, right?
00:15:56.080 --> 00:15:57.600
Because the disease itself could change.
00:15:57.919 --> 00:15:58.240
Yeah.
00:15:58.480 --> 00:16:00.320
Or, or maybe there's a new disease.
00:16:00.399 --> 00:16:03.600
Like when COVID came, it's like now you have something new.
00:16:04.000 --> 00:16:06.080
The system hasn't encountered it before.
00:16:06.240 --> 00:16:09.600
It's going to alter, you know, what the pneumonia rates are.
00:16:09.759 --> 00:16:12.320
It's going to alter what the mortality rates are.
00:16:12.559 --> 00:16:20.000
And so if it's trained on stuff from before the pandemic, it's not going to do as well with this data that it hasn't seen before.
00:16:20.159 --> 00:16:20.320
Right.
00:16:20.480 --> 00:16:22.639
Because this hasn't seen this disease before either.
00:16:22.720 --> 00:16:30.240
And so it's like, well, how do you update quickly when you know some some new outcome sh sh crops up?
00:16:30.480 --> 00:16:31.039
Yeah.
00:16:32.159 --> 00:16:42.799
And then the third area of drift, which you already, you know, touched on as well, was the idea of concept drift, like the relationship between the X and Y.
00:16:43.360 --> 00:16:47.200
And that could be a variety of different things.
00:16:47.360 --> 00:16:51.919
You talked about how, like, hey, what if we update the the clinical guidelines?
00:16:52.080 --> 00:16:55.120
What if we update it, you know, what's the diagnosis of high blood pressure?
00:16:55.519 --> 00:16:56.799
Pretty infrequent, pretty frequently, right?
00:16:56.879 --> 00:16:58.480
I mean, people it definitely happens.
00:16:58.559 --> 00:16:58.799
Yeah.
00:16:58.960 --> 00:16:59.440
For sure.
00:16:59.600 --> 00:17:05.279
I mean, if you ask people, um, I'll ask you actually, is 120 over 80 a normal blood pressure?
00:17:06.000 --> 00:17:06.960
Seems high.
00:17:07.200 --> 00:17:08.480
Oh, you're actually correct.
00:17:08.559 --> 00:17:09.599
Oh, okay, good job.
00:17:09.680 --> 00:17:12.640
I feel like a lot of people think that that's like a normal blood pressure.
00:17:12.799 --> 00:17:14.240
In fact, they'll like just put it on their forms.
00:17:14.559 --> 00:17:16.160
Hey, you've got cooler than this podcast.
00:17:16.240 --> 00:17:18.160
I got the high blood pressure on this podcast.
00:17:18.240 --> 00:17:18.880
So there you go.
00:17:19.119 --> 00:17:21.039
We've taught each other something along the way.
00:17:21.279 --> 00:17:21.599
Something.
00:17:21.839 --> 00:17:22.240
Yeah.
00:17:22.480 --> 00:17:23.519
Um, but that's it.
00:17:23.680 --> 00:17:28.799
It's like, hey, 120 over 80, which many people do think is normal, is not normal.
00:17:28.880 --> 00:17:30.400
That's hypertension stage one.
00:17:30.559 --> 00:17:36.079
And so that that's a guideline that was changed, you know, some semi-recently at least.
00:17:36.160 --> 00:17:37.759
And that can happen over time.
00:17:37.920 --> 00:17:42.400
Um, but also we can change how diseases are annotated, how we label them, how we code them.
00:17:42.480 --> 00:17:48.400
And of course, we know that how we annotate and code things is all the information that goes into the AI.
00:17:48.480 --> 00:17:51.279
So if you have bad information in, you get bad information out.
00:17:51.359 --> 00:17:54.480
You want to make sure that you have everything labeled really well.
00:17:54.720 --> 00:18:01.839
So if there's updates to the coding system like that we use for billing and diagnostics, yeah, then the data set needs to be recoded.
00:18:02.319 --> 00:18:02.640
Exactly.
00:18:02.799 --> 00:18:02.880
Yeah.
00:18:03.200 --> 00:18:05.119
Oh my gosh, and that might that could take forever.
00:18:05.279 --> 00:18:05.599
Right.
00:18:05.839 --> 00:18:13.039
Um, or maybe you change, like, hey, what's the threshold at which point we're going to make this decision?
00:18:13.279 --> 00:18:20.720
Um and then, you know, how the labeling happens, that could like that could affect things too.
00:18:20.880 --> 00:18:34.960
And so I think there's a lot of great concepts in this paper that go through hey, all of these types of drift are possible, the X's, the Y's, and the relationship between the two, and they can all happen at the same time, unfortunately for us.
00:18:35.200 --> 00:18:36.400
Right, right, right, right.
00:18:36.480 --> 00:18:38.079
Which is what makes it really challenging.
00:18:38.240 --> 00:18:45.519
And, you know, some of it has to do with maybe you just have something that you can't predict up front and it just happens, like COVID.
00:18:46.000 --> 00:19:04.480
You know, some things are truly drift, which is it's a very similar situation, except like the dialect example is a great one, because that is an example of of drift and potentially could be solvable by uh scoping, you know, kind of teaching the users that you can't, you know, these are the limits of what you can use it for.
00:19:04.880 --> 00:19:10.799
And or training on, you know, people from different geographic areas who use different words and have different accents and whatever, right?
00:19:10.880 --> 00:19:11.839
You can do either of those things.
00:19:12.079 --> 00:19:12.240
Yeah.
00:19:12.319 --> 00:19:19.279
But I mean, if the if the use case was anticipated to be across these different geographic areas, then maybe they would have trained it in the first place, right?
00:19:19.440 --> 00:19:31.759
It's not like they didn't have the information before, but the clinical guideline example is a great one because there's no if that changes, there's no like it wasn't like they they could have done it before because none of the changes happened well after the model was out there.
00:19:31.920 --> 00:19:32.160
Right.
00:19:32.319 --> 00:19:37.759
So that to me is an even clearer example of drift because now everything now the rules have changed.
00:19:37.839 --> 00:19:39.359
So these models don't work anymore.
00:19:39.759 --> 00:19:40.160
Totally.
00:19:40.319 --> 00:19:51.359
And that's why I think it it's important to have some sort of monitoring plan in place before you deploy these things so that you can take action if depending on what needs to be done.
00:19:51.440 --> 00:20:01.440
And I think this article is actually really does does a really nice job of talking through like, hey, not all risks are the same when it comes to drift.
00:20:01.519 --> 00:20:07.920
Like it matters, it matters a lot like how quickly, um, how quickly this is happening, right?
00:20:08.079 --> 00:20:11.759
Maybe the onset of COVID was like abrupt, right?
00:20:11.839 --> 00:20:14.079
But then there are other or like a new clinical guideline.
00:20:14.160 --> 00:20:17.119
That's a great example of like, hey, that changed overnight, basically.
00:20:17.359 --> 00:20:17.599
Right.
00:20:17.839 --> 00:20:19.119
As far as the model knows.
00:20:19.279 --> 00:20:23.920
Um, but then there are some things that might be more gradual that happen over like five or ten years.
00:20:24.240 --> 00:20:25.200
Right, right, right.
00:20:25.440 --> 00:20:25.759
Exactly.
00:20:25.920 --> 00:20:34.000
That's so so I think what I liked about this paper also was talked about how that plan for monitoring should happen before, right?
00:20:34.160 --> 00:20:35.200
Before deployment.
00:20:35.519 --> 00:20:35.839
Right.
00:20:36.000 --> 00:20:45.039
That you should have a plan and that you should have triggers at which you case you say, like, oh, I need a warning at this point, or I need to like make a change.
00:20:45.599 --> 00:20:50.000
And that may depend on how quickly something is happening, but also how severe it is.
00:20:50.240 --> 00:20:55.920
It's like if something isn't that severe, then maybe you can let it go for a little bit longer, right?
00:20:56.160 --> 00:21:00.640
Until it reaches a threshold where you you can take your time and think it through or whatever.
00:21:00.880 --> 00:21:15.759
Whereas if you have something sending people in for triple the mammograms and you've now overloaded your system and you can't get people scheduled for their screening mammograms anymore, and now you're costing the system three times as much money, maybe you need to do something a little bit faster.
00:21:15.920 --> 00:21:16.319
Right, right.
00:21:16.480 --> 00:21:18.480
Um, or maybe you need to be able to turn it off even.
00:21:18.720 --> 00:21:19.440
Yeah, right, right, right.
00:21:19.519 --> 00:21:20.160
Pull the plug.
00:21:20.240 --> 00:21:20.640
Yeah, exactly.
00:21:20.880 --> 00:21:30.319
And so you have to think about not just the clinical impact, which of course, uh, you know, can can be large, but also like, how does it, you know, what's the operational impact?
00:21:30.559 --> 00:21:31.920
What's the ethical impact?
00:21:32.000 --> 00:21:33.440
Um, is the data protected?
00:21:33.599 --> 00:21:37.920
All of these other things where you could introduce bias to the system, right?
00:21:38.000 --> 00:21:42.480
You could have uh outputs that are not accurate, and it that can affect patients too.
00:21:42.640 --> 00:21:59.759
And so all of these things are things that you can't necessarily predict the exact thing that would happen, but I I do think that you should be considering what those are and figuring out hey, these are the action pathways, and this is when we need a human to pop in here and review it.
00:22:00.240 --> 00:22:04.079
And also have people understand this is how the model works.
00:22:04.400 --> 00:22:05.519
Yeah, yeah, that's completely true.
00:22:05.599 --> 00:22:15.039
And you know, some of this plays into regulation and how the regulatory agencies are intended to approve certain products and not, like the FDA, for example, right?
00:22:15.279 --> 00:22:20.400
And what they need is some kind of assurance that this will work through the work through the course of its lifetime.
00:22:20.559 --> 00:22:21.519
This product will work.
00:22:21.680 --> 00:22:23.039
And that's part of this, right?
00:22:23.200 --> 00:22:24.559
What does it mean to work?
00:22:24.880 --> 00:22:32.559
You know, it means to work if it can handle, if it's robust to drift, if it can handle potential foreseeable drift in some sense, right?
00:22:32.799 --> 00:22:33.039
Right.
00:22:33.200 --> 00:22:41.039
Um, of course, unforeseeable things happen, and but then there should be a plan for unforeseeable things happening, uh, just from the perspective of maybe pulling the plug or something, right?
00:22:41.119 --> 00:22:46.720
If some metrics don't match anymore and you don't have anything to know why it's happening, then maybe there should be a plan for that too.
00:22:46.880 --> 00:22:53.519
So it is not surprising that you have you you have you think about these kinds of it's there in other uh other sectors, right?
00:22:53.599 --> 00:23:10.799
In in in uh um in I believe in in some of the um aircraft and um I'm forgetting the word, but like aerospace sectors, there there are such redundancies, there are such continuous monitoring methods, there are human failed safes, there are all these things that exist.
00:23:10.960 --> 00:23:13.680
People think about this stuff, and it probably should exist here too.
00:23:14.000 --> 00:23:15.200
I totally agree with that.
00:23:15.359 --> 00:23:21.279
And um just to add on, you know, we talk about updates all the time as if they were good things.
00:23:21.440 --> 00:23:26.400
And like oftentimes they are, but have you ever been in a situation where an update like wasn't so great?
00:23:26.559 --> 00:23:31.440
Yes, happens all the time with our iPhones sometimes, our phones, our computers where we don't want the update.
00:23:31.519 --> 00:23:32.160
People like email.
00:23:32.400 --> 00:23:33.680
There's like a Reddit feed about this, right?
00:23:33.759 --> 00:23:36.000
It says, Oh, don't update to the latest 27.4.
00:23:37.519 --> 00:23:37.680
Right.
00:23:37.759 --> 00:23:38.720
And then wait for the next one.
00:23:38.799 --> 00:23:41.200
So you got to wait for them to fix the update and then you wait for the next.
00:23:41.440 --> 00:23:42.960
So yeah, that kind of stuff happens all the time.
00:23:43.119 --> 00:23:43.279
Right.
00:23:43.359 --> 00:23:54.880
And we've had clinical examples of that where, you know, they updated um a chat bot for people with eating disorders to an LLM that sort of went rogue and was like giving people diet advice that it shouldn't have been.
00:23:55.119 --> 00:23:58.960
Um, and and so that's the thing is like when you have an update, you need to audit it.
00:23:59.039 --> 00:24:00.799
You need to make sure that that update is safe.
00:24:00.880 --> 00:24:04.160
And that needs to be something that's continuously monitored as well.
00:24:04.319 --> 00:24:10.720
You can't just assume, like, hey, because it worked yesterday and we put an update in, now it's gonna work as intended today.
00:24:10.880 --> 00:24:11.039
Right.
00:24:11.200 --> 00:24:24.000
And I think people who work in this field, not just clinicians, but you know, tech, AI, computer science, um, product people need to like have the literacy too to say, hey, this this is weird.
00:24:24.160 --> 00:24:35.599
Like, why are these, why are three times the number of people getting called back for mammograms, or why is there an anomaly in this subgroup that's being um flagged every time, or whatever it is?
00:24:35.839 --> 00:24:47.039
People need to understand how the models work and how the AI works so they can also understand when and why this drift might be happening and what what to do about it.
00:24:47.200 --> 00:24:47.519
Yeah.
00:24:50.559 --> 00:24:53.359
I think that's the hardest part too, is the what to do about it.
00:24:53.440 --> 00:25:00.960
It's like, hey, we need to we need to figure out how we monitor the performance, and that is expected.
00:25:01.039 --> 00:25:06.720
Like you don't just release something and like expect it to work well forever.
00:25:06.960 --> 00:25:16.319
You need to monitor it and then and you need to detect this drift in the real world, and then you need to adapt to whatever it is, right?
00:25:16.400 --> 00:25:19.359
We need to we need to be able to figure out what is the underlying cause.
00:25:19.440 --> 00:25:20.799
Was it a covariate drift?
00:25:20.880 --> 00:25:21.599
Did you like that?
00:25:22.000 --> 00:25:23.599
Was it like an outcome drift?
00:25:23.759 --> 00:25:25.680
Was it a relationship between the two?
00:25:25.839 --> 00:25:37.920
Um, or was it, you know, all of the above, and then be able to intervene so that when the drift is identified, you can stop some of the downstream problems that could be happening.
00:25:39.359 --> 00:25:41.359
Yeah, great place to end.
00:25:41.519 --> 00:25:42.960
Um, thank you for joining us.
00:25:43.200 --> 00:25:44.319
The work never stops.
00:25:44.400 --> 00:25:46.400
We'll see you next time on Code and Cure.