#65 - Can care managers trust AI summaries?

WEBVTT
00:00:00.400 --> 00:00:03.439
A patient's story is scattered across dozens of data points.
00:00:04.320 --> 00:00:14.240
What if AI could pull those signals together and give care managers a clearer picture of what's happening in a person's life and where intervention might actually help?
00:00:14.800 --> 00:00:20.239
Basically, can AI turn data overload into person-centered care?
00:00:28.640 --> 00:00:33.759
Hello and welcome back to Code and Cure, where we discuss decoding health in the age of AI.
00:00:34.079 --> 00:00:35.679
My name is Vassant Sarathi.
00:00:35.840 --> 00:00:38.399
I'm an AI researcher and cognitive scientist.
00:00:38.479 --> 00:00:41.200
And I'm here as always with Laura Hagopian.
00:00:41.359 --> 00:00:44.479
I'm an emergency medicine physician and I work in digital health.
00:00:44.719 --> 00:00:53.119
And I was particularly excited about this paper because it feels like a great real-world application of AI.
00:00:53.280 --> 00:00:56.560
But I know when I showed it to you, you were like, I'm a little bit hesitant.
00:00:56.880 --> 00:00:57.280
No, no, no.
00:00:57.439 --> 00:00:57.840
No, no, no.
00:00:57.920 --> 00:01:00.880
I think, I think, well, not not hesitant necessarily.
00:01:01.039 --> 00:01:11.599
I think for me, I am probably the most excited about this episode that I've ever been because this is directly in line with some of my own personal research work.
00:01:11.760 --> 00:01:16.560
My my my my work as a AI researcher is in this space.
00:01:16.879 --> 00:01:23.680
And I am deeply aware of all of the pitfalls and the positives of uh work in this space.
00:01:23.840 --> 00:01:26.719
And so it gets me very excited to talk about it.
00:01:26.879 --> 00:01:30.959
And I am literally sitting on the edge of my seat here, pretty, you know, psyched about this.
00:01:31.040 --> 00:01:31.840
So let's go.
00:01:32.159 --> 00:01:32.560
Awesome.
00:01:32.719 --> 00:01:32.959
Yeah.
00:01:33.120 --> 00:01:42.239
So what they did in this study is they basically said, hey, we have all of this data from all these different sources on patients.
00:01:42.719 --> 00:01:48.719
And um, it could be quantitative and structured, or it could be qualitative data.
00:01:48.879 --> 00:01:52.640
It could be something that's just like written in a note somewhere randomly.
00:01:52.799 --> 00:01:56.480
It could be from 10 years ago or one year ago or yesterday.
00:01:56.719 --> 00:02:03.680
There's all this different data stored in different places at different times and in in different ways.
00:02:03.920 --> 00:02:27.759
And the whole concept is hey, if you have a care manager, they need to like really understand where a patient is in the moment so they can intervene and help in a very targeted way with whatever clinical needs they may have, but also whatever social needs they might have, whether it's um issues with housing or food insecurity or uh loneliness or transportation or or whatnot.
00:02:28.000 --> 00:02:39.439
And so the whole concept here is like, hey, can we take this large volume of data, which might take a care manager hours, days, I don't even know how long to kind of sift through before they talk to a patient.
00:02:39.599 --> 00:02:44.800
And can we get a quicker summary so we can intervene on the things that really matter for that patient?
00:02:45.120 --> 00:02:46.240
Right, right, right.
00:02:46.479 --> 00:02:56.159
And the technical name for this subfield is multi-document summarization, which is the idea that you provide lots of different types of documents and you get a summary from it, right?
00:02:56.319 --> 00:02:59.840
And if you think about it, we use AI for that all the time.
00:02:59.919 --> 00:03:10.159
I mean, people who are familiar with using AI tools like ChatGPT, you can think about the times you update you upload a PDF or maybe three or four PDFs or something, and you ask for a summary, right?
00:03:10.319 --> 00:03:13.840
That is a very common task that we ask of AI these days.
00:03:14.159 --> 00:03:21.680
I actually I I mean, I guess I would say one of the places I see it the most is when I do like a Google search now, where I'm like, hey, what's the answer to this question?
00:03:21.840 --> 00:03:23.520
And it pops up with a summary.
00:03:23.680 --> 00:03:26.639
But I will tell you that that summary is not always correct.
00:03:26.960 --> 00:03:27.439
Well, yes.
00:03:27.520 --> 00:03:29.280
I mean, I think everyone's aware of that, right?
00:03:29.520 --> 00:03:45.919
I was talking to someone else recently who was working in the legal space, and they um were able to, you know, they opened up Adobe Reader or something for opening up a legal brief that they were or case that they were reading, and it automatically provided a summary of that document.
00:03:46.080 --> 00:03:47.840
And they were the legal experts reading it.
00:03:47.919 --> 00:03:51.599
So when they read the summary, the summary was all wrong, all kinds of issues, right?
00:03:51.840 --> 00:03:56.800
But the popping up of the summary is a like automated feature now is part of Adobe's reader.
00:03:56.879 --> 00:03:57.199
Yeah.
00:03:57.439 --> 00:04:00.560
And um, which, you know, in some ways can be really helpful.
00:04:00.719 --> 00:04:02.960
But if it's wrong, then it's not helpful at all.
00:04:03.039 --> 00:04:03.680
It's hurtful, right?
00:04:03.759 --> 00:04:07.360
And so I think that that's there's a challenge there, and we'll get to that in a second.
00:04:07.520 --> 00:04:27.120
But I think the uh what I like about this paper is the underlying challenge and that the approach that they've taken to incorporate all of these other systemic challenges that goes into care, all of the sociocultural, all of the stuff that's outside of the core medical things, right, that are also important.
00:04:27.680 --> 00:04:28.399
Absolutely.
00:04:28.560 --> 00:04:33.040
And I think those things can impact your health more than anything else, right?
00:04:33.199 --> 00:04:39.439
Like if you um, for example, are food insecure, then you're not gonna be managing your diabetes.
00:04:39.519 --> 00:04:42.800
You're eating whatever you can get your hands hands on, for example.
00:04:42.879 --> 00:04:43.040
Yeah.
00:04:43.199 --> 00:04:51.519
And that may be more, you know, processed foods or things that have um, you know, the glycemic index that you don't want, all that kind of stuff may be true.
00:04:51.680 --> 00:05:00.560
And so it's like if you want to intervene on the whole person, it's not just, you know, the medical diagnoses that they carry or the behavioral health diagnoses that they carry.
00:05:00.639 --> 00:05:14.000
It's like the integrated, you know, whole person view where you understand not only their medical and behavioral health concerns, but what challenges they may have accessing care and what social concerns they may have as well.
00:05:14.160 --> 00:05:14.240
Right.
00:05:14.399 --> 00:05:16.319
And that's when you can sort of treat the whole person.
00:05:16.399 --> 00:05:20.000
But then that's like, okay, well, now there's data coming in from all these different places.
00:05:20.160 --> 00:05:22.000
And the data changes over time, right?
00:05:22.160 --> 00:05:26.319
Someone may have had housing insecurity and they no longer have it and now they have it again.
00:05:26.480 --> 00:05:26.720
Right.
00:05:26.800 --> 00:05:29.759
So the timelines I think are interesting.
00:05:29.920 --> 00:05:35.199
And there's it's so dynamic that it's like, well, we need to know what's happening now.
00:05:35.600 --> 00:05:38.720
And the fact that so much of it is not structured data.
00:05:38.879 --> 00:05:39.120
Yeah.
00:05:39.279 --> 00:05:43.519
Like it might just be, oh, I talked to someone on the phone and this is what they note wrote in a note.
00:05:43.680 --> 00:05:44.000
Yes.
00:05:44.160 --> 00:05:48.160
Rather than they checked off a box that said, you know, patient has food insecurity.
00:05:48.240 --> 00:05:48.399
Right.
00:05:48.639 --> 00:05:49.519
That was a year ago.
00:05:49.600 --> 00:05:56.639
And now someone wrote in a note, hey, they actually have been doing well, and they um have access to SNAP benefits or whatever it is.
00:05:56.879 --> 00:06:08.000
So like tracking that change over time in different data sources, some of which could be structured, checkbox, and some of which could be unstructured, just like a note that someone happened to write.
00:06:08.240 --> 00:06:14.959
So I think this is what gets people so excited is the promise of large language models being able to do something like this.
00:06:15.120 --> 00:06:16.720
I think that's what people are.
00:06:16.959 --> 00:06:33.199
I I think there is a set sort of general in our psyche is that, oh my God, we have these amazing new AI systems that we can just dump a bunch of text to and they will give out something coherent that either summarizes it, helps us answer questions about it, or links things in it, whatever, right?
00:06:33.279 --> 00:06:38.480
I mean, this is at the very core of why we're so excited about these systems and applying it in different settings.
00:06:38.720 --> 00:06:38.959
Right.
00:06:39.120 --> 00:06:42.639
Because in this case, you can say, okay, well, now I have a summary of this patient.
00:06:42.800 --> 00:06:44.319
Now I know where to intervene.
00:06:44.480 --> 00:06:44.800
Yes.
00:06:44.959 --> 00:06:45.199
Right?
00:06:45.360 --> 00:06:47.439
Oh, like they don't have access to care.
00:06:47.519 --> 00:06:55.040
That's a place I need to intervene before I can work on um, you know, lowering their A1C value with them or whatever it is.
00:06:55.279 --> 00:07:11.600
And so what they did here was they did they kind of iterated over time and they they had this cross-functional team, which I think was really cool, where they had, you know, um engineers and product managers, and they involved the um, you know, the care team um as well.
00:07:11.680 --> 00:07:15.920
And they said, okay, well, we're gonna start off by drafting like a very simple prompt.
00:07:16.560 --> 00:07:26.399
We're gonna prompt the AI to take information from the notes um and and and basically summarize it and do it as as simply as possible.
00:07:26.720 --> 00:07:32.879
And then they saw what the output was and they said, okay, well, this isn't quite what we want as the output.
00:07:33.199 --> 00:07:43.759
Let's refine, let's refine this prompt so that we can get it sort of um clearly summarized in the areas that we think is are most important.
00:07:43.920 --> 00:07:44.079
Yeah.
00:07:44.319 --> 00:07:50.240
Let's add like a formatting structure so things are bold, there are headings so that it's like easier to skim and read.
00:07:50.480 --> 00:07:55.839
Um, and let's make sure that we have all the information that we want to be included on every patient.
00:07:56.480 --> 00:08:05.600
And then they said, okay, let's send this to case managers and let's have the case managers react to it and see if we can make it even better.
00:08:05.920 --> 00:08:14.959
And so they and so they over time collected feedback from the case managers and kept refining and updating the prompt to incorporate those changes.
00:08:15.120 --> 00:08:18.879
Um, and then went back to the to the case managers and said, Hey, is this working?
00:08:19.040 --> 00:08:27.759
And and what they were able to do over time by working with the case managers is say, Hey, we understand what topics you want.
00:08:27.920 --> 00:08:28.959
We're gonna update those.
00:08:29.120 --> 00:08:33.120
We're gonna make sure that there's like a summary at the top that hadn't initially been included.
00:08:33.360 --> 00:08:37.200
We're gonna um change the formatting so it's easier for you to use.
00:08:37.360 --> 00:08:48.240
And so over time, they had, you know, not just the headings and focused topic areas, but like a high-level summary paragraph and a list of what the patient's goals might be, etc.
00:08:48.720 --> 00:08:56.000
And so it just showed that there was like a lot of cross-functional work and they iterated on the prompt a lot to get what they wanted out of it.
00:08:56.480 --> 00:09:05.360
And then, and this is the part where I'm interested to hear what you think, Vasanth, is that they said, okay, we have this ready.
00:09:05.519 --> 00:09:07.039
Now we need to evaluate it.
00:09:07.200 --> 00:09:08.960
Now we need to check how accurate it is.
00:09:09.120 --> 00:09:17.039
Because of course, if you're intervening on stuff that's like not correct, then you're not, you know, you're not, you're not carrying out what you're trying to do.
00:09:17.279 --> 00:09:17.519
Yeah.
00:09:17.759 --> 00:09:22.879
And so they conducted some evaluations to check for accuracy here.
00:09:23.200 --> 00:09:23.440
Yeah.
00:09:23.600 --> 00:09:23.840
Yeah.
00:09:23.919 --> 00:09:25.840
I mean, there's so much to unpack, right?
00:09:25.919 --> 00:09:27.919
So my like my mind is sort of teeming here.
00:09:28.080 --> 00:09:29.759
I think there's so much to unpack.
00:09:29.919 --> 00:09:41.840
And even before we get to the evaluations question, I think I want to take a step back and give people a sense for what's actually happening in these LLMs or these AI systems when they're doing this.
00:09:42.080 --> 00:09:46.320
Because I think that provides a little bit more kind of foundation to move forward.
00:09:46.559 --> 00:09:49.600
So I think we've talked about this before, but it's worth repeating.
00:09:49.840 --> 00:09:57.519
Um, you know, the modern day AI breakthroughs are all based on a particular kind of neural network called a transformer.
00:09:57.759 --> 00:10:04.320
But the big breakthroughs are termed, I mean, the the architecture is called a large language model or an LLM.
00:10:04.480 --> 00:10:14.080
And at the very core of it, you don't have to worry about how that works exactly, but at the very core of it, it's trained on all the internet data and it's trained to do one thing really well, which is predict the next word.
00:10:14.240 --> 00:10:20.639
So you give it a bunch of text and it predicts the next word, and you take that word, put it back to the text that you had before, and then predict the next word.
00:10:20.799 --> 00:10:24.879
So over the sequence, or you know, when you do that over and over again, it gives you sentences.
00:10:25.120 --> 00:10:34.960
And what we've learned is if you train these models on very large amounts of data, that the sentences that come out are actually kind of coherent, grammatically correct, and interesting.
00:10:35.200 --> 00:10:44.799
And as people have worked more and more with these large language models, they've begun to realize that, hey, you can just not only get the next sentence out, but you can sort of train it to answer questions.
00:10:45.039 --> 00:10:51.039
So you can say things like, hey, summarize this for me, and then give it a block of text, and then it produces a summary.
00:10:51.279 --> 00:10:55.919
And how exactly it does that is still a bit of a mystery to most AI researchers.
00:10:56.000 --> 00:11:07.200
Uh, we don't really know how it is that it's run learned these skills of reasoning and uh understanding what you want and understanding what a summary even is to produce the right, to produce what seems like the right thing.
00:11:07.519 --> 00:11:07.759
Magic.
00:11:07.840 --> 00:11:08.480
It's magic, clearly.
00:11:08.799 --> 00:11:09.600
It is a bit of magic.
00:11:09.759 --> 00:11:15.519
So we have are in that world right now where we don't fully understand how these systems are producing what they're what they're producing.
00:11:15.759 --> 00:11:20.159
But they are producing uh, you know, so you give it the input, and that's often called a prompt.
00:11:20.240 --> 00:11:26.480
Uh, some it's also uh another term is um a prompt plus all the other text you provide as input is also called context.
00:11:26.639 --> 00:11:31.200
So you give it the context, and then what it what you know it produces some kind of output, right?
00:11:31.360 --> 00:11:32.559
And some kind of response.
00:11:32.639 --> 00:11:34.799
That's what oftentimes people will use for a word.
00:11:34.960 --> 00:11:38.879
So you get the prompt input and and the context and you get a response from it.
00:11:39.200 --> 00:11:47.840
And uh what the problem, of course, is when you have lots and lots and lots of documents, you the context, uh, the amount of context grows, and that's called the context window.
00:11:48.080 --> 00:11:49.120
It's pretty large.
00:11:49.279 --> 00:11:51.679
And early language models couldn't handle that as well.
00:11:51.840 --> 00:12:02.159
But now more and more language models, that new ones like the OpenAI models and so on, are actually set up so they can handle very large context windows, that is, words like a million words or whatever, right?
00:12:02.320 --> 00:12:03.279
Something ridiculous.
00:12:03.440 --> 00:12:05.759
Um, and so they can handle very big context windows.
00:12:05.840 --> 00:12:10.159
You can provide a lot of input and it provides a lot of it provides output uh that makes sense.
00:12:10.399 --> 00:12:11.519
There are challenges with this.
00:12:11.600 --> 00:12:16.639
These large language models are known to be not so great in dealing with context.
00:12:16.799 --> 00:12:26.399
Now, if you just asked it a question about a patient without providing any context, it's going to give you an answer, and that answer uh will be based on whatever it's learned, right?
00:12:26.559 --> 00:12:29.919
But it's it's got no relevance to you because it's got nothing to do with the patient, right?
00:12:30.000 --> 00:12:31.600
All the knowledge is just internal to it.
00:12:31.840 --> 00:12:34.799
So you provide the knowledge by adding to that context.
00:12:34.960 --> 00:12:37.919
You provide all the documents and you provide all the background in the context.
00:12:38.000 --> 00:12:39.200
That's one way to do it.
00:12:39.360 --> 00:12:44.720
Um, another way to do it is to say, hey, I have all this context stored separately in this database.
00:12:44.960 --> 00:12:47.679
You language model decide when you need to look it up.
00:12:47.840 --> 00:12:48.000
Uh-huh.
00:12:48.240 --> 00:12:51.039
Pull up what you need, and then answer my question.
00:12:51.679 --> 00:12:59.120
So that's what uh is popularly called rag systems or retrieval augmented generation systems, where you ask the LLM a question.
00:12:59.279 --> 00:13:05.120
It doesn't just use what it knows from random Reddit in internet pre- uh training that it did a long time ago.
00:13:05.279 --> 00:13:14.799
Instead, it looks up a database that you provided it and then pulls whatever pieces of snippets are relevant into its own context and then answers the question.
00:13:15.360 --> 00:13:20.000
And in that case, you're like the LLM is taking it from the spot you want it to.
00:13:20.080 --> 00:13:25.759
Like the information, I mean, I mean that's the question is like, does that make it kind of more accurate, I guess?
00:13:26.159 --> 00:13:26.639
That's right.
00:13:26.799 --> 00:13:29.919
The purpose of the RAG systems is to make it more accurate, right?
00:13:30.159 --> 00:13:33.759
And to deal with uh changes of uh data over time.
00:13:33.919 --> 00:13:35.519
Like new information comes in.
00:13:35.679 --> 00:13:43.360
The LLM that was originally trained wasn't trained on this new information, but it's still smart enough to be able to handle data.
00:13:43.600 --> 00:13:48.879
So you just keep updating the database with new information and use the same LLM to read off of that database.
00:13:49.120 --> 00:13:49.679
Does that make sense?
00:13:49.919 --> 00:13:50.879
Yeah, that totally makes sense.
00:13:51.039 --> 00:14:03.120
Yeah, so I I think that the this is important to understand is that they developed these RAC systems for the purposes of improving accuracy because you don't want it just making up stuff, uh, which by the way, the the jargon word is hallucinations.
00:14:03.360 --> 00:14:06.879
You don't want it hallucinating um or fabricating information.
00:14:07.039 --> 00:14:16.000
You want it to actually look up the correct answer and give it to you, and be grounded in the data that is appropriate in this specific use case for this specific patient, for example.
00:14:16.559 --> 00:14:21.919
Now, there are it's a big subfield, um, and uh that kind of retrieval augmented generation.
00:14:22.000 --> 00:14:23.440
There's been a lot of innovations.
00:14:23.519 --> 00:14:27.039
Um, some of the challenges involve do you even know?
00:14:27.120 --> 00:14:31.200
I mean, why would the LLM use the database and not its own internal knowledge?
00:14:31.519 --> 00:14:33.919
There's been studies about that because sometimes they don't look it up.
00:14:34.000 --> 00:14:36.960
The LLMs don't look it up, they just make it up anyways, right?
00:14:37.120 --> 00:14:38.559
There's no control over that.
00:14:38.639 --> 00:14:42.080
You could say, you could prompt the LLM to say, hey, make sure to look it up.
00:14:42.320 --> 00:14:48.799
But again, these prompts are just inputs that are text and the just they just you know result in outputs being more text.
00:14:48.879 --> 00:14:52.240
It's not like you're forcing the LLM to follow a specific procedure, right?
00:14:52.480 --> 00:14:56.720
Even if you tell it the specific procedure, it's unlikely to exactly follow it all the time.
00:14:56.960 --> 00:14:59.039
So Which actually happened in this study.
00:14:59.120 --> 00:15:08.159
They had to like adjust the prompt that's to say, like, oh, don't infer, don't speculate, don't create information that's not explicitly stated in the provided data.
00:15:08.320 --> 00:15:08.480
Yes.
00:15:08.639 --> 00:15:15.519
So clearly this was happening to them in this study where there were issues where hallucinations and stuff were coming through.
00:15:15.679 --> 00:15:23.840
Um, if there was like an incorrect diagnosis or an incorrect conclusion that came through, they had to say, okay, well, how can we can how can we fix the prompt?
00:15:24.000 --> 00:15:27.840
And is that, I mean, I guess my question is, is that enough to fix the prompt?
00:15:28.559 --> 00:15:28.879
I don't know.
00:15:29.360 --> 00:15:30.720
Notice what just happened there, right?
00:15:30.879 --> 00:15:35.759
One issue is that you had the human identify the problem and say, oh, wait a minute, this wasn't right.
00:15:35.919 --> 00:15:37.919
Let me go fix the prompt to deal with this.
00:15:38.159 --> 00:15:43.279
And to me, at some level, without seeming too negative, it seems like that's a little bit of a band-aid, right?
00:15:43.440 --> 00:15:53.679
Because you could have a new error mode that you hadn't seen before pop up later, and you have to hope that there's a human there noticing that error mode and then fixing the prompt.
00:15:53.840 --> 00:15:56.639
But there's no guarantee that the error mode is gonna be the same, right?
00:15:56.960 --> 00:16:04.399
I can also imagine over time, if you like start to trust a tool like this and you use it all the time, you're not like checking every single location.
00:16:04.879 --> 00:16:05.440
Right to that.
00:16:05.519 --> 00:16:05.600
Yes.
00:16:05.840 --> 00:16:09.519
And so you reach the point where you're like sort of have this, oh, it's automated.
00:16:09.600 --> 00:16:10.799
I have this automation bias.
00:16:10.960 --> 00:16:14.000
I'm just gonna like proceed as if this is correct.
00:16:14.159 --> 00:16:14.480
Yes.
00:16:14.639 --> 00:16:19.519
And then, you know, over time, maybe different error modes pop up that you never even identify.
00:16:19.759 --> 00:16:21.679
It cannot guarantee accuracy, right?
00:16:21.759 --> 00:16:23.200
So there's that problem already.
00:16:23.279 --> 00:16:31.840
But there's another problem, which is you bring in these pieces, large bodies of text, even if you use a quote rag system and allow the LLM to pull in what's relevant.
00:16:32.000 --> 00:16:37.360
If that context window, that input to the LLM is very large, it has issues like recency bias.
00:16:37.440 --> 00:16:41.360
That is, it remembers or only focuses on things that are at the end and not at the beginning of the context.
00:16:41.440 --> 00:16:42.799
It forgets things along the way.
00:16:42.960 --> 00:16:51.679
There's issues like that too that pop up that can cause, even when you pull the right snippet to produce the right summary from that snippet could still be could still have issues with it.
00:16:51.919 --> 00:16:57.200
So basically, like if it would be harder for a human to summarize, then it's harder for an LLM to summarize too.
00:16:57.360 --> 00:17:04.000
If you have too much context, it's gonna have an issue like figuring out, hey, what's the most relevant stuff to pull up?
00:17:04.319 --> 00:17:04.559
Yes.
00:17:04.720 --> 00:17:13.920
And there's another set of issues which comes up, which I think people don't talk about enough because hallucinations or fabrications or making up stuff is not the only problem.
00:17:14.160 --> 00:17:18.559
Even if you had a perfectly correct response, you could be omitting things.
00:17:18.799 --> 00:17:28.079
The LLM could be leaving out things that are actually relevant, that are actually important, and that are actually, you know, uh required in the final summary.
00:17:28.400 --> 00:17:33.279
And even worse so, the LLM could be removing caveats that were there in the original document.
00:17:33.440 --> 00:17:40.480
Because the original documents might have said something and then caveat it with something else, might have said, you know, this is true, except when it's sunny or something, right?
00:17:40.640 --> 00:17:58.000
And if you just say this is true as the output, then it's going to read if you have somebody reviewing it, they're going to say this is correct because it is saying that from the sources, but it's leaving out a big caveat that the except when it's sunny part, and that's a problem too, because you need the caveat to really understand what's being said.
00:17:58.400 --> 00:18:01.759
I think omissions are really difficult as a human to identify.
00:18:02.000 --> 00:18:13.519
And this is the problem that I've had when I've used um like the the uh AI to write my notes when I'm working at like an urgent care clinic is I sometimes I'll see, oh, it omitted.
00:18:13.599 --> 00:18:17.359
Like I asked these red flag questions and the note doesn't say any of those things.
00:18:17.440 --> 00:18:20.160
Sometimes I'll identify it, but I wonder how often I don't.
00:18:20.319 --> 00:18:25.519
Because it's easy to see when something is like wrong, blatantly wrong, like written in front of me.
00:18:25.759 --> 00:18:30.640
Um, you know, like a wrong medication or a wrong dosage or whatever.
00:18:30.799 --> 00:18:34.720
Oh, I know the patient didn't say that because their their mother said that, or whatever it is.
00:18:34.880 --> 00:18:42.319
But when it's omitted, it's like you have to like really be scouring to figure that out and have your brain like totally turned on.
00:18:42.480 --> 00:18:44.960
And I wonder how often I I miss an omission.
00:18:45.200 --> 00:18:45.599
And you do.
00:18:45.759 --> 00:18:55.839
And in fact, I've done studies as of this past few months on this, on human subjects, to study exactly what mode of errors that are easier to track and easier to capture and what ones are not.
00:18:55.920 --> 00:18:57.680
And omissions are exceedingly difficult.
00:18:57.839 --> 00:18:58.799
And it's not surprising.
00:18:58.960 --> 00:19:09.839
If you think about it, um, I am also a lawyer by training, and one of the steps to becoming a lawyer is to do this exam called the LSAT, which is the exam that's, you know, kind of helps you get into law school.
00:19:10.079 --> 00:19:18.400
And some of the questions on those are having to do with like deep critical reason thinking and reasoning, which doesn't come naturally to most people.
00:19:18.559 --> 00:19:23.599
Like if I give you a paragraph of text and say, hey, what are the assumptions that this person is making when they're writing this text?
00:19:23.839 --> 00:19:26.160
It is really hard because the assumptions are not written.
00:19:26.240 --> 00:19:32.960
They're all implicit and they all, you know, once I tell you what the assumption is, it's pretty straightforward to understand it.
00:19:33.119 --> 00:19:34.799
But it's really hard to pick it off.
00:19:35.039 --> 00:19:43.039
And if you are a human and you start to see good responses from these LLMs, you'll start to realize that if it omits things, you're not gonna catch it.
00:19:43.119 --> 00:19:44.720
You're you have automation bias on.
00:19:44.799 --> 00:19:48.799
You're just gonna you're gonna say, okay, this makes sense, this makes sense, nothing weird, nothing weird.
00:19:48.960 --> 00:19:52.559
I don't see any hallucinated claims because it's not seeing random stuff.
00:19:52.720 --> 00:19:56.880
But uh, I don't see any, I should say, I don't see any obviously hallucinated claims.
00:19:56.960 --> 00:20:00.319
If it's subtle and it makes small differences, it's very hard to spot.
00:20:00.480 --> 00:20:00.720
Yeah.
00:20:00.960 --> 00:20:05.039
Um, and it's also dependent on your own expertise and your own expert knowledge.
00:20:05.279 --> 00:20:10.000
And so, um, but if it's got omitted things, you don't even know to look for them.
00:20:10.160 --> 00:20:17.759
How do you know that it's you know, document B, somewhere uh you know, off the 500 documents hasn't has a caveat that you've left out.
00:20:17.920 --> 00:20:21.359
But it turns out that everything in the case turns on that one caveat, right?
00:20:21.680 --> 00:20:22.000
Right.
00:20:22.160 --> 00:20:29.599
And I think it gets harder and harder with a larger volume of documentation because you're like, well, how are you how are you verifying this?
00:20:29.759 --> 00:20:29.920
Yes.
00:20:30.160 --> 00:20:35.839
So I think we should talk a little bit about how they tried to verify hey, are these summaries accurate?
00:20:36.160 --> 00:20:36.480
Yes.
00:20:36.640 --> 00:20:36.960
Yes.
00:20:37.200 --> 00:20:48.960
So uh the space of what I call multi-document summarization has its own little community and has its own set of evaluation metrics, which are constantly, you know, being sort of evaluated in a sense.
00:20:49.039 --> 00:20:52.240
People in the research space are looking at new ways to measure this, right?
00:20:52.480 --> 00:20:53.920
And there are many ways to measure this.
00:20:54.000 --> 00:20:58.319
There's data sets out there that have uh what they call gold summaries, that is the correct summaries.
00:20:58.400 --> 00:21:07.119
So they have a set of documents and they have humans who have created those quote correct summaries, which is, you know, you can argue about whether or not those humans had the right summaries in the first place.
00:21:07.359 --> 00:21:09.599
Um and you gotta start somewhere, though.
00:21:09.920 --> 00:21:12.400
Summaries are also but summaries are also task dependent.
00:21:12.480 --> 00:21:16.720
So some summaries are better for some certain tasks, and maybe you want a different summary for a different task.
00:21:17.039 --> 00:21:17.279
Sure.
00:21:17.359 --> 00:21:24.319
I mean, in this case, we're talking about um whether care management providers are getting an appropriate summary.
00:21:24.400 --> 00:21:30.880
And so, you know, that's gonna include things like housing, financial concerns, food concerns, in addition to clinical, for example.
00:21:32.240 --> 00:21:40.160
If a provider was looking for a summary, maybe they're more interested in in medical episodes and primary care access and that kind of stuff.
00:21:40.240 --> 00:21:43.279
So, yes, totally context dependent, what type of summary yeah.
00:21:43.839 --> 00:21:53.839
So um the evaluation metrics are things like faithfulness and what faith, for example, faithfulness is something like it given my summary, is my summary faithful to my sources?
00:21:54.079 --> 00:21:54.319
Right.
00:21:54.559 --> 00:21:59.759
And the way they measure that is by looking at whether or not every single um sort of aspect.
00:22:00.240 --> 00:22:03.119
Of the summary is was present in the sources, right?
00:22:03.279 --> 00:22:04.400
Was supported by the sources.
00:22:04.480 --> 00:22:05.359
That means it's faithful.
00:22:05.519 --> 00:22:07.519
But notice it leaves out omissions, right?
00:22:07.680 --> 00:22:11.599
Because you can be fully supported and leave out the caveat, right?
00:22:11.759 --> 00:22:12.720
And and correct.
00:22:12.799 --> 00:22:13.920
Like was everything there?
00:22:14.160 --> 00:22:15.759
And so the answer is like, yeah, sure.
00:22:15.839 --> 00:22:16.640
Everything was there.
00:22:16.880 --> 00:22:17.119
Yes.
00:22:17.359 --> 00:22:21.680
Maybe that means there's not a hallucination, but it doesn't mean that there's not an omission.
00:22:21.920 --> 00:22:25.920
Or a piece of counter evidence, something that comp directly objects to think to the thing you're saying.
00:22:26.160 --> 00:22:34.160
Also, the thing that you're looking at, like if we talk about um, I don't know, food insecurity, that may change over time.
00:22:34.799 --> 00:22:35.119
Right.
00:22:35.359 --> 00:22:45.119
And so it could be, it could have been faithful two years ago to what was said in the chart, but maybe it's not faithful to what was said two weeks ago in the chart.
00:22:45.359 --> 00:22:45.519
Yes.
00:22:45.680 --> 00:22:49.119
So that's an interesting one where it's like, hey, there it's dynamic.
00:22:49.200 --> 00:22:49.599
Yes.
00:22:49.839 --> 00:22:55.039
And understanding the timeline is very important when a clinician is looking at it.
00:22:55.200 --> 00:23:01.680
I'm not sure how the AI looks at it, but that's another area where it could be particularly challenging.
00:23:02.000 --> 00:23:09.359
And you could say that it was faithful because it was faithful to something two years ago, but really what's more important is what happened two weeks ago.
00:23:09.599 --> 00:23:09.920
Yeah.
00:23:10.079 --> 00:23:21.759
And the way they actually measure this, there's different ways of measuring whether something in the summary is faithful, has been faithful to the sources, is by saying, um, let's just, let's just look at what's called entailment.
00:23:21.920 --> 00:23:25.440
That means if this thing in the source is true, then the thing in the summary must be true.
00:23:25.519 --> 00:23:27.200
It's kind of logical in nature.
00:23:27.440 --> 00:23:30.559
Um, and that's again a measure that people use.
00:23:30.799 --> 00:23:35.200
But the actual implementation of these metrics is, I think, the problem, is the challenge.
00:23:35.440 --> 00:23:44.960
Because at the end of the day, these metrics are either measured by a neural network that's been trained to do entailment checks, to do checks between if this is true, then is this true?
00:23:45.200 --> 00:23:51.440
Um that's like an uh a large language model as a judge, judging and no, that's that's option number two.
00:23:51.599 --> 00:23:55.039
So this entailment check is just a neural network that's being trained to do one specific thing.
00:23:55.119 --> 00:23:56.160
It's not a large language model.
00:23:56.319 --> 00:23:56.480
Okay.
00:23:56.640 --> 00:24:05.519
But it's trained to do like if I give if if I give it two sentences, a premise and a hypothesis, if the premise is true, tell me if that if the hypothesis is true or contradicting.
00:24:06.319 --> 00:24:06.799
So they do that.
00:24:06.880 --> 00:24:14.559
That's a very common way of measuring whether or not um a particular, you know if the source is true, is the summary true, right?
00:24:14.720 --> 00:24:17.440
Or a sentence in the source is true, then a sentence and a summary is true.
00:24:17.599 --> 00:24:20.960
So you can do the entailment checks, but the actual implementation is the problem.
00:24:21.200 --> 00:24:25.519
Because if you actually look at the models that are being used, they have very basic issues.
00:24:25.680 --> 00:24:28.559
For example, they have trouble handle handling negations.
00:24:28.720 --> 00:24:36.000
So if I had a knot in there somewhere, you know, it's sunny outside, it's not sunny outside, then those look very similar in many ways.
00:24:36.079 --> 00:24:40.319
And even the entailment models struggle a little bit with that, which they should not, but they they do.
00:24:40.559 --> 00:24:42.240
Yeah, but it's a it's a one-word difference.
00:24:42.319 --> 00:24:44.960
So you're like, wow, 90% of them are the same.
00:24:45.119 --> 00:24:46.960
It's just this 10% that's different.
00:24:47.359 --> 00:24:47.519
Yes.
00:24:47.680 --> 00:24:49.039
Oh, that looks accurate enough.
00:24:49.279 --> 00:24:49.599
Exactly.
00:24:49.759 --> 00:24:52.319
So that's one type of accurate accuracy metric.
00:24:52.480 --> 00:25:00.319
The other more common accuracy accuracy metric across the board has been LLM as a judge, which is to idea that you use a different LLM.
00:25:00.559 --> 00:25:03.759
So not the same one that you use to create the summary, but a different one.
00:25:04.079 --> 00:25:05.039
A different one.
00:25:05.200 --> 00:25:10.880
Um use a different LLM in order and you give it some prompts about how to evaluate it and you do the same thing.
00:25:10.960 --> 00:25:12.079
You ask it, is it consistent?
00:25:12.240 --> 00:25:14.880
Is the sources, you know, and and the summaries consistent?
00:25:15.119 --> 00:25:19.759
And just explain um, if you might, if you may, about why you use a different LLM for this.
00:25:20.160 --> 00:25:32.640
Yeah, I mean, I think the idea is whatever biases exist in the first LLM that created the summary, you don't want the same uh LLM biases to translate into the judge.
00:25:32.880 --> 00:25:35.200
Because it's almost like you're it's like circular.
00:25:35.279 --> 00:25:36.559
You're evaluating yourself.
00:25:36.720 --> 00:25:40.400
You're gonna give yourself the same score as you would the first time because you're the one who created it.
00:25:40.720 --> 00:25:40.960
Right.
00:25:41.039 --> 00:25:45.039
I mean, understandably, you're giving the LLM a different task.
00:25:45.200 --> 00:25:49.200
You're saying, hey, evaluate this with a with a critical eye or whatever, right?
00:25:49.359 --> 00:25:56.480
And presumably the answer would be maybe different, but whatever implicit biases that exist in that LLM, you don't want that to translate over.
00:25:56.640 --> 00:25:58.559
Um, and so that's why you use a different LLM.
00:25:58.640 --> 00:26:06.960
Now, there can be arguments about whether or not that actually works because ultimately a lot of these LLMs are trained in the same data, same internet data, right?
00:26:07.119 --> 00:26:07.200
Right.
00:26:07.359 --> 00:26:09.359
So it's some, but then the architectures are all different.
00:26:09.519 --> 00:26:18.720
So like Anthropics Claude model and the OpenAI's model are all slightly different in different ways to make them better suited, have kind of their own little personalities in a sense.
00:26:18.880 --> 00:26:27.039
So it is going to give you different answers, and there's no nothing wrong in using a in fact, I think it's good practice to use a different LLM to evaluate it.
00:26:27.200 --> 00:26:29.920
But it's still an LLM looking at the same set of words.
00:26:30.000 --> 00:26:37.440
And unless you prompt it the way one should, in the perfect way possible, you're not going to get at the same set of issues.
00:26:37.519 --> 00:26:41.279
And you're you're just sort of transferring the evaluation problem to another LLM.
00:26:41.359 --> 00:26:42.960
And how do you know that one's right?
00:26:43.200 --> 00:26:45.839
So, you know, that's I think there's a there's an issue there.
00:26:46.079 --> 00:26:46.400
Yeah.
00:26:46.480 --> 00:26:54.880
And I think they did have a bunch of hurdles that they were trying to overcome as they as they worked through, hey, how can we get a better summary?
00:26:55.119 --> 00:26:55.359
Yeah.
00:26:55.519 --> 00:26:59.519
And they walked through, you know, some of the things they did to try to resolve these.
00:26:59.599 --> 00:27:09.759
And we talked about how they adjusted their prompt to say, like, don't, don't, don't make a speculation or um don't say something unless it's explicitly in the data.
00:27:10.000 --> 00:27:14.559
Because it was just like making inferences or making things, like it was hallucinating, right?
00:27:14.799 --> 00:27:22.079
Um, but they also found, and I uh this does not surprise me that that it was having trouble with comp complex timelines.
00:27:22.240 --> 00:27:25.839
This happened, then that happened, then this happened, or whatnot.
00:27:26.000 --> 00:27:33.279
So it's like, you know, it's conflating events or um just like messing up the details that were coming in.
00:27:33.599 --> 00:27:38.400
And so again, they tried to like adjust a prompt to help fix this.
00:27:38.640 --> 00:27:44.240
Oh, make sure you review all the dates, include the dates uh when available with any claim that was made.
00:27:44.400 --> 00:27:49.759
But they said, you know, this is not, this is not something that they were able to like totally fix.
00:27:49.839 --> 00:27:52.799
This is something that kind of remained an issue.
00:27:52.960 --> 00:27:59.200
Um, and that they are continuing to try to sort of figure out how we can fix it.
00:27:59.359 --> 00:28:03.200
And part of what they're doing is trying to fix the data that goes into it, too.
00:28:03.519 --> 00:28:03.680
Yeah.
00:28:03.839 --> 00:28:16.319
Is hey, can we make the data more structured or can we uh format it better, or can we store it in a better way, or can we make sure that the updates uh truly look chronological to the LLM?
00:28:16.480 --> 00:28:23.599
And can we make this, you know, can we if you have like worse data going in, then it's obviously like harder for the LLM to process.
00:28:23.680 --> 00:28:25.279
So can we make the data look better?
00:28:25.359 --> 00:28:28.319
Yeah, can we improve the prompt to try to get at this?
00:28:28.480 --> 00:28:32.720
I I think this is a really difficult problem to try to solve.
00:28:33.039 --> 00:28:33.200
Yeah.
00:28:33.359 --> 00:28:41.839
And you know, honestly, this is a problem that is pretty deep in the sense that the mo the error modes that can happen are themselves on an unknown, the space of that.
00:28:42.000 --> 00:28:46.799
Like we talked about hallucinations and fabrications, we talked about omissions, but there's others.
00:28:46.880 --> 00:28:58.720
There's distortion, which is the idea that you might have a claim that says, uh, you know, this incident was really bad, but the hallucination or the hallucination, sorry, the LLM claim says this incident was catastrophic.
00:28:58.880 --> 00:29:05.359
It's a small word change, it's in the right direction, but it just inflated the the the degree to which something was said.
00:29:05.519 --> 00:29:07.279
So that's a distortion in a sense.
00:29:07.440 --> 00:29:16.480
And that's that happens all the time with hallucinations with the with the LLMs because they are, you know, they're set up in a certain way that makes them say things in a particular direction.
00:29:16.799 --> 00:29:18.160
Um there's also framing.
00:29:18.240 --> 00:29:23.440
So when you have a summary, uh the summary is not just simply quoting text from the docs, right?
00:29:23.519 --> 00:29:24.480
From the source docs.
00:29:24.640 --> 00:29:26.480
It's summarizing kind of abstractively.
00:29:26.559 --> 00:29:34.720
It's saying, you know, it's kind of um it's giving you not the specific things that happened, you know, it says it says our higher level stuff.
00:29:34.799 --> 00:29:42.319
So for example, if a if a document said something like, you know, the patient went for a walk at 8 a.m., the summary might say the patient went for a walk in the morning.
00:29:42.640 --> 00:29:42.799
Right?
00:29:42.960 --> 00:29:45.279
That's an abstraction of the specific 8 a.m.
00:29:45.440 --> 00:29:45.759
aspect.
00:29:45.920 --> 00:29:46.799
It's not wrong.
00:29:47.119 --> 00:29:50.079
In fact, I would say that's fine as a summary in that in that instance.
00:29:50.400 --> 00:29:52.160
But it could be wrong in other instances.
00:29:52.480 --> 00:29:53.839
Yeah, but as and you're right.
00:29:53.920 --> 00:29:58.720
And and in fact, as you're doing this summarization, you are going to be slightly different from your sources.
00:29:58.880 --> 00:30:05.359
And I think that makes evaluation harder because it's okay to deviate from the sources, but only only in a certain way.
00:30:05.519 --> 00:30:07.920
And we as humans don't fully understand what that means.
00:30:08.079 --> 00:30:09.119
Only in the relevant way.
00:30:09.200 --> 00:30:10.160
Maybe the fact of 8 a.m.
00:30:10.319 --> 00:30:11.680
walks is not relevant at all.
00:30:11.839 --> 00:30:17.039
In which case, it's okay to say you just went went for a walk in the morning, because that's not a relevant you know, fact.
00:30:17.440 --> 00:30:19.839
Um, but it's important enough that you put it in there.
00:30:20.000 --> 00:30:20.720
So there's that.
00:30:20.799 --> 00:30:23.920
So there's distortions, there's omissions, there's also attribution errors.
00:30:24.000 --> 00:30:31.920
Like so-and-so said X, you know, like John said X, um, LLMs are known to change that to somebody else's name.
00:30:33.119 --> 00:30:38.480
And so you how do you how do you know what the source at which it's misattribution is another issue.
00:30:38.640 --> 00:30:40.720
So you don't know who whether the output is correct.
00:30:40.799 --> 00:30:42.880
So there needs to be evaluations along those lines.
00:30:43.200 --> 00:30:56.720
And then you have other range of issues, which is um if you have um, you know, and this doesn't, I don't know, pop up as much maybe in the medical space, but it this sort of issue pops up in other spaces where you're reading a news report and they're reporting about what somebody else said.
00:30:56.960 --> 00:31:03.599
You know, the the the the New York Times story might say sources said blah, blah, blah, then that's a different attribution.
00:31:03.759 --> 00:31:05.759
And that gets really complex very quickly.
00:31:06.000 --> 00:31:16.400
My point is there is a whole the way we speak and the way we use language has so many nuances that getting to the right summary is actually turns out to be a really challenging task.
00:31:16.559 --> 00:31:19.680
And what it means for a right summary itself is a challenging task.
00:31:19.839 --> 00:31:24.000
And I think these guys are doing it systematically, going through the prompt and building out the system.
00:31:24.319 --> 00:31:24.640
For sure.
00:31:24.799 --> 00:31:27.680
I mean, I think this is like absolutely a great use case.
00:31:28.319 --> 00:31:38.079
And if you think about what healthcare data looks like, a lot of it is in in weird formats because it might be in the medical record, which was created to bill insurance.
00:31:38.160 --> 00:31:38.319
Yeah.
00:31:38.559 --> 00:31:44.880
It might be in, you know, phone notes from a discussion with the patient, it could be in lots of different locations.
00:31:45.039 --> 00:31:48.960
It could be sometimes places have like multiple different record systems.
00:31:49.119 --> 00:31:50.000
That's really common.
00:31:50.160 --> 00:32:09.440
And so it's like, well, how are you if if I'm a human going through, I'm like clicking through this system and that system, and then I'm reading these notes, and then I'm going and looking at the labs and the structured data, and I'm trying to synthesize all that in my mind, and then a thousand clicks later, like you can imagine how long that takes someone to try to figure out and synthesize.
00:32:09.680 --> 00:32:17.359
And so this seems like a great use case as long as it's like reason reasonably accurate, is I guess that's the problem.
00:32:17.519 --> 00:32:22.400
It's like it needs to be accurate if you're gonna be helping the patient.
00:32:22.720 --> 00:32:22.880
Right.
00:32:22.960 --> 00:32:28.640
Accurate, relevant, the right, right level of concision, chronologically, all of that, right?
00:32:28.880 --> 00:32:32.720
And um, and an appropriately responsive to a query you might ask.
00:32:32.880 --> 00:32:36.720
Maybe you're not asking just for summary, but you're saying, did the patient do XYZ?
00:32:37.039 --> 00:32:43.119
And that's also available in the records, but the answer needs to be uh has to have the correct reasoning patterns.
00:32:43.279 --> 00:33:03.519
Uh a lot of my work right now has to do with trying to figure out how can we incorporate structured reasoning patterns so that the outputs that are produced are at least based, not just um not we don't just rely on the LLM's own internal process, but we kind of structure the process so that we get something that's can be can be tracked and we have some kind of auditing capability, right?
00:33:03.680 --> 00:33:08.559
At the end of the day, if the LLM gives you wrong answers, you kind of want to know why, so that you can try to fix it.
00:33:08.799 --> 00:33:14.960
Um it's really hard to do that with today's LLMs because they're giant black boxes that we have no idea how they work internally.
00:33:15.279 --> 00:33:22.880
Um so hopefully going forward, we'll have better systems that can sort of scaffold that LLM so that at least we can start to trust the output more.
00:33:23.119 --> 00:33:24.079
Yeah, I think that's it.
00:33:24.160 --> 00:33:30.079
It's like we need to have really good guardrails on something like this because this has great clinical utility.
00:33:30.160 --> 00:33:34.559
And in fact, in this paper, they were talking about how they make 300 summaries every weekday.
00:33:34.720 --> 00:33:40.000
And that, you know, most people said, hey, this this decreases the time that I spend trying to review a chart.
00:33:40.079 --> 00:33:40.240
Yeah.
00:33:40.480 --> 00:33:41.680
This is useful.
00:33:41.920 --> 00:33:48.240
Um, you know, the most people said they didn't see factually incorrect stuff, but some people did.
00:33:48.400 --> 00:33:55.279
And as you were mentioning, just because you don't see it does doesn't mean it isn't there, especially when it comes to like an omission or a caveat.
00:33:55.440 --> 00:33:58.079
So that that I take with like sort of a grain of salt.
00:33:58.400 --> 00:34:12.639
However, I can imagine something like this kind of speeding up how quickly you can understand who a patient is, what they're going through, what their concerns might be, what their goals might be, and then say, okay, well, these are the next steps that we're gonna take.
00:34:12.719 --> 00:34:21.199
And so you spend the time working with the patient on their interventions that are gonna help them in the long run, whether they're, you know, social or other, other.
00:34:21.599 --> 00:34:27.119
Um, and then you spend less time sort of in that chart review mode, which can just take forever.
00:34:27.360 --> 00:34:27.920
Yeah, yeah, yeah.
00:34:28.000 --> 00:34:38.880
And and you know, you as a human might be, I mean, the often argument the argument is made is that the human is also makes you know omissions and errors and hallucinations and all that stuff.
00:34:38.960 --> 00:34:40.079
That I mean, maybe not hallucinations.
00:34:40.400 --> 00:34:45.840
I was gonna say, I'm not sure about the hallucinations as much, but I'm uh yes, when you're reviewing tons and tons of material, admitting it things.
00:34:46.239 --> 00:34:47.440
You might miss things, right?
00:34:47.599 --> 00:34:56.000
Or you might see something chronologically and not realize that X has has happened already and then why at why time it was fixed or whatever.
00:34:56.159 --> 00:35:00.639
Like, sure, we're not um, yeah, we we are error prone too, right?
00:35:00.719 --> 00:35:02.559
Like, yeah, but in a different way.
00:35:02.880 --> 00:35:13.039
But I was gonna say, I want people to realize that the reason this seems so exciting is because we can dump a ton of text to the LLM and it produces what seems like a coherent summary.
00:35:13.199 --> 00:35:16.880
That is very exciting and it almost seems like a silver bullet to all our problems.
00:35:17.039 --> 00:35:24.880
And it what I'm trying to state is that it's not, and there are issues, some of them much more subtle than others, that surface at times that you have no way to predict.
00:35:25.039 --> 00:35:30.800
And that makes this super challenging as um uh a tool to use because it's kind of random.
00:35:30.880 --> 00:35:36.320
It can do things, it can do amazing things, but it can also do things that you can't control exactly.
00:35:36.559 --> 00:35:44.000
And and that makes, I think, using these tools, one has to be very careful in how they're deployed, how they're used, and then how they're evaluated, right?
00:35:44.079 --> 00:35:45.360
We talked about that as well.
00:35:45.519 --> 00:35:47.760
Um, I'm excited for this direction.
00:35:47.920 --> 00:35:48.239
Yeah, I know.
00:35:48.320 --> 00:35:49.039
That's what I was gonna say.
00:35:49.119 --> 00:35:51.679
Like, oh, well, so what do we do in this case?
00:35:51.920 --> 00:35:55.199
Because I do feel like it's a really great use case.
00:35:55.360 --> 00:36:09.760
I mean, I think there's some component you were talking about, how it's a little bit of a band-aid because you have different data and different sources and you're trying to like synthesize all of it and it's coming across in dynamic and nuanced ways, and some of it's qualitative and some of it's quantitative.
00:36:09.920 --> 00:36:14.480
And so this is like, oh, well, can we just dump it all into the LLM and it will just tell us?
00:36:14.800 --> 00:36:24.000
So I do think there's like a little bit of a band-aid component here where it's like, well, maybe if the data were structured better and stored better, it would be easier for anyone to look at.
00:36:24.159 --> 00:36:24.400
Yes.
00:36:24.719 --> 00:36:49.599
But I also think this is this is like a really cool and very practical opportunity when you think about a population as a whole, is like, hey, how can we intervene not just on like 15 people, but 500 people because we can get the summaries faster, which means we can spend our human time actually talking to the patient and figuring out, hey, like we need to get you housing or hey, like we need to get you access to a new primary care provider, whatever it is.
00:36:49.840 --> 00:36:50.079
Yes.
00:36:50.239 --> 00:37:05.679
And I think the way to get better at this, and I think they're doing some of that, is in fact having humans not just look at the output and see if it's correct, but humans when they when summaries are generated, whether it's generated by a human or an LM, to ask why one summary works and why another doesn't.
00:37:05.840 --> 00:37:08.320
Like, what about the summary is correct?
00:37:08.559 --> 00:37:12.559
What about it feels more correct than maybe changing a word here or changing something else?
00:37:12.719 --> 00:37:14.719
Like, why is this the right kind of summary?
00:37:14.960 --> 00:37:20.159
Understand the level of detail, then you'll better understand what the fabrications and omissions are, what the distortions are.
00:37:20.320 --> 00:37:26.320
And understanding that then can guide you in your prompt development because then um you can write better prompts for these LLMs.
00:37:26.400 --> 00:37:33.199
But I think the baseline issue there is that the human has to do a bit of the hard work in really digging in and internalizing the data.
00:37:33.360 --> 00:37:35.519
And I say this because you mentioned this before.
00:37:35.760 --> 00:37:38.000
Humans and LLMs have different ways of thinking.
00:37:38.159 --> 00:37:46.719
When we are given a patient's history or any piece of set of documents that we have to make sense of, our mental process is building world models.
00:37:46.800 --> 00:37:56.639
That means we imagine the patient, we imagine, you know, if it says the patient went for a walk at 8 a.m., we immediately have a visual of that whole setup, what it means to go for a walk at 8 a.m.
00:37:56.880 --> 00:38:00.079
We immediately understand the patient woke up before 8 a.m., obviously.
00:38:00.320 --> 00:38:02.159
We understand things that follow from that.
00:38:02.239 --> 00:38:05.280
They went for the walk, went for a walk, so they must have taken X number of steps.
00:38:05.440 --> 00:38:06.480
We know that, right?
00:38:06.639 --> 00:38:09.119
We know all kinds of information that they're active that morning.
00:38:09.199 --> 00:38:14.400
We things we imply, things we presuppose, all of that is captured in our mental model of the person.
00:38:14.559 --> 00:38:18.960
So as we read more text, we incorporate that, and all of that has to make sense for us all the time.
00:38:19.199 --> 00:38:21.199
That is not how LLMs think.
00:38:21.360 --> 00:38:25.119
They're not sitting there thinking, hey, is this the pay what's this patient about?
00:38:25.280 --> 00:38:25.920
What are they doing?
00:38:26.000 --> 00:38:26.880
Does this all make sense?
00:38:27.039 --> 00:38:28.880
They're not building that kind of model in their head.
00:38:29.039 --> 00:38:38.400
So that's why I think we need a human, at least at these initial stages of development, to still look at some of this data and understand it and understand why the summary works and why it doesn't.
00:38:38.639 --> 00:38:44.239
And once we're able to really synthesize that into the prompt, then we're gonna get better output from the LLMs, I think.
00:38:44.480 --> 00:38:44.960
That's great.
00:38:45.039 --> 00:38:55.039
I mean, I'm so hopeful of this as a use case because I do think um it would let us spend the person hours really intervening on the patients where it matters the most.
00:38:55.119 --> 00:38:55.360
Yes.
00:38:55.519 --> 00:39:02.159
And so having a summarization tool that is accurate and trustworthy is definitely something worth digging into.
00:39:02.320 --> 00:39:03.199
And I think you're right.
00:39:03.360 --> 00:39:09.440
The the more we work on figuring out, hey, what is a human, what would a human do in this situation?
00:39:09.599 --> 00:39:11.519
How is a human thinking this through?
00:39:11.679 --> 00:39:14.639
How can we improve that the accuracy?
00:39:14.800 --> 00:39:22.159
LLM as a judge is one component of it, but there's lots of different components that need to go in, the better we're gonna get at this problem.
00:39:22.400 --> 00:39:22.960
Absolutely.
00:39:23.360 --> 00:39:23.760
All right.
00:39:23.920 --> 00:39:25.119
Well, thanks for joining us.
00:39:25.199 --> 00:39:27.840
I think we'll end here and we'll see you next time on Code and Cure.
00:39:28.079 --> 00:39:28.960
Bye for now,