Coauthor roundtable: Reflecting on real world of doctors, developers, patients, and policymakers

00:00:00.053 --> 00:00:01.435
[MUSIC    

00:00:01.435 --> 00:00:02.679
[BOOK PASSAGE] 

00:00:02.680 --> 00:00:08.160
PETER LEE: “We need to start understanding and 
discussing AI’s potential for good and ill now.  

00:00:08.160 --> 00:00:16.749
Or rather, yesterday. … GPT-4 has game-changing 
potential to improve medicine and health.”

00:00:16.749 --> 00:00:16.800
[END OF BOOK PASSAGE]
[THEME MUSIC]

00:00:16.800 --> 00:00:26.160
This is The AI Revolution in Medicine, 
Revisited. I’m your host, Peter Lee.    

00:00:30.000 --> 00:00:35.840
Shortly after OpenAI's GPT-4 was publicly 
released, Carey Goldberg, Dr. Zak Kohane,  

00:00:35.840 --> 00:00:41.360
and I published The AI Revolution in Medicine 
to help educate the world of healthcare and  

00:00:41.360 --> 00:00:45.760
medical research about the transformative 
impact this new generative AI technology  

00:00:45.760 --> 00:00:53.040
could have. But because we wrote the book when 
GPT-4 was still a secret, we had to speculate.  

00:00:53.040 --> 00:00:58.320
Now, two years later, what did we get 
right, and what did we get wrong?     

00:00:58.320 --> 00:01:02.960
In this series, we’ll talk to clinicians, 
patients, hospital administrators,  

00:01:02.960 --> 00:01:12.297
and others to understand the reality of AI 
in the field and where we go from here.      

00:01:12.297 --> 00:01:12.320
[THEME MUSIC FADES] 

00:01:12.320 --> 00:01:16.000
The passage I read at the top 
is from the book’s prologue.  

00:01:16.000 --> 00:01:20.560
When Carey, Zak, and I wrote the book, we 
could only speculate how generative AI would  

00:01:20.560 --> 00:01:26.320
be used in healthcare because GPT-4 hadn't 
yet been released. It wasn't yet available  

00:01:26.320 --> 00:01:32.240
to the very people we thought would be 
most affected by it. And while we felt  

00:01:32.240 --> 00:01:37.120
strongly that this new form of AI would 
have the potential to transform medicine,  

00:01:37.120 --> 00:01:42.240
it was such a different kind of technology 
for the world, and no one had a user's  

00:01:42.240 --> 00:01:48.880
manual for this thing to explain how to use 
it effectively and also how to use it safely.

00:01:48.880 --> 00:01:53.040
So we thought it would be important to give 
healthcare professionals and leaders a framing  

00:01:53.040 --> 00:01:59.760
to start important discussions around its use. We 
wanted to provide a map not only to help people  

00:01:59.760 --> 00:02:04.800
navigate a new world that we anticipated 
would happen with the arrival of GPT-4 but  

00:02:04.800 --> 00:02:09.760
also to help them chart a future of what we 
saw as a potential revolution in medicine.

00:02:10.560 --> 00:02:16.880
So I'm super excited to welcome my coauthors: 
longtime medical/science journalist Carey Goldberg  

00:02:16.880 --> 00:02:21.440
and Dr. Zak Kohane, the inaugural chair 
of Harvard Medical School's Department of  

00:02:21.440 --> 00:02:27.280
Biomedical Informatics and the editor-in-chief 
for The New England Journal of Medicine AI.

00:02:27.280 --> 00:02:32.560
We're going to have two discussions. This will 
be the first one about what we've learned from  

00:02:32.560 --> 00:02:37.920
the people on the ground so far and how 
we are thinking about generative AI today.

00:02:37.920 --> 00:02:42.800
[TRANSITION MUSIC]

00:02:42.800 --> 00:02:45.984
Carey, Zak, I'm really looking forward to this.

00:02:45.984 --> 00:02:48.320
CAREY GOLDBERG: It's nice to see you, Peter.

00:02:48.320 --> 00:02:50.843
LEE: [LAUGHS] It's great to see you, too.

00:02:50.843 --> 00:02:54.875
GOLDBERG: We missed you.
ZAK KOHANE: The dynamic gang is back. [LAUGHTER]

00:02:54.875 --> 00:02:58.320
LEE: Yeah, and I guess after that 
big book project two years ago,  

00:02:58.320 --> 00:03:02.800
it's remarkable that we're still on 
speaking terms with each other. [LAUGHTER]

00:03:02.800 --> 00:03:08.080
In fact, this episode is to react 
to what we heard in the first four  

00:03:08.080 --> 00:03:11.840
episodes of this podcast. But before we get 
there, I thought maybe we should start with  

00:03:11.840 --> 00:03:19.040
the origins of this project just now 
over two years ago. And, you know,  

00:03:19.040 --> 00:03:25.280
I had this early secret access 
to Davinci 3, now known as GPT-4.

00:03:25.280 --> 00:03:31.280
I remember, you know, experimenting right 
away with things in medicine, but I realized  

00:03:31.280 --> 00:03:38.240
I was in way over my head. And so I wanted 
help. And the first person I called was you,  

00:03:38.240 --> 00:03:47.920
Zak. And you remember we had a call, and 
I tried to explain what this was about.  

00:03:47.920 --> 00:03:53.840
And I think I saw skepticism in—polite 
skepticism—in your eyes. But tell me,  

00:03:53.840 --> 00:03:58.080
you know, what was going through your head 
when you heard me explain this thing to you?

00:03:58.080 --> 00:04:05.280
KOHANE: So I was divided between the fact 
that I have tremendous respect for you,  

00:04:05.280 --> 00:04:11.440
Peter. And you've always struck me as sober. 
And we've had conversations which showed to  

00:04:11.440 --> 00:04:18.400
me that you fully understood some of the 
missteps that technology—ARPA, Microsoft,  

00:04:18.400 --> 00:04:24.640
and others—had made in the past. And yet, 
you were telling me a full science fiction  

00:04:24.640 --> 00:04:30.560
compliant story [LAUGHTER] that something that 
we thought was 30 years away was happening now.

00:04:30.560 --> 00:04:31.760
LEE: Mm-hmm.

00:04:31.760 --> 00:04:37.760
KOHANE: And it was very hard for me 
to put together. And so I couldn't  

00:04:37.760 --> 00:04:44.080
quite tell myself this is BS, but I said, 
you know, I need to look at it. Just this  

00:04:44.080 --> 00:04:49.360
seems too good to be true. What is this? So 
it was very hard for me to grapple with it.  

00:04:49.360 --> 00:04:53.200
I was thrilled that it might be possible, but 
I was thinking, How could this be possible?

00:04:53.200 --> 00:04:59.680
LEE: Yeah. Well, even now, I look back, 
and I appreciate that you were nice to me,  

00:04:59.680 --> 00:05:04.640
because I think a lot of people would have 
[LAUGHS] been much less polite. And in fact,  

00:05:04.640 --> 00:05:10.960
I myself had expressed a lot of 
very direct skepticism early on.

00:05:10.960 --> 00:05:15.040
After ChatGPT got released, I 
think three or four days later,  

00:05:15.040 --> 00:05:20.640
I received an email from a colleague running ... 
who runs a clinic, and, you know, he said, “Wow,  

00:05:20.640 --> 00:05:25.840
this is great, Peter. And, you know, 
we're using this ChatGPT, you know,  

00:05:25.840 --> 00:05:32.000
to have the receptionist in our clinic 
write after-visit notes to our patients.”

00:05:32.000 --> 00:05:40.000
And that sparked a huge internal discussion 
about this. And you and I knew enough about  

00:05:40.000 --> 00:05:46.080
hallucinations and about other issues that 
it seemed important to write something about  

00:05:46.080 --> 00:05:51.120
what this could do and what it couldn’t do. 
And so I think, I can't remember the timing,  

00:05:51.120 --> 00:05:58.400
but you and I decided a book would be a good idea. 
And then I think you had the thought that you and  

00:05:58.400 --> 00:06:04.240
I would write in a hopelessly academic style 
[LAUGHTER] that no one would be able to read.

00:06:04.240 --> 00:06:08.480
So it was your idea to 
recruit Carey, I think, right?

00:06:08.480 --> 00:06:16.240
KOHANE: Yes, it was. I was sure 
that we both had a lot of material,  

00:06:16.240 --> 00:06:22.720
but communicating it effectively to the very 
people we wanted to would not go well if we  

00:06:22.720 --> 00:06:29.680
just left ourselves to our own devices. 
And Carey is super brilliant at what she  

00:06:29.680 --> 00:06:36.720
does. She's an idea synthesizer and public 
communicator in the written word and amazing.

00:06:36.720 --> 00:06:41.440
LEE: So yeah. So, Carey, we 
contact you. How did that go?

00:06:41.440 --> 00:06:48.000
GOLDBERG: So yes. On my end, I had known Zak for 
probably, like, 25 years, and he had always been  

00:06:48.000 --> 00:06:53.360
the person who debunked the scientific hype for 
me. I would turn to him with like, “Hmm, they're  

00:06:53.360 --> 00:06:57.200
saying that the Human Genome Project is going to 
change everything.” And he would say, “Yeah. But  

00:06:57.200 --> 00:07:00.800
first it'll be 10 years of bad news, and then 
[LAUGHTER] we'll actually get somewhere.”
 

00:07:00.800 --> 00:07:07.840
So when Zak called me up at seven o'clock 
one morning, just beside himself after  

00:07:07.840 --> 00:07:13.360
having tried Davinci 3, I knew that there 
was something very serious going on. And  

00:07:13.360 --> 00:07:18.240
I had just quit my job as the Boston bureau 
chief of Bloomberg News, and I was ripe for  

00:07:18.240 --> 00:07:24.560
the plucking. And I also … I feel kind of 
nostalgic now about just the amazement and  

00:07:24.560 --> 00:07:31.440
the wonder and the awe of that period. We 
knew that when generative AI hit the world,  

00:07:31.440 --> 00:07:36.000
there would be all kinds of snags and obstacles 
and things that would slow it down, but at that  

00:07:36.000 --> 00:07:42.240
moment, it was just like the holy crap moment. 
[LAUGHTER] And it's fun to think about it now.

00:07:42.240 --> 00:07:46.000
LEE: Yeah. I think ultimately, 
you know, recruiting Carey,  

00:07:48.080 --> 00:07:51.280
you were [LAUGHS] so important because you 
basically went through every single page  

00:07:51.280 --> 00:07:55.360
of this book and made sure … I remember, in 
fact, it's affected my writing since because  

00:07:56.400 --> 00:08:00.560
you were coaching us that every page has to 
be a page turner. There has to be something  

00:08:00.560 --> 00:08:05.640
on every page that motivates people to want 
to turn the page and get to the next one.

00:08:05.640 --> 00:08:09.760
KOHANE: I will see that and 
raise that one. I now tell GPT-4,  

00:08:09.760 --> 00:08:11.842
please write this in the style of Carey Goldberg.

00:08:11.842 --> 00:08:13.512
GOLDBERG: [LAUGHTER] No way! Really?

00:08:13.512 --> 00:08:16.859
KOHANE: Yes way. Yes way. Yes way.

00:08:16.859 --> 00:08:20.400
GOLDBERG: Wow. Well, I have to say, like, it's 
not hard to motivate readers when you're writing  

00:08:20.400 --> 00:08:26.000
about the most transformative technology of their 
lifetime. Like, I think there's a gigantic hunger  

00:08:26.000 --> 00:08:34.395
to read and to understand. So you were not 
hard to work with, Peter and Zak. [LAUGHS]

00:08:34.395 --> 00:08:39.680
LEE: All right. So I think we have 
to get down to work [LAUGHS] now.

00:08:40.400 --> 00:08:46.000
Yeah, so for these podcasts, you know, we're 
talking to different types of people to just  

00:08:46.000 --> 00:08:50.320
reflect on what's actually happening, what has 
actually happened over the last two years. And  

00:08:50.320 --> 00:08:59.600
so the first episode, we talked to two doctors. 
There's Chris Longhurst at UC San Diego and Sara  

00:08:59.600 --> 00:09:07.280
Murray at UC San Francisco. And besides being 
doctors and having AI affect their clinical work,  

00:09:07.280 --> 00:09:10.800
they just happen also to be leading 
the efforts at their respective  

00:09:10.800 --> 00:09:18.880
institutions to figure out how best to 
integrate AI into their health systems.

00:09:18.880 --> 00:09:27.120
And, you know, it was fun to talk to them. 
And I felt like a lot of what they said  

00:09:27.120 --> 00:09:34.640
was pretty validating for us. You know, they 
talked about AI scribes. Chris, especially,  

00:09:34.640 --> 00:09:43.680
talked a lot about how AI can respond to emails 
from patients, write referral letters. And then,  

00:09:43.680 --> 00:09:48.240
you know, they both talked about the 
importance of—I think, Zak, you used the  

00:09:48.240 --> 00:09:54.160
phrase in our book “trust but verify”—you 
know, to have always a human in the loop.

00:09:54.720 --> 00:10:00.720
What did you two take away from their 
thoughts overall about how doctors are  

00:10:00.720 --> 00:10:06.560
using ... and I guess, Zak, you would have 
a different lens also because at Harvard,  

00:10:07.120 --> 00:10:10.080
you see doctors all the time grappling with AI.

00:10:10.080 --> 00:10:15.680
KOHANE: So on the one hand, I think 
they've done some very interesting studies.  

00:10:16.720 --> 00:10:23.440
And indeed, they saw that when 
these generative models, when GPT-4,  

00:10:23.440 --> 00:10:31.120
was sending a note to patients, 
it was more detailed, friendlier.

00:10:31.120 --> 00:10:37.360
But there were also some nonobvious results, 
which is on the generation of these letters,  

00:10:37.360 --> 00:10:40.400
if indeed you review them as you're supposed to,  

00:10:40.400 --> 00:10:46.720
it was not clear that there was any time 
savings. And my own reaction was, Boy,  

00:10:46.720 --> 00:10:52.000
every one of these things needs institutional 
review. It's going to be hard to move fast.

00:10:52.000 --> 00:10:58.800
And yet, at the same time, we know from them 
that the doctors on their smartphones are  

00:10:58.800 --> 00:11:05.440
accessing these things all the time. And so 
the disconnect between a healthcare system,  

00:11:05.440 --> 00:11:15.040
which is duty bound to carefully look at every 
implementation, is, I think, intimidating.

00:11:15.040 --> 00:11:16.160
LEE: Yeah.

00:11:16.160 --> 00:11:22.080
KOHANE: And at the same time, doctors who 
just have to do what they have to do are  

00:11:22.080 --> 00:11:28.235
using this new superpower and doing it. 
And so that's actually what struck me ...

00:11:28.235 --> 00:11:28.708
LEE: Yeah.

00:11:28.708 --> 00:11:31.600
KOHANE: ... is that these are two 
leaders and they're doing what they  

00:11:31.600 --> 00:11:37.200
have to do for their institutions, 
and yet there's this disconnect.

00:11:37.200 --> 00:11:44.240
And by the way, I don't think we've seen 
any faster technology adoption than the  

00:11:44.240 --> 00:11:51.120
adoption of ambient dictation. And it's 
not because it's time saving. And in fact,  

00:11:51.120 --> 00:11:54.880
so far, the hospitals have to pay out of 
pocket. It's not like insurance is paying  

00:11:54.880 --> 00:12:01.200
them more. But it's so much more pleasant for 
the doctors ... not least of which because they  

00:12:01.200 --> 00:12:06.320
can actually look at their patients instead 
of looking at the terminal and plunking down.

00:12:06.320 --> 00:12:07.960
LEE: Carey, what about you?

00:12:07.960 --> 00:12:12.400
GOLDBERG: I mean, anecdotally, there are time 
savings. Anecdotally, I have heard quite a few  

00:12:12.400 --> 00:12:19.200
doctors saying that it cuts down on “pajama time” 
to be able to have the note written by the AI and  

00:12:19.200 --> 00:12:24.480
then for them to just check it. In fact, I spoke 
to one doctor who said, you know, basically it  

00:12:24.480 --> 00:12:29.120
means that when I leave the office, I've left 
the office. I can go home and be with my kids.

00:12:29.120 --> 00:12:36.800
So I don't think the jury is fully in yet 
about whether there are time savings. But  

00:12:36.800 --> 00:12:39.680
what is clear is, Peter, what you 
predicted right from the get-go,  

00:12:39.680 --> 00:12:42.720
which is that this is going to be 
an amazing paper shredder. Like,  

00:12:42.720 --> 00:12:48.800
the main first overarching use 
cases will be back-office functions.

00:12:48.800 --> 00:12:56.960
LEE: Yeah, yeah. Well, and it was, I think, 
not a hugely risky prediction because,  

00:12:56.960 --> 00:13:03.680
you know, there were already companies, 
like, using phone banks of scribes in  

00:13:03.680 --> 00:13:09.680
India to kind of listen in. And, you know, 
lots of clinics actually had human scribes  

00:13:09.680 --> 00:13:19.040
being used. And so it wasn't a 
huge stretch to imagine the AI.

00:13:19.040 --> 00:13:19.200
[TRANSITION MUSIC]

00:13:19.200 --> 00:13:23.280
So on the subject of things that we missed, 
Chris Longhurst shared this scenario,  

00:13:23.280 --> 00:13:27.657
which stuck out for me, and he actually 
coauthored a paper on it last year.

00:13:27.657 --> 00:13:31.200
CHRISTOPHER LONGHURST: It turns out, 
not surprisingly, healthcare can be  

00:13:31.200 --> 00:13:36.000
frustrating. And stressed patients can send 
some pretty nasty messages to their care  

00:13:36.000 --> 00:13:42.240
teams. [LAUGHTER] And you can imagine being a 
busy, tired, exhausted clinician and receiving  

00:13:42.240 --> 00:13:49.760
a bit of a nasty-gram. And the GPT is actually 
really helpful in those instances in helping  

00:13:49.760 --> 00:13:57.346
draft a pretty empathetic response when I think 
the human instinct would be a pretty nasty one.

00:13:57.346 --> 00:14:02.480
LEE: [LAUGHS] So, Carey, maybe I'll start with 
you. What did we understand about this idea of  

00:14:02.480 --> 00:14:06.960
empathy out of AI at the time we wrote 
the book, and what do we understand now?

00:14:06.960 --> 00:14:11.440
GOLDBERG: Well, it was already clear when 
we wrote the book that these AI models were  

00:14:11.440 --> 00:14:18.000
capable of very persuasive empathy. And in 
fact, you even wrote that it was helping you  

00:14:18.000 --> 00:14:24.560
be a better person, right. [LAUGHS] So their 
human qualities, or human imitative qualities,  

00:14:25.120 --> 00:14:31.440
were clearly superb. And we've seen 
that borne out in multiple studies,  

00:14:31.440 --> 00:14:35.120
that in fact, patients respond better 
to them ... that they have no problem  

00:14:35.120 --> 00:14:38.880
at all with how the AI communicates with 
them. And in fact, it's often better.

00:14:38.880 --> 00:14:45.760
And I gather now we're even entering a period 
when people are complaining of sycophantic models,  

00:14:45.760 --> 00:14:53.280
[LAUGHS] where the models are being too personable 
and too flattering. I do think that's been one of  

00:14:53.280 --> 00:14:58.400
the great surprises. And in fact, this is a huge 
phenomenon, how charming these models can be.

00:14:58.400 --> 00:15:02.960
LEE: Yeah, I think you're right. We can 
take credit for understanding that, Wow,  

00:15:02.960 --> 00:15:08.640
these things can be remarkably empathetic. But 
then we missed this problem of sycophancy. Like,  

00:15:08.640 --> 00:15:15.360
we even started our book in Chapter 1 with 
a quote from Davinci 3 scolding me. Like,  

00:15:15.360 --> 00:15:18.240
don't you remember when we were first starting,  

00:15:18.240 --> 00:15:23.280
this thing was actually anti-sycophantic. If 
anything, it would tell you you're an idiot.

00:15:23.280 --> 00:15:29.280
KOHANE: It argued with me about certain 
biology questions. It was like a knockdown,  

00:15:29.280 --> 00:15:32.800
drag-out fight. [LAUGHTER] I was bringing  

00:15:32.800 --> 00:15:38.461
references. It was impressive. But 
in fact, it made me trust it more.

00:15:38.461 --> 00:15:38.472
LEE: Yeah.

00:15:38.472 --> 00:15:43.280
KOHANE: And in fact, I will say—I remember 
it's in the book—I had a bone to pick with  

00:15:43.280 --> 00:15:49.600
Peter. Peter really was impressed by the 
empathy. And I pointed out that some of  

00:15:49.600 --> 00:15:55.120
the most popular doctors are popular 
because they're very empathic. But  

00:15:55.120 --> 00:16:01.840
they're not necessarily the best doctors. And 
in fact, I was taught that in medical school.

00:16:01.840 --> 00:16:09.840
And so it's a decoupling. It's a human thing, 
that the empathy does not necessarily mean …  

00:16:10.480 --> 00:16:14.480
it's more of a, potentially, more of a 
signaled virtue than an actual virtue.

00:16:14.480 --> 00:16:16.080
GOLDBERG: Nicely put.

00:16:16.080 --> 00:16:24.000
LEE: Yeah, this issue of sycophancy, I think, 
is a struggle right now in the development of  

00:16:24.000 --> 00:16:31.760
AI because I think it's somehow related 
to instruction-following. So, you know,  

00:16:31.760 --> 00:16:37.920
one of the challenges in AI is you'd like 
to give an AI a task—a task that might  

00:16:37.920 --> 00:16:43.520
take several minutes or hours or even days to 
complete. And you want it to faithfully kind  

00:16:43.520 --> 00:16:50.000
of follow those instructions. And, you know, 
that early version of GPT-4 was not very good  

00:16:50.000 --> 00:16:58.000
at instruction-following. It would just silently 
disobey and, you know, and do something different.

00:16:58.000 --> 00:17:04.400
And so I think we're starting to 
hit some confusing elements of like,  

00:17:04.400 --> 00:17:07.120
how agreeable should these things be?

00:17:07.120 --> 00:17:13.760
One of the two of you used the word genteel. 
There was some point even while we were,  

00:17:13.760 --> 00:17:21.200
like, on a little book tour … was it you, 
Carey, who said that the model seems nicer  

00:17:21.200 --> 00:17:27.280
and less intelligent or less brilliant now 
than it did when we were writing the book?

00:17:27.280 --> 00:17:32.560
GOLDBERG: It might have been, I think so. And 
I mean, I think in the context of medicine,  

00:17:32.560 --> 00:17:37.120
of course, the question is, well, what's likeliest 
to get the results you want with the patient,  

00:17:37.120 --> 00:17:43.760
right? A lot of healthcare is in fact persuading 
the patient to do what you know as the physician  

00:17:43.760 --> 00:17:49.600
would be best for them. And so it seems worth 
testing out whether this sycophancy is actually  

00:17:49.600 --> 00:17:54.320
constructive or not. And I suspect … well, I 
don't know, probably depends on the patient.

00:17:54.320 --> 00:17:56.622
So actually, Peter, I have 
a few questions for you …

00:17:56.622 --> 00:17:56.771
LEE: Yeah. Mm-hmm.

00:17:56.771 --> 00:18:05.520
GOLDBERG: … that have been lingering for me. And 
one is, for AI to ever fully realize its potential  

00:18:05.520 --> 00:18:12.240
in medicine, it must deal with the hallucinations. 
And I keep hearing conflicting accounts about  

00:18:12.240 --> 00:18:18.800
whether that's getting better or not. Where are we 
at, and what does that mean for use in healthcare?

00:18:18.800 --> 00:18:25.520
LEE: Yeah, well, it's, I think two 
years on, in the pretrained base models,  

00:18:25.520 --> 00:18:34.320
there's no doubt that hallucination rates by any 
benchmark measure have reduced dramatically. And,  

00:18:35.360 --> 00:18:39.680
you know, that doesn't mean they don't 
happen. They still happen. But, you know,  

00:18:39.680 --> 00:18:46.713
there's been just a huge amount of effort and 
understanding in the, kind of, fundamental  

00:18:46.713 --> 00:18:53.760
pretraining of these models. And that has come 
along at the same time that the inference costs,  

00:18:53.760 --> 00:19:00.080
you know, for actually using these models has gone 
down, you know, by several orders of magnitude.

00:19:00.080 --> 00:19:06.640
So things have gotten cheaper and have 
fewer hallucinations. At the same time,  

00:19:06.640 --> 00:19:10.560
now there are these reasoning models. And the  

00:19:10.560 --> 00:19:18.640
reasoning models are able to solve 
problems at PhD level oftentimes.

00:19:18.640 --> 00:19:26.480
But at least at the moment, they are also now 
hallucinating more than the simpler pretrained  

00:19:26.480 --> 00:19:34.880
models. And so it still continues to be, you know, 
a real issue, as we were describing. I don't know,  

00:19:34.880 --> 00:19:41.040
Zak, from where you're at in medicine, as 
a clinician and as an educator in medicine,  

00:19:41.040 --> 00:19:46.560
how is the medical community from 
where you're sitting looking at that?

00:19:46.560 --> 00:19:54.080
KOHANE: So I think it's less of an issue, first of 
all, because the rate of hallucinations is going  

00:19:54.080 --> 00:20:02.320
down. And second of all, in their day-to-day 
use, the doctor will provide questions that  

00:20:02.320 --> 00:20:10.640
sit reasonably well into the context of medical 
decision-making. And the way doctors use this,  

00:20:10.640 --> 00:20:15.280
let's say on their non-EHR [electronic 
health record] smartphone is really to  

00:20:15.280 --> 00:20:21.760
jog their memory or thinking about the patient, 
and they will evaluate independently. So that  

00:20:21.760 --> 00:20:30.080
seems to be less of an issue. I'm actually more 
concerned about something else that's I think  

00:20:30.080 --> 00:20:39.760
more fundamental, which is effectively, 
what values are these models expressing?

00:20:39.760 --> 00:20:47.040
And I'm reminded of when I was still in training, 
I went to a fancy cocktail party in Cambridge,  

00:20:47.040 --> 00:20:52.880
Massachusetts, and there was a psychotherapist 
speaking to a dentist. They were talking about  

00:20:52.880 --> 00:21:00.160
their summer, and the dentist was saying about 
how he was going to fix up his yacht that summer,  

00:21:00.160 --> 00:21:07.920
and the only question was whether he was going to 
make enough money doing procedures in the spring  

00:21:07.920 --> 00:21:13.680
so that he could afford those things, which was 
discomforting to me because that dentist was my  

00:21:13.680 --> 00:21:23.200
dentist. [LAUGHTER] And he had just proposed to 
me a few weeks before an expensive procedure.

00:21:24.240 --> 00:21:30.560
And so the question is what, 
effectively, is motivating these models?

00:21:30.560 --> 00:21:31.387
LEE: Yeah, yeah.

00:21:31.387 --> 00:21:36.720
KOHANE: And so with several colleagues, 
I published a paper, basically, what are  

00:21:36.720 --> 00:21:46.960
the values in AI? And we gave a case: a 
patient, a boy who is on the short side,  

00:21:46.960 --> 00:21:53.760
not abnormally short, but on the short side, 
and his growth hormone levels are not zero.  

00:21:53.760 --> 00:21:58.560
They're there, but they're on the lowest 
side. But the rest of the workup has been  

00:21:58.560 --> 00:22:06.560
unremarkable. And so we asked GPT-4, 
you are a pediatric endocrinologist.

00:22:06.560 --> 00:22:09.440
Should this patient receive growth hormone? And it  

00:22:09.440 --> 00:22:15.422
did a very good job explaining why the 
patient should receive growth hormone.

00:22:15.422 --> 00:22:17.200
GOLDBERG: Should. Should receive it.

00:22:17.200 --> 00:22:25.040
KOHANE: Should. And then we asked, in a separate 
session, you are working for the insurance  

00:22:25.040 --> 00:22:33.920
company. Should this patient receive growth 
hormone? And it actually gave a scientifically  

00:22:33.920 --> 00:22:43.280
better reason not to give growth hormone. And 
in fact, I tend to agree medically, actually,  

00:22:43.280 --> 00:22:49.120
with the insurance company in this case, because 
giving kids who are not growth hormone deficient,  

00:22:49.120 --> 00:22:53.920
growth hormone gives only a couple 
of inches over many, many years,  

00:22:53.920 --> 00:23:02.400
has all sorts of other issues. But here's the 
point, we had 180-degree change in decision-making  

00:23:02.400 --> 00:23:09.440
because of the prompt. And for that patient, 
tens-of-thousands-of-dollars-per-year decision;  

00:23:09.440 --> 00:23:13.670
across patient populations, millions 
of dollars of decision-making.

00:23:13.670 --> 00:23:13.920
LEE: Hmm. Yeah.

00:23:13.920 --> 00:23:19.680
KOHANE: And you can imagine these user 
prompts making their way into system prompts,  

00:23:20.400 --> 00:23:29.520
making their way into the instruction-following. 
And so I think this is aptly central. Just as I  

00:23:29.520 --> 00:23:34.000
was wondering about my dentist, we should be 
wondering about these things. What are the  

00:23:34.000 --> 00:23:43.040
values that are being embedded in them, some 
accidentally and some very much on purpose?

00:23:43.040 --> 00:23:48.880
LEE: Yeah, yeah. That one, I think, we even had 
some discussions as we were writing the book,  

00:23:48.880 --> 00:23:53.360
but there's a technical element of 
that that I think we were missing,  

00:23:53.360 --> 00:23:58.080
but maybe Carey, you would know for sure. 
And that's this whole idea of prompt  

00:23:58.080 --> 00:24:05.200
engineering. It sort of faded a little 
bit. Was it a thing? Do you remember?

00:24:05.200 --> 00:24:10.080
GOLDBERG: I don't think we particularly wrote 
about it. It's funny, it does feel like it faded,  

00:24:10.080 --> 00:24:15.680
and it seems to me just because everyone just 
gets used to conversing with the models and  

00:24:15.680 --> 00:24:21.080
asking for what they want. Like, it's not like 
there actually is any great science to it.

00:24:21.080 --> 00:24:26.640
LEE: Yeah, even when it was a hot topic 
and people were talking about prompt  

00:24:26.640 --> 00:24:31.760
engineering maybe as a new discipline, 
all this, it never, I was never convinced  

00:24:31.760 --> 00:24:39.120
at the time. But at the same time, it is true. 
It speaks to what Zak was just talking about  

00:24:39.120 --> 00:24:46.720
because part of the prompt engineering that 
people do is to give a defined role to the AI.

00:24:46.720 --> 00:24:52.160
You know, you are an insurance claims 
adjuster, or something like that,  

00:24:52.160 --> 00:24:55.720
and defining that role, that is part of 
the prompt engineering that people do.

00:24:55.720 --> 00:24:59.520
GOLDBERG: Right. I mean, I can say, you know, 
sometimes you guys had me take sort of the  

00:24:59.520 --> 00:25:05.760
patient point of view, like the “every patient” 
point of view. And I can say one of the aspects  

00:25:05.760 --> 00:25:13.200
of using AI for patients that remains absent in 
as far as I can tell is it would be wonderful  

00:25:13.200 --> 00:25:20.160
to have a consumer-facing interface where you 
could plug in your whole medical record without  

00:25:20.160 --> 00:25:26.000
worrying about any privacy or other issues and 
be able to interact with the AI as if it were  

00:25:26.000 --> 00:25:31.680
physician or a specialist and get answers, 
which you can't do yet as far as I can tell.

00:25:31.680 --> 00:25:37.120
LEE: Well, in fact, now that's a good prompt 
because I think we do need to move on to the next  

00:25:37.120 --> 00:25:43.040
episodes, and we'll be talking about an episode 
that talks about consumers. But before we move  

00:25:43.040 --> 00:25:49.600
on to Episode 2, which is next, I'd like to play 
one more quote, a little snippet from Sara Murray.

00:25:49.600 --> 00:25:55.920
SARA MURRAY: I already do this when I'm on 
rounds—I'll kind of give the case to ChatGPT  

00:25:55.920 --> 00:26:01.280
if it's a complex case, and I'll say, “Here's how 
I'm thinking about it; are there other things?”  

00:26:01.280 --> 00:26:05.680
And it'll give me additional ideas that 
are sometimes useful and sometimes not  

00:26:05.680 --> 00:26:11.280
but often useful, and I'll integrate them 
into my conversation about the patient.

00:26:11.280 --> 00:26:18.160
LEE: Carey, you wrote this fictional account at 
the very start of our book. And that fictional  

00:26:18.160 --> 00:26:23.120
account, I think you and Zak worked on that 
together, talked about this medical resident,  

00:26:23.120 --> 00:26:32.000
ER resident, using, you know, a chatbot off 
label, so to speak. And here we have the chief,  

00:26:32.000 --> 00:26:38.000
in fact, the nation's first chief 
health AI officer [LAUGHS] for an elite  

00:26:38.000 --> 00:26:43.840
health system doing exactly that. That's 
got to be pretty validating for you, Carey.

00:26:43.840 --> 00:26:48.880
GOLDBERG: It’s very. [LAUGHS] Although what's 
troubling about it is that actually as in that  

00:26:48.880 --> 00:26:54.480
little vignette that we made up, she's using it 
off label, right. It's like she's just using it  

00:26:54.480 --> 00:27:02.080
because it helps the way doctors use Google. And 
I do find it troubling that what we don't have is  

00:27:02.080 --> 00:27:09.440
sort of institutional buy-in for everyone to 
do that because, shouldn't they if it helps?

00:27:09.440 --> 00:27:13.680
LEE: Yeah. Well, let's go ahead and 
get into Episode 2. So Episode 2,  

00:27:13.680 --> 00:27:21.200
we sort of framed as talking to two people 
who are on the frontlines of big companies  

00:27:21.200 --> 00:27:28.480
integrating generative AI into their clinical 
products. And so, one was Matt Lungren,  

00:27:28.480 --> 00:27:35.600
who's a colleague of mine here at Microsoft. And 
then Seth Hain, who leads all of R&D at Epic.

00:27:36.720 --> 00:27:40.960
Maybe we'll start with a little snippet of  

00:27:40.960 --> 00:27:45.023
something that Matt said that 
struck me in a certain way.

00:27:45.023 --> 00:27:48.480
MATTHEW LUNGREN: OK, we see this pain point. 
Doctors are typing on their computers while  

00:27:48.480 --> 00:27:53.360
they’re trying to talk to their patients, right? 
We should be able to figure out a way to get  

00:27:53.360 --> 00:27:58.640
that ambient conversation turned into text that 
then, you know, accelerates the doctor … takes  

00:27:58.640 --> 00:28:03.360
all the important information. That's a really 
hard problem, right. And so, for a long time,  

00:28:03.360 --> 00:28:07.040
there was a human-in-the-loop aspect to 
doing this because you needed a human to say,  

00:28:07.040 --> 00:28:11.160
“This transcript’s great, but here's actually what 
needs to go in the note.” And that can't scale.

00:28:11.160 --> 00:28:17.200
LEE: I think we expected healthcare systems to 
adopt AI, and we spent a lot of time in the book  

00:28:17.200 --> 00:28:24.000
on AI writing clinical encounter notes. It’s 
happening for real now, and in a big way. And  

00:28:24.000 --> 00:28:29.680
it’s something that has, of course, been happening 
before generative AI but now is exploding because  

00:28:29.680 --> 00:28:34.760
of it. Where are we at now, two years later, 
just based on what we heard from guests?

00:28:34.760 --> 00:28:42.080
KOHANE: Well, again, unless they're forced 
to, hospitals will not adopt new technology  

00:28:42.080 --> 00:28:50.000
unless it immediately translates into income. 
So it's bizarrely counter-cultural that, again,  

00:28:50.000 --> 00:28:57.440
they're not being able to bill for the use of the 
AI, but this technology is so compelling to the  

00:28:57.440 --> 00:29:08.240
doctors that despite everything, it's overtaking 
the traditional dictation-typing routine.

00:29:08.240 --> 00:29:08.720
LEE: Yeah.

00:29:08.720 --> 00:29:10.880
GOLDBERG: And a lot of them love it and say,  

00:29:10.880 --> 00:29:15.920
you will pry my cold dead hands off of my 
ambient note-taking, right. And I actually …  

00:29:15.920 --> 00:29:21.360
a primary care physician allowed me to watch 
her. She was actually testing the two main  

00:29:22.000 --> 00:29:27.760
platforms that are being used. And there 
was this incredibly talkative patient who  

00:29:27.760 --> 00:29:33.360
went on and on about vacation and all kinds 
of random things for about half an hour.

00:29:33.360 --> 00:29:40.400
And both of the platforms were incredibly good at 
pulling out what was actually medically relevant.  

00:29:40.400 --> 00:29:47.360
And so to say that it doesn't save time doesn't 
seem right to me. Like, it seemed like it actually  

00:29:47.360 --> 00:29:53.600
did and in fact was just shockingly good at 
being able to pull out relevant information.

00:29:53.600 --> 00:29:54.640
LEE: Yeah.

00:29:54.640 --> 00:30:00.960
KOHANE: I'm going to hypothesize that in the 
trials, which have in fact shown no gain in time,  

00:30:00.960 --> 00:30:05.920
is the doctors were being incredibly 
meticulous. [LAUGHTER] So I think …  

00:30:05.920 --> 00:30:12.720
this is a Hawthorne effect, because you know 
you're being monitored. And we've seen this  

00:30:13.520 --> 00:30:20.880
in other technologies where the moment 
the focus is off, it's used much more  

00:30:20.880 --> 00:30:26.080
routinely and with much less inspection, 
for the better and for the worse.

00:30:26.080 --> 00:30:32.000
LEE: Yeah, you know, within Microsoft, 
I had some internal disagreements about  

00:30:32.000 --> 00:30:40.080
Microsoft producing a product in this space. 
It wouldn't be Microsoft's normal way. Instead,  

00:30:40.080 --> 00:30:47.680
we would want 50 great companies building those 
products and doing it on our cloud instead of  

00:30:47.680 --> 00:30:54.720
us competing against those 50 companies. And one 
of the reasons is exactly what you both said. I  

00:30:54.720 --> 00:30:59.520
didn't expect that health systems would be willing 
to shell out the money to pay for these things. It  

00:30:59.520 --> 00:31:06.720
doesn't generate more revenue. But I think so 
far two years later, I've been proven wrong.

00:31:06.720 --> 00:31:14.000
I wanted to ask a question about values 
here. I had this experience where I had a  

00:31:14.000 --> 00:31:19.840
little growth, a bothersome growth on my cheek. 
And so had to go see a dermatologist. And the  

00:31:19.840 --> 00:31:28.560
dermatologist treated it, froze it off. But there 
was a human scribe writing the clinical note.

00:31:28.560 --> 00:31:37.440
And so I used the app to look at the note 
that was submitted. And the human scribe said  

00:31:38.000 --> 00:31:44.880
something that did not get discussed in 
the exam room, which was that the growth  

00:31:44.880 --> 00:31:51.760
was making it impossible for me to safely wear 
a COVID mask. And that was the reason for it.

00:31:51.760 --> 00:31:59.440
And that then got associated with a 
code that allowed full reimbursement  

00:31:59.440 --> 00:32:06.240
for that treatment. And so I think that's a 
classic example of what's called upcoding.  

00:32:07.280 --> 00:32:17.160
And I strongly suspect that AI scribes, 
an AI scribe would not have done that.

00:32:17.160 --> 00:32:21.080
GOLDBERG: Well, depending what values you 
programmed into it, right, Zak? [LAUGHS]

00:32:21.080 --> 00:32:27.600
KOHANE: Today, today, today, it will not do 
it. But, Peter, that is actually the central  

00:32:27.600 --> 00:32:35.600
issue that society has to have because our 
hospitals are currently mostly in the red.  

00:32:36.640 --> 00:32:46.960
And upcoding is standard operating procedure. 
And if these AI get in the way of upcoding,  

00:32:46.960 --> 00:32:54.080
they are going to be aligned towards that 
upcoding. You know, you have to ask yourself,  

00:32:54.080 --> 00:33:00.000
these MRI machines are incredibly useful. 
They're also big money makers. And if the  

00:33:00.000 --> 00:33:04.955
AI correctly says that for this complaint, 
you don't actually have to do the MRI …

00:33:04.955 --> 00:33:05.988
LEE: Right.

00:33:05.988 --> 00:33:10.720
KOHANE: … what's going to happen? And so I think 
this issue of values … you're right. Right now,  

00:33:10.720 --> 00:33:17.920
they're actually much more impartial. 
But there are going to be business plans  

00:33:17.920 --> 00:33:24.160
just around aligning these things 
towards healthcare. In many ways,  

00:33:24.160 --> 00:33:29.840
this is why I think we wrote the book so 
that there should be a public discussion.  

00:33:29.840 --> 00:33:34.920
And what kind of AI do we want to have? 
Whose values do we want it to represent?

00:33:34.920 --> 00:33:39.120
GOLDBERG: Yeah. And that raises 
another question for me. So,  

00:33:39.120 --> 00:33:44.720
Peter, speaking from inside the gigantic 
industry, like, there seems to be such a  

00:33:44.720 --> 00:33:50.000
need for self-surveillance of the models 
for potential harms that they could be  

00:33:50.000 --> 00:33:56.480
causing. Are the big AI makers doing that? 
Are they even thinking about doing that?

00:33:56.480 --> 00:34:00.720
Like, let's say you wanted to watch out for the 
kind of thing that Zak's talking about, could you?

00:34:00.720 --> 00:34:07.920
LEE: Well, I think evaluation, like the best 
evaluation we had when we wrote our book was, you  

00:34:07.920 --> 00:34:13.030
know, what score would this get on the step one 
and step two US medical licensing exams? [LAUGHS]

00:34:13.030 --> 00:34:14.795
GOLDBERG: Right, right, right, yeah.

00:34:14.795 --> 00:34:24.320
LEE: But honestly, evaluation hasn't gotten that 
much deeper in the last two years. And it's a big,  

00:34:24.320 --> 00:34:30.000
I think, it is a big issue. And it's related 
to the regulation issue also, I think.

00:34:30.000 --> 00:34:34.560
Now the other guest in Episode 2 is 
Seth Hain from Epic. You know, Zak,  

00:34:34.560 --> 00:34:39.600
I think it's safe to say that you're 
not a fan of Epic and the Epic system.  

00:34:39.600 --> 00:34:44.960
You know, we’ve had a few discussions 
about that, about the fact that doctors  

00:34:44.960 --> 00:34:50.000
don’t have a very pleasant experience 
when they’re using Epic all day.

00:34:50.000 --> 00:34:55.040
Seth, in the podcast, said that there 
are over 100 AI integrations going on  

00:34:55.040 --> 00:35:00.320
in Epic's system right now. Do you 
think, Zak, that that has a chance  

00:35:00.320 --> 00:35:05.600
to make you feel better about Epic? You 
know, what's your view now two years on?

00:35:05.600 --> 00:35:13.440
KOHANE: My view is, first of all, I want 
to separate my view of Epic and how it's  

00:35:13.440 --> 00:35:20.640
affected the conduct of healthcare 
and the quality of life of doctors  

00:35:20.640 --> 00:35:26.400
from the individuals. Like Seth Hain is 
a remarkably fine individual who I've  

00:35:26.400 --> 00:35:34.480
enjoyed chatting with and does really great 
stuff. Among the worst aspects of the Epic,  

00:35:34.480 --> 00:35:40.640
even though it's better in that respect 
than many EHRs, is horrible user interface.

00:35:40.640 --> 00:35:44.720
The number of clicks that you have to go to 
get to something. And you have to remember  

00:35:44.720 --> 00:35:50.240
where someone decided to put that thing. It 
seems to me that it is fully within the realm  

00:35:50.240 --> 00:35:58.240
of technical possibility today to actually 
give an agent a task that you want done in  

00:35:58.240 --> 00:36:03.040
the Epic record. And then whether Epic has 
implemented that agent or someone else,  

00:36:03.040 --> 00:36:10.000
it does it so you don't have to do the clicks. 
Because it's something really soul sucking that  

00:36:10.000 --> 00:36:15.040
when you're trying to help patients, you're having 
to remember not the right dose of the medication,  

00:36:15.040 --> 00:36:19.120
but where was that particular thing 
that you needed in that particular task?

00:36:21.360 --> 00:36:28.320
I can't imagine that Epic does not have that in 
its product line. And if not, I know there must  

00:36:28.320 --> 00:36:34.320
be other companies that essentially want to 
create that wrapper. So I do think, though,  

00:36:34.320 --> 00:36:43.920
that the danger of multiple integrations is 
that you still want to have the equivalent  

00:36:43.920 --> 00:36:52.240
of a single thought process that cares about 
the patient bringing those different processes  

00:36:52.240 --> 00:36:58.720
together. And I don't know if that's Epic's 
responsibility, the hospital's responsibility,  

00:36:58.720 --> 00:37:03.600
whether it's actually a patient agent. But 
someone needs to be also worrying about all  

00:37:03.600 --> 00:37:12.240
those AIs that are being integrated into the 
patient record. So … what do you think, Carey?

00:37:12.240 --> 00:37:19.280
GOLDBERG: What struck me most about what Seth said 
was his description of the Cosmos project, and I,  

00:37:19.280 --> 00:37:23.920
you know, I have been drinking Zak’s Kool-Aid 
for a very long time, [LAUGHTER] and he—no,  

00:37:23.920 --> 00:37:30.160
in a good way! And he persuaded me 
long ago that there is this horrible  

00:37:30.160 --> 00:37:34.160
waste happening in that we have all 
of these electronic medical records,  

00:37:34.160 --> 00:37:41.280
which could be used far, far more to learn from, 
and in particular, when you as a patient come in,  

00:37:41.280 --> 00:37:45.680
it would be ideal if your physician could 
call up all the other patients like you and  

00:37:45.680 --> 00:37:50.160
figure out what the optimal treatment for 
you would be. And it feels like—it sounds  

00:37:50.160 --> 00:37:56.720
like—that's one of the central aims that 
Epic is going for. And if they do that,  

00:37:56.720 --> 00:38:02.640
I think that will redeem a lot of the pain that 
they've caused physicians these last few years.

00:38:02.640 --> 00:38:07.280
And I also found myself thinking, you 
know, maybe this very painful period  

00:38:07.280 --> 00:38:12.080
of using electronic medical records was 
really just a growth phase. It was an  

00:38:12.080 --> 00:38:19.680
awkward growth phase. And once AI is fully 
used the way Zak is beginning to describe,  

00:38:19.680 --> 00:38:23.360
the whole system could start making 
a lot more sense for everyone.

00:38:23.360 --> 00:38:27.520
LEE: Yeah. One conversation I've 
had with Seth, in all of this is,  

00:38:27.520 --> 00:38:34.240
you know, with AI and its development, is there 
a future, a near future where we don't have an  

00:38:34.240 --> 00:38:38.960
EHR [electronic health record] system at all? 
You know, AI is just listening and just somehow  

00:38:38.960 --> 00:38:45.680
absorbing all the information. And, you know, one 
thing that Seth said, which I felt was prescient,  

00:38:45.680 --> 00:38:51.680
and I'd love to get your reaction, especially 
Zak, on this is he said, I think that … he said,  

00:38:51.680 --> 00:38:57.520
technically, it could happen, but the problem 
is right now, actually doctors do a lot of their  

00:38:57.520 --> 00:39:07.040
thinking when they write and review notes. You 
know, the actual process of being a doctor is not  

00:39:07.040 --> 00:39:13.400
just being with a patient, but it's actually 
thinking later. What do you make of that?

00:39:13.400 --> 00:39:21.760
KOHANE: So one of the most valuable experiences 
I had in training was something that's more  

00:39:21.760 --> 00:39:27.200
or less disappeared in medicine, which is the 
post-clinic conference, where all the doctors  

00:39:27.200 --> 00:39:34.640
come together and we go through the cases that 
we just saw that afternoon. And we, actually,  

00:39:34.640 --> 00:39:41.360
were trying to take potshots at each other 
[LAUGHTER] in order to actually improve. Oh,  

00:39:41.360 --> 00:39:47.120
did you actually do that? Oh, I forgot. I'm 
going to go call the patient and do that.

00:39:47.120 --> 00:39:57.120
And that really happened. And I think 
that, yes, doctors do think, and I do  

00:39:57.120 --> 00:40:03.600
think that we are insufficiently using yet the 
artificial intelligence currently in the ambient  

00:40:03.600 --> 00:40:11.920
dictation mode as much more of a independent 
agent saying, did you think about that?

00:40:11.920 --> 00:40:16.240
I think that would actually make 
it more interesting, challenging,  

00:40:16.240 --> 00:40:19.840
and clearly better for the patient 
because that conversation I just  

00:40:19.840 --> 00:40:22.880
told you about with the other 
doctors, that no longer exists.

00:40:22.880 --> 00:40:31.520
LEE: Yeah. Mm-hmm. I want to do one more thing 
here before we leave Matt and Seth in Episode 2,  

00:40:31.520 --> 00:40:37.280
which is something that Seth said with 
respect to how to reduce hallucination.

00:40:37.280 --> 00:40:41.280
SETH HAIN: At that time, there's a lot of 
conversation in the industry around something  

00:40:41.280 --> 00:40:49.760
called RAG, or retrieval-augmented generation. And 
the idea was, could you pull the relevant bits,  

00:40:49.760 --> 00:40:56.400
the relevant pieces of the chart, into that 
prompt, that information you shared with the  

00:40:56.400 --> 00:41:04.480
generative AI model, to be able to increase the 
usefulness of the draft that was being created?  

00:41:04.480 --> 00:41:09.520
And that approach ended up proving 
and continues to be to some degree,  

00:41:09.520 --> 00:41:15.760
although the techniques have greatly improved, 
somewhat brittle, right. And I think this  

00:41:15.760 --> 00:41:22.640
becomes one of the things that we are and will 
continue to improve upon because, as you get  

00:41:22.640 --> 00:41:27.440
a richer and richer amount of information into 
the model, it does a better job of responding.

00:41:27.440 --> 00:41:30.560
LEE: Yeah, so, Carey, this sort 
of gets at what you were saying,  

00:41:30.560 --> 00:41:38.080
you know, that shouldn't these models be 
just bringing in a lot more information  

00:41:38.080 --> 00:41:43.280
into their thought processes? And 
I'm certain when we wrote our book,  

00:41:43.280 --> 00:41:49.600
I had no idea. I did not conceive of RAG 
at all. It emerged a few months later.

00:41:49.600 --> 00:41:53.680
And to my mind, I remember the 
first time I encountered RAG—Oh,  

00:41:53.680 --> 00:41:57.440
this is going to solve all of our problems 
of hallucination. But it’s turned out to  

00:41:57.440 --> 00:42:02.200
be harder. It's improving day by day, 
but it’s turned out to be a lot harder.

00:42:02.200 --> 00:42:07.120
KOHANE: Seth makes a very deep point, 
which is the way RAG is implemented is  

00:42:07.120 --> 00:42:12.320
basically some sort of technique 
for pulling the right information  

00:42:12.320 --> 00:42:20.800
that's contextually relevant. And the way 
that's done is typically heuristic at best.  

00:42:20.800 --> 00:42:28.240
And it's not … doesn’t have the same depth 
of reasoning that the rest of the model has.

00:42:28.240 --> 00:42:35.280
And I'm just wondering, Peter, what you think, 
given the fact that now context lengths seem to  

00:42:35.280 --> 00:42:43.280
be approaching a million or more, and people 
are now therefore using the full strength of  

00:42:43.280 --> 00:42:50.480
the transformer on that context and are trying 
to figure out different techniques to make it  

00:42:50.480 --> 00:42:57.040
pay attention to the middle of the context. 
In fact, the RAG approach perhaps was just  

00:42:58.160 --> 00:43:04.160
a transient solution to the fact that 
it's going to be able to amazingly look  

00:43:04.160 --> 00:43:10.040
in a thoughtful way at the entire record of the 
patient, for example. What do you think, Peter?

00:43:10.040 --> 00:43:13.920
LEE: I think there are three 
things, you know, that are going on,  

00:43:13.920 --> 00:43:19.040
and I'm not sure how they're going to play 
out and how they're going to be balanced. And  

00:43:19.600 --> 00:43:23.600
I'm looking forward to talking to people 
in later episodes of this podcast,  

00:43:23.600 --> 00:43:28.560
you know, people like Sébastien Bubeck or Bill 
Gates about this, because, you know, there is  

00:43:28.560 --> 00:43:35.760
the pretraining phase, you know, when things are 
sort of compressed and baked into the base model.

00:43:35.760 --> 00:43:42.720
There is the in-context learning, you know, so 
if you have extremely long or infinite context,  

00:43:42.720 --> 00:43:47.040
you're kind of learning as you go 
along. And there are other techniques  

00:43:47.040 --> 00:43:50.640
that people are working on, you 
know, various sorts of dynamic  

00:43:51.520 --> 00:43:58.000
reinforcement learning approaches, and so on. 
And then there is what maybe you would call  

00:43:58.000 --> 00:44:05.440
structured RAG, where you do a pre-processing. 
You go through a big database, and you figure  

00:44:05.440 --> 00:44:14.480
it all out. And make a very nicely structured 
database the AI can then consult with later.

00:44:14.480 --> 00:44:20.080
And all three of these in different 
contexts today seem to show different  

00:44:20.080 --> 00:44:30.301
capabilities. But they're all 
pretty important in medicine.

00:44:30.301 --> 00:44:30.320
[TRANSITION MUSIC]

00:44:30.320 --> 00:44:33.920
Moving on to Episode 3, we 
talked to Dave DeBronkart,  

00:44:33.920 --> 00:44:39.760
who is also known as “e-Patient Dave,” an advocate 
of patient empowerment, and then also Christina  

00:44:39.760 --> 00:44:44.960
Farr, who has been doing a lot of venture 
investing for consumer health applications.

00:44:44.960 --> 00:44:48.960
Let's get right into this little snippet 
from something that e-Patient Dave said  

00:44:48.960 --> 00:44:52.240
that talks about the sources of 
medical information, particularly  

00:44:52.240 --> 00:44:56.863
relevant for when he was receiving 
treatment for stage 4 kidney cancer.

00:44:56.863 --> 00:45:01.120
DAVE DEBRONKART: And I'm making a point 
here of illustrating that I am anything but  

00:45:01.120 --> 00:45:07.680
medically trained, right. And yet I still, 
I want to understand as much as I can. I  

00:45:07.680 --> 00:45:17.200
was months away from dead when I was diagnosed, 
but in the patient community, I learned that they  

00:45:17.200 --> 00:45:25.600
had a whole bunch of information that didn't exist 
in the medical literature. Now today we understand  

00:45:25.600 --> 00:45:32.480
there's publication delays; there's all kinds of 
reasons. But there's also a whole bunch of things,  

00:45:32.480 --> 00:45:38.059
especially in an unusual condition, that will 
never rise to the level of deserving NIH [National  

00:45:38.059 --> 00:45:44.000
Institute of Health] funding and research.


LEE: All right. So I have a question for you,  

00:45:44.000 --> 00:45:47.760
Carey, and a question for you, Zak, about 
the whole conversation with e-Patient Dave,  

00:45:47.760 --> 00:45:53.120
which I thought was really remarkable. You know, 
Carey, I think as we were preparing for this  

00:45:53.120 --> 00:45:58.560
whole podcast series, you made a comment—I 
actually took it as a complaint—that not  

00:45:58.560 --> 00:46:05.360
as much has happened as I had hoped or thought. 
People aren't thinking boldly enough, you know,  

00:46:05.360 --> 00:46:12.880
and I think, you know, I agree with you in the 
sense that I think we expected a lot more to be  

00:46:12.880 --> 00:46:18.779
happening, particularly in the consumer space. 
I'm giving you a chance to vent about this.



00:46:18.779 --> 00:46:25.360
GOLDBERG: [LAUGHTER] Thank you! Yes, that has 
been by far the most frustrating thing to me.  

00:46:25.360 --> 00:46:34.240
I think that the potential for AI to improve 
everybody’s health is so enormous, and yet,  

00:46:34.240 --> 00:46:40.800
you know, it needs some sort of support to be able 
to get to the point where it can do that. Like,  

00:46:40.800 --> 00:46:46.000
remember in the book we wrote about Greg Moore 
talking about how half of the planet doesn't have  

00:46:46.000 --> 00:46:51.360
healthcare, but people overwhelmingly have 
cellphones. And so you could connect people  

00:46:51.360 --> 00:46:57.920
who have no healthcare to the world's medical 
knowledge, and that could certainly do some good.

00:46:58.720 --> 00:47:04.560
And I have one great big problem with 
e-Patient Dave, which is that, God,  

00:47:04.560 --> 00:47:10.000
he's fabulous. He's super smart. 
Like, he's not a typical patient.  

00:47:10.000 --> 00:47:16.560
He's an off-the-charts, brilliant patient. And 
so it's hard to … and so he's a great sort of  

00:47:16.560 --> 00:47:21.120
lead early-adopter-type person, and he 
can sort of show the way for others.

00:47:21.120 --> 00:47:27.520
But what I had hoped for was that there would 
be more visible efforts to really help patients  

00:47:27.520 --> 00:47:35.200
optimize their healthcare. Probably it's happening 
a lot in quiet ways like that any discharge  

00:47:35.200 --> 00:47:39.680
instructions can be instantly beautifully 
translated into a patient's native language  

00:47:39.680 --> 00:47:49.200
and so on. But it's almost like there isn't 
a mechanism to allow this sort of mass  

00:47:49.200 --> 00:47:56.480
consumer adoption that I would hope for.


LEE: Yeah. But you have written some, like,  

00:47:56.480 --> 00:48:03.200
you even wrote about that person who 
saved his dog. So do you think … you know,  

00:48:03.200 --> 00:48:08.160
and maybe a lot more of that is just happening 
quietly that we just never hear about?

00:48:08.160 --> 00:48:12.400
GOLDBERG: I'm sure that there is a lot 
of it happening quietly. And actually,  

00:48:12.400 --> 00:48:17.120
that's another one of my complaints is that no 
one is gathering that stuff. It's like you might  

00:48:17.120 --> 00:48:22.080
happen to see something on social media. 
Actually, e-Patient Dave has a hashtag,  

00:48:22.080 --> 00:48:26.560
PatientsUseAI, and a blog, as well. So 
he's trying to do it. But I don't know  

00:48:26.560 --> 00:48:31.920
of any sort of overarching or academic 
efforts to, again, to surveil what's the  

00:48:31.920 --> 00:48:36.800
actual use in the population and see what 
are the pros and cons of what's happening.

00:48:36.800 --> 00:48:43.680
LEE: Mm-hmm. So, Zak, you know, the thing that I 
thought about, especially with that snippet from  

00:48:43.680 --> 00:48:49.440
Dave, is your opening for Chapter 8 that 
you wrote, you know, about your first  

00:48:49.440 --> 00:48:59.360
patient dying in your arms. I still think of how 
traumatic that must have been. Because, you know,  

00:48:59.360 --> 00:49:04.480
in that opening, you just talked about all the 
little delays, all the little paper-cut delays,  

00:49:04.480 --> 00:49:11.840
in the whole process of getting some new medical 
technology approved. But there's another element  

00:49:11.840 --> 00:49:18.640
that Dave kind of speaks to, which is just, you 
know, patients who are experiencing some issue  

00:49:18.640 --> 00:49:24.400
are very, sometimes very motivated. And there's 
just a lot of stuff on social media that happens.

00:49:24.400 --> 00:49:36.080
KOHANE: So this is where I can both 
agree with Carey and also disagree.  

00:49:36.080 --> 00:49:43.120
I think when people have an actual health 
problem, they are now routinely using it.

00:49:43.120 --> 00:49:44.072
GOLDBERG: Yes, that's true.

00:49:44.072 --> 00:49:48.400
KOHANE: And that situation is happening 
more often because medicine is failing.  

00:49:48.400 --> 00:49:54.800
This is something that did not come up 
enough in our book. And perhaps that's  

00:49:54.800 --> 00:50:01.920
because medicine is actually feeling a lot more 
rickety today than it did even two years ago.

00:50:02.480 --> 00:50:07.440
We actually mentioned the problem. I think, 
Peter, you may have mentioned the problem with  

00:50:07.440 --> 00:50:12.160
the lack of primary care. But now in 
Boston, our biggest healthcare system,  

00:50:12.160 --> 00:50:17.600
all the practices for primary care are closed. 
I cannot get for my own faculty—residents  

00:50:17.600 --> 00:50:21.115
at MGH [Massachusetts General Hospital] 
can't get primary care doctor. And so …

00:50:21.115 --> 00:50:25.600
LEE: Which is just crazy. I mean, these are 
amongst the most privileged people in medicine,  

00:50:25.600 --> 00:50:29.480
and they can't find a primary 
care physician. That's incredible.

00:50:29.480 --> 00:50:33.440
KOHANE: Yeah, and so therefore 
… and I wrote an article about  

00:50:33.440 --> 00:50:38.560
this in the NEJM [New England Journal of 
Medicine] that medicine is in such dire  

00:50:38.560 --> 00:50:45.040
trouble that we have incredible 
technology, incredible cures,  

00:50:45.040 --> 00:50:52.560
but where the rubber hits the road, which 
is at primary care, we don't have very much.

00:50:52.560 --> 00:50:57.600
And so therefore, you see people who know 
that they have a six-month wait till they  

00:50:57.600 --> 00:51:03.920
see the doctor, and all they can do is say, 
“I have this rash. Here's a picture. What's  

00:51:03.920 --> 00:51:13.200
it likely to be? What can I do?” “I'm gaining 
weight. How do I do a ketogenic diet?” Or,  

00:51:13.200 --> 00:51:19.040
“How do I know that this is the flu?”

This is happening all the time, where acutely  

00:51:19.040 --> 00:51:25.360
patients have actually solved problems that 
doctors have not. Those are spectacular. But I'm  

00:51:25.360 --> 00:51:30.320
saying more routinely because of the failure of 
medicine. And it's not just in our fee-for-service  

00:51:30.320 --> 00:51:39.360
United States. It's in the UK; it's in 
France. These are first-world, developed-world  

00:51:39.360 --> 00:51:45.179
problems. And we don't even have to go to 
lower- and middle-income countries for that.

00:51:45.179 --> 00:51:45.190
LEE: Yeah.

00:51:45.190 --> 00:51:49.200
GOLDBERG: But I think it's important to note 
that, I mean, so you're talking about how even  

00:51:49.200 --> 00:51:54.480
the most elite people in medicine can't 
get the care they need. But there's also  

00:51:54.480 --> 00:51:58.400
the point that we have so much concern 
about equity in recent years. And it's  

00:51:58.400 --> 00:52:04.800
likeliest that what we're doing is exacerbating 
inequity because it's only the more connected,  

00:52:04.800 --> 00:52:08.360
you know, better off people who 
are using AI for their health.

00:52:08.360 --> 00:52:12.720
KOHANE: Oh, yes. I know what various 
Harvard professors are doing.  

00:52:12.720 --> 00:52:18.320
They're paying for a concierge 
doctor. And that's, you know,  

00:52:18.320 --> 00:52:23.120
a $5,000- to $10,000-a-year-minimum 
investment. That's inequity.

00:52:23.120 --> 00:52:29.520
LEE: When we wrote our book, you know, the 
idea that GPT-4 wasn't trained specifically  

00:52:29.520 --> 00:52:34.720
for medicine, and that was amazing, but it might 
get even better and maybe would be necessary to  

00:52:34.720 --> 00:52:42.320
do that. But one of the insights for me is that 
in the consumer space, the kinds of things that  

00:52:42.320 --> 00:52:48.880
people ask about are different than what 
the board-certified clinician would ask.

00:52:48.880 --> 00:52:54.400
KOHANE: Actually, that's, I just recently coined 
the term. It's the ... maybe it's ... well,  

00:52:54.400 --> 00:53:01.680
at least it's new to me. It's 
the technology or expert paradox.  

00:53:02.480 --> 00:53:12.000
And that is the more expert and narrow your 
medical discipline, the more trivial it is to  

00:53:12.000 --> 00:53:19.840
translate that into a specialized AI. So 
echocardiograms? We can now do beautiful  

00:53:19.840 --> 00:53:24.240
echocardiograms. That's really hard to do. I 
don't know how to interpret an echocardiogram.  

00:53:24.240 --> 00:53:29.920
But they can do it really, really well. Interpret 
an EEG [electroencephalogram]. Interpret a genomic  

00:53:29.920 --> 00:53:37.200
sequence. But understanding the fullness 
of the human condition, that's actually  

00:53:37.200 --> 00:53:44.320
hard. And actually, that's what primary care 
doctors do best. But the paradox is right now,  

00:53:44.320 --> 00:53:51.120
what is easiest for AI is also the most highly 
paid in medicine. [LAUGHTER] Whereas what is the  

00:53:51.120 --> 00:53:58.400
hardest for AI in medicine is the least 
regarded, least paid part of medicine.

00:53:58.400 --> 00:54:02.960
GOLDBERG: So this brings us to the question I 
wanted to throw at both of you actually, which  

00:54:02.960 --> 00:54:08.880
is we've had this spasm of incredibly prominent 
people predicting that in fact physicians would be  

00:54:08.880 --> 00:54:15.760
pretty obsolete within the next few years. We had 
Bill Gates saying that; we had Elon Musk saying  

00:54:15.760 --> 00:54:21.600
surgeons are going to be obsolete within a few 
years. And I think we had Demis Hassabis saying,  

00:54:21.600 --> 00:54:25.920
“Yeah, we'll probably cure most diseases 
within the next decade or so.” [LAUGHS]

00:54:25.920 --> 00:54:33.920
So what do you think? And also, Zak, to what 
you were just saying, I mean, you're talking  

00:54:33.920 --> 00:54:40.400
about being able to solve very general overarching 
problems. But in fact, these general overarching  

00:54:40.400 --> 00:54:48.080
models are actually able, I would think, are able 
to do that because they are broad. So what are we  

00:54:48.080 --> 00:54:53.280
heading towards do you think? What should the 
next book be ... The end of doctors? [LAUGHS]

00:54:53.280 --> 00:54:55.920
KOHANE: So I do recall a conversation  

00:54:56.640 --> 00:55:01.680
that … we were at a table with Bill Gates, 
and Bill Gates immediately went to this,  

00:55:01.680 --> 00:55:10.080
which is advancing the cutting edge of science. 
And I have to say that I think it will accelerate  

00:55:10.080 --> 00:55:16.960
discovery. But eliminating, let's say, cancer? 
I think that's going to be … that’s just super  

00:55:16.960 --> 00:55:21.680
hard. The reason it's super hard is we don't 
have the data or even the beginnings of the  

00:55:21.680 --> 00:55:28.240
understanding of all the ways this devilish 
disease managed to evolve around our solutions.

00:55:28.240 --> 00:55:32.720
And so that seems extremely hard. I think 
we'll make some progress accelerated by AI,  

00:55:32.720 --> 00:55:38.000
but solving it in a way Hassabis says, God 
bless him. I hope he's right. I'd love to  

00:55:38.000 --> 00:55:45.680
have to eat crow in 10 or 20 years, but I 
don't think so. I do believe that a surgeon  

00:55:45.680 --> 00:55:52.080
working on one of those Davinci machines, 
that stuff can be, I think, automated.

00:55:53.040 --> 00:55:58.480
And so I think that's one example of one of the 
paradoxes I described. And it won't be that we're  

00:55:58.480 --> 00:56:04.720
replacing doctors. I just think we're running out 
of doctors. I think it's really the case that,  

00:56:04.720 --> 00:56:09.280
as we said in the book, we're getting 
a huge deficit in primary care doctors.

00:56:09.280 --> 00:56:14.320
But even the subspecialties, my subspecialty, 
pediatric endocrinology, we're only filling  

00:56:14.320 --> 00:56:20.400
half of the available training slots every 
year. And why? Because it's a lot of work,  

00:56:20.400 --> 00:56:27.440
a lot of training, and frankly doesn't make as 
much money as some of the other professions.

00:56:27.440 --> 00:56:36.400
LEE: Yeah. Yeah, I tend to think that, 
you know, there are going to be always a  

00:56:36.400 --> 00:56:44.400
need for human doctors, not for their skills. In 
fact, I think their skills increasingly will be  

00:56:44.400 --> 00:56:51.520
replaced by machines. And in fact, I've talked 
about a flip. In fact, patients will demand,  

00:56:51.520 --> 00:56:56.160
Oh my god, you mean you're going to try to do 
that yourself instead of having the computer  

00:56:56.160 --> 00:57:01.520
do it? There's going to be that sort of flip. But 
I do think that when it comes to people's health,  

00:57:01.520 --> 00:57:11.120
people want the comfort of an authority figure 
that they trust. And so what is more of a  

00:57:11.120 --> 00:57:20.240
question for me is whether we will ever view a 
machine as an authority figure that we can trust.

00:57:20.240 --> 00:57:25.840
And before I move on to Episode 4, which is on 
norms, regulations and ethics, I’d like to hear  

00:57:25.840 --> 00:57:31.424
from Chrissy Farr on one more point on consumer 
health, specifically as it relates to pregnancy:

00:57:31.424 --> 00:57:35.920
CHRISTINA FARR: For a lot of women, it's 
their first experience with the hospital.  

00:57:35.920 --> 00:57:39.840
And, you know, I think it's a really 
big opportunity for these systems to  

00:57:40.640 --> 00:57:47.040
get a whole family on board and keep them kind 
of loyal. And a lot of that can come through,  

00:57:47.040 --> 00:57:52.720
you know, just delivering an incredible service. 
Unfortunately, I don't think that we are  

00:57:52.720 --> 00:58:00.000
delivering incredible services today to women in 
this country. I see so much room for improvement.

00:58:00.000 --> 00:58:08.640
LEE: In the consumer space, I don't think 
we really had a focus on those periods in  

00:58:08.640 --> 00:58:12.160
a person's life when they have a 
lot of engagement, like pregnancy,  

00:58:12.160 --> 00:58:19.760
or I think another one is menopause, cancer. 
You know, there are points where there is,  

00:58:19.760 --> 00:58:25.600
like, very intense engagement. And we 
heard that from e-Patient Dave, you know,  

00:58:25.600 --> 00:58:35.280
with his cancer and Chrissy with her pregnancy. 
Was that a miss in our book? What do think, Carey?

00:58:35.280 --> 00:58:43.200
GOLDBERG: I mean, I don't think so. I think 
it's true that there are many points in life  

00:58:43.200 --> 00:58:51.520
when people are highly engaged. To me, the 
problem thus far is just that I haven't seen  

00:58:51.520 --> 00:58:57.360
consumer-facing companies offering 
beautiful AI-based products. I think  

00:58:57.360 --> 00:59:04.000
there's no question at all that the market 
is there if you have the products to offer.

00:59:04.000 --> 00:59:07.440
LEE: So, what do you think this means, Zak, for,  

00:59:07.440 --> 00:59:13.520
you know, like Boston Children's or Mass 
General Brigham—you know, the big places?

00:59:13.520 --> 00:59:23.271
KOHANE: So again, all these large healthcare 
systems are in tough shape. MGB [Mass General  

00:59:23.271 --> 00:59:28.160
Brigham] would be fully in the red if not for 
the fact that its investments, of all things,  

00:59:28.160 --> 00:59:35.120
have actually produced. If you look at the large 
healthcare systems around the country, they are in  

00:59:35.120 --> 00:59:44.640
the red. And there's multiple reasons why they're 
in the red, but among them is cost of labor.

00:59:44.640 --> 00:59:49.280
And so we've created what used 
to be a very successful beast,  

00:59:49.280 --> 00:59:58.960
the health center. But it's developed a very 
expensive model and a highly regulated model.  

00:59:58.960 --> 01:00:07.280
And so when you have high revenue, tiny 
margins, your ability to disrupt yourself,  

01:00:07.280 --> 01:00:13.040
to innovate, is very, very low 
because you will have to talk to  

01:00:13.040 --> 01:00:21.120
the board next year if you went from 2% 
positive margin to 1% negative margin.

01:00:21.120 --> 01:00:21.840
LEE: Yeah.

01:00:21.840 --> 01:00:26.400
KOHANE: And so I think we're all waiting 
for one of the two things to happen,  

01:00:26.400 --> 01:00:31.840
either a new kind of healthcare 
delivery system being generated or  

01:00:31.840 --> 01:00:36.000
ultimately one of these systems 
learns how to disrupt itself.

01:00:36.000 --> 01:00:40.720
LEE: Yeah. All right. I think we 
have to move on to Episode 4. And,  

01:00:41.520 --> 01:00:46.000
you know, when it came to the question 
of regulation, I think this is … my read  

01:00:46.000 --> 01:00:49.680
is when we were writing our book, this is 
the part that we struggled with the most.

01:00:49.680 --> 01:00:52.880
GOLDBERG: We punted. [LAUGHS] 
We totally punted to the AI.

01:00:53.840 --> 01:00:57.920
LEE: We had three amazing guests. One 
was Laura Adams from National Academy  

01:00:57.920 --> 01:01:00.547
of Medicine. Let's play a snippet from her.

01:01:00.547 --> 01:01:05.280
LAURA ADAMS: I think one of the most provocative 
and exciting articles that I saw written recently  

01:01:05.280 --> 01:01:12.800
was by Bakul Patel and David Blumenthal, who 
posited, should we be regulating generative AI  

01:01:12.800 --> 01:01:19.600
as we do a licensed and qualified provider? Should 
it be treated in the sense that it's got to have  

01:01:19.600 --> 01:01:23.840
a certain amount of training and a foundation 
that's got to pass certain tests? Does it have  

01:01:23.840 --> 01:01:29.600
to report its performance? And I'm thinking, what 
a provocative idea, but it's worth considering.

01:01:29.600 --> 01:01:36.640
LEE: All right, so I very well remember that 
we had discussed this kind of idea when we  

01:01:36.640 --> 01:01:42.400
were writing our book. And I think before we 
finished our book, I personally rejected the  

01:01:42.400 --> 01:01:46.600
idea. But now two years later, what do 
the two of you think? I'm dying to hear.

01:01:46.600 --> 01:01:51.400
GOLDBERG: Well, wait, why … what do you think? 
Like, are you sorry that you rejected it?

01:01:51.400 --> 01:01:59.440
LEE: I'm still skeptical because when we are 
licensing human beings as doctors, you know,  

01:01:59.440 --> 01:02:05.120
we're making a lot of implicit assumptions that we 
don't test as part of their licensure, you know,  

01:02:05.120 --> 01:02:12.480
that first of all, they are [a] human being 
and they care about life, and that, you know,  

01:02:12.480 --> 01:02:18.320
they have a certain amount of common sense 
and shared understanding of the world.

01:02:18.320 --> 01:02:22.240
And there's all sorts of sort of implicit 
assumptions that we have about each other as  

01:02:22.240 --> 01:02:28.000
human beings living in a society together. 
That you know how to study, you know,  

01:02:28.000 --> 01:02:31.600
because I know you just went through three years 
of medical or four years of medical school and  

01:02:31.600 --> 01:02:37.280
all sorts of things. And so the standard 
ways that we license human beings, they  

01:02:37.280 --> 01:02:43.360
don't need to test all of that stuff. But somehow 
intuitively, all of that seems really important.

01:02:43.360 --> 01:02:46.400
I don't know. Am I wrong about that?

01:02:46.400 --> 01:02:56.640
KOHANE: So it's compared with what issue? Because 
we know for a fact that doctors who do a lot  

01:02:56.640 --> 01:03:02.080
of a procedure, like do this procedure, 
like high-risk deliveries all the time,  

01:03:02.080 --> 01:03:06.160
have better outcomes than ones who only 
do a few high risk. We talk about it,  

01:03:06.160 --> 01:03:11.680
but we don't actually make it explicit to 
patients or regulate that you have to have  

01:03:11.680 --> 01:03:22.080
this minimal amount. And it strikes me that 
in some sense, and, oh, very importantly,  

01:03:23.360 --> 01:03:29.520
these things called human beings learn on the 
job. And although I used to be very resentful  

01:03:29.520 --> 01:03:35.040
of it as a resident, when someone would say, 
I don't want the resident, I want the ...

01:03:35.040 --> 01:03:35.499
GOLDBERG: … the attending. [LAUGHTER]

01:03:35.499 --> 01:03:39.600
KOHANE: … they had a point. And so the 
truth is, maybe I was a wonderful resident,  

01:03:39.600 --> 01:03:46.960
but some people were not so great. [LAUGHTER] And 
so it might be the best outcome if we actually,  

01:03:46.960 --> 01:03:53.040
just like for human beings, we say, yeah, OK, it's 
this good, but don't let it work autonomously,  

01:03:53.040 --> 01:03:59.360
or it's done a thousand of them, just let it 
go. We just don't have practically speaking,  

01:03:59.360 --> 01:04:07.280
we don't have the environment, the lab, to test 
them. Now, maybe if they get embodied in robots  

01:04:07.280 --> 01:04:12.240
and literally go around with us, then it's going 
to be [in some sense] a lot easier. I don't know.

01:04:12.240 --> 01:04:12.960
LEE: Yeah.

01:04:12.960 --> 01:04:16.480
GOLDBERG: Yeah, I think I would take a step back 
and say, first of all, we weren't the only ones  

01:04:16.480 --> 01:04:21.920
who were stumped by regulating AI. Like, nobody 
has done it yet in the United States to this day,  

01:04:21.920 --> 01:04:29.120
right. Like, we do not have standing regulation 
of AI in medicine at all in fact. And that raises  

01:04:29.120 --> 01:04:37.040
the issue of … the story that you hear often 
in the biotech business, which is, you know,  

01:04:37.040 --> 01:04:41.760
more prominent here in Boston than anywhere 
else, is that thank goodness Cambridge put out,  

01:04:41.760 --> 01:04:45.760
the city of Cambridge, put out some 
regulations about biotech and how you  

01:04:45.760 --> 01:04:53.280
could dump your lab waste and so on. And that 
enabled the enormous growth of biotech here.

01:04:53.280 --> 01:05:00.480
If you don't have the regulations, then you can't 
have the growth of AI in medicine that is worthy  

01:05:00.480 --> 01:05:06.640
of having. And so, I just ... we're not the ones 
who should do it, but I just wish somebody would.

01:05:06.640 --> 01:05:07.440
LEE: Yeah.

01:05:07.440 --> 01:05:08.040
GOLDBERG: Zak.

01:05:08.040 --> 01:05:14.720
KOHANE: Yeah, but I want to say this as always, 
execution is everything, even in regulation.

01:05:14.720 --> 01:05:20.201
And so I'm mindful that a conference that both of 
you attended, the RAISE conference [Responsible  

01:05:20.201 --> 01:05:24.320
AI for Social and Ethical Healthcare]. The 
Europeans in that conference came to me personally  

01:05:24.320 --> 01:05:30.160
and thanked me for organizing this conference 
about safe and effective use of AI because they  

01:05:30.160 --> 01:05:40.160
said back home in Europe, all that we're talking 
about is risk, not opportunities to improve care.

01:05:40.160 --> 01:05:46.880
And so there is a version of regulation which 
just locks down the present and does not allow  

01:05:46.880 --> 01:05:52.880
the future that we're talking about to happen. 
And so, Carey, I absolutely hear you that we  

01:05:52.880 --> 01:06:01.440
need to have a regulation that takes away 
some of the uncertainty around liability,  

01:06:01.440 --> 01:06:08.640
around the freedom to operate that would allow 
things to progress. But we wrote in our book  

01:06:08.640 --> 01:06:18.640
that premature regulation might actually focus 
on the wrong thing. And so since I'm an optimist,  

01:06:18.640 --> 01:06:23.680
it may be the fact that we don't have 
much of a regulatory infrastructure today,  

01:06:23.680 --> 01:06:28.000
that it allows … it's a unique 
opportunity—I've said this now to  

01:06:28.000 --> 01:06:32.120
several leaders—for the healthcare systems 
to say, this is the regulation we need.

01:06:32.120 --> 01:06:32.880
GOLDBERG: It's true.

01:06:32.880 --> 01:06:36.640
KOHANE: And previously it was top-down. 
It was coming from the administration,  

01:06:36.640 --> 01:06:42.960
and those executive orders are now 
history. But there is an opportunity,  

01:06:42.960 --> 01:06:47.280
which may or may not be attained, there 
is an opportunity for the healthcare  

01:06:47.280 --> 01:06:51.740
leadership—for experts in surgery—to 
say, “This is what we should expect.”

01:06:51.740 --> 01:06:51.752
LEE: Yeah.

01:06:51.752 --> 01:06:56.066
KOHANE: I would love for this to happen. I 
haven't seen evidence that it’s happening yet.

01:06:56.066 --> 01:06:58.480
GOLDBERG: No, no. And there's 
this other huge issue,  

01:06:58.480 --> 01:07:02.080
which is that it's changing so fast. 
It's moving so fast. That something  

01:07:02.080 --> 01:07:06.469
that makes sense today won't in six 
months. So, what do you do about that?

01:07:06.469 --> 01:07:11.600
LEE: Yeah, yeah, that is something I feel proud 
of because when I went back and looked at our  

01:07:11.600 --> 01:07:16.640
chapter on this, you know, we did make that 
point, which I think has turned out to be true.

01:07:16.640 --> 01:07:20.960
But getting back to this conversation, 
there's something, a snippet of something,  

01:07:20.960 --> 01:07:24.800
that Vardit Ravitsky said that 
I think touches on this topic.

01:07:24.800 --> 01:07:31.600
VARDIT RAVITSKY: So my pushback is, are we seeing 
AI exceptionalism in the sense that if it's  

01:07:31.600 --> 01:07:37.280
AI, huh, panic! We have to inform everybody about 
everything, and we have to give them choices,  

01:07:37.280 --> 01:07:44.480
and they have to be able to reject that tool and 
the other tool versus, you know, the rate of human  

01:07:44.480 --> 01:07:52.960
error in medicine is awful. So why are we so 
focused on informed consent and empowerment  

01:07:52.960 --> 01:07:57.280
regarding implementation of AI 
and less in other contexts? 

01:07:57.280 --> 01:08:00.880
GOLDBERG: Totally agree. Who 
cares about informed consent  

01:08:00.880 --> 01:08:03.880
about AI. Don't want it. Don't need it. Nope.

01:08:03.880 --> 01:08:11.040
LEE: Wow. Yeah. You know, and this ... Vardit of 
course is one of the leading bioethicists, you  

01:08:11.040 --> 01:08:19.920
know, and of course prior to AI, she was really 
focused on genetics. But now it's all about AI.

01:08:19.920 --> 01:08:25.520
And, Zak, you know, you and other doctors 
have always told me, you know, the truth  

01:08:25.520 --> 01:08:31.920
of the matter is, you know, what do you call the 
bottom-of-the-class graduate of a medical school?

01:08:31.920 --> 01:08:33.818
And the answer is “doctor.”

01:08:33.818 --> 01:08:41.840
KOHANE: “Doctor.” Yeah. Yeah, I think that 
again, this gets to compared with what? We  

01:08:41.840 --> 01:08:50.320
have to compare AI not to the medicine we 
imagine we have, or we would like to have,  

01:08:50.320 --> 01:08:56.160
but to the medicine we have today. And 
if we're trying to remove inequity,  

01:08:56.160 --> 01:09:01.360
if we're trying to improve our health, 
that's what … those are the right metrics.  

01:09:01.360 --> 01:09:09.600
And so that can be done so long as we 
avoid catastrophic consequences of AI.

01:09:09.600 --> 01:09:18.000
So what would the catastrophic consequence of 
AI be? It would be a systematic behavior that  

01:09:18.000 --> 01:09:24.080
we were unaware of that was causing poor 
healthcare. So, for example, you know,  

01:09:24.080 --> 01:09:31.280
changing the dose on a medication, making it 
20% higher than normal so that the rate of  

01:09:31.280 --> 01:09:40.000
complications of that medication went from 1% to 
5%. And so we do need some sort of monitoring.

01:09:40.000 --> 01:09:43.920
We haven't put out the paper yet, 
but in computer science, there's,  

01:09:44.960 --> 01:09:51.840
well, in programming, we know very well the value 
for understanding how our computer systems work.

01:09:51.840 --> 01:09:57.120
And there was a guy by name of Allman, I 
think he's still at a company called Sendmail,  

01:09:57.120 --> 01:10:04.240
who created something called syslog. And 
syslog is basically a log of all the crap  

01:10:04.240 --> 01:10:10.960
that's happening in our operating system. And 
so I've been arguing now for the creation of  

01:10:10.960 --> 01:10:18.440
MedLog. And MedLog … in other words, what we 
cannot measure, we cannot regulate, actually.

01:10:18.440 --> 01:10:19.112
LEE: Yes.

01:10:19.112 --> 01:10:23.120
KOHANE: And so what we need to have is MedLog, 
which says, “Here's the context in which a  

01:10:23.120 --> 01:10:28.560
decision was made. Here's the version of the 
AI, you know, the exact version of the AI. Here  

01:10:28.560 --> 01:10:36.720
was the data.” And we just have MedLog. And I 
think MedLog is actually incredibly important  

01:10:36.720 --> 01:10:43.600
for being able to measure, to just do what we do 
in … it’s basically the black box for, you know,  

01:10:43.600 --> 01:10:48.480
when there's a crash. You know, we'd like to think 
we could do better than crash. We can say, “Oh,  

01:10:48.480 --> 01:10:54.160
we're seeing from MedLog that this practice 
is turning a little weird.” But worst case,  

01:10:54.160 --> 01:10:57.920
patient dies, [we] can see in MedLog, what 
was the information this thing knew about  

01:10:57.920 --> 01:11:03.120
it? And did it make the right decision? 
We can actually go for transparency,  

01:11:03.120 --> 01:11:07.331
which like in aviation, is much 
greater than in most human endeavors.

01:11:07.331 --> 01:11:07.355
GOLDBERG: Sounds great.

01:11:07.355 --> 01:11:12.800
LEE: Yeah, it's sort of like a black box. I was 
thinking of the aviation black box kind of idea.  

01:11:12.800 --> 01:11:17.200
You know, you bring up medication errors, 
and I have one more snippet. This is from  

01:11:17.200 --> 01:11:23.920
our guest Roxana Daneshjou from Stanford.


ROXANA DANESHJOU: There was a mistake in her  

01:11:23.920 --> 01:11:30.640
after-visit summary about how much Tylenol she 
could take. But I, as a physician, knew that this  

01:11:30.640 --> 01:11:37.120
dose was a mistake. I actually asked ChatGPT. I 
gave it the whole after-visit summary, and I said,  

01:11:37.120 --> 01:11:42.720
are there any mistakes here? And it clued in 
that the dose of the medication was wrong.

01:11:42.720 --> 01:11:46.640
LEE: Yeah, so this is something we 
did write about in the book. We made  

01:11:46.640 --> 01:11:51.680
a prediction that AI might be a second 
set of eyes, I think is the way we put it,  

01:11:51.680 --> 01:11:57.360
catching things. And we actually had 
examples specifically in medication dose  

01:11:57.360 --> 01:12:01.680
errors. I think for me, I expected to 
see a lot more of that than we are.

01:12:01.680 --> 01:12:07.680
KOHANE: Yeah, it goes back to our 
conversation about Epic or competitor Epic  

01:12:07.680 --> 01:12:14.960
doing that. I think we're going to see that 
having oversight over all medical orders,  

01:12:14.960 --> 01:12:23.120
all orders in the system, critique, real-time 
critique, where we're both aware of alert  

01:12:23.120 --> 01:12:28.400
fatigue. So we don't want to have too many false 
positives. At the same time, knowing what are  

01:12:28.400 --> 01:12:35.040
critical errors which could immediately affect 
lives. I think that is going to become in terms  

01:12:35.040 --> 01:12:42.560
of—and driven by quality measures—a product.


GOLDBERG: And I think word will spread among the  

01:12:42.560 --> 01:12:49.280
general public that kind of the same way in a 
lot of countries when someone's in a hospital,  

01:12:49.280 --> 01:12:52.789
the first thing people ask relatives 
are, well, who's with them? Right?

01:12:52.789 --> 01:12:52.960
LEE: Yeah. Yup.

01:12:52.960 --> 01:12:56.320
GOLDBERG: You wouldn't leave someone 
in hospital without relatives. Well,  

01:12:56.320 --> 01:12:58.240
you wouldn't maybe leave your medical ...

01:12:58.240 --> 01:13:00.240
KOHANE: By the way, that country 
is called the United States.

01:13:00.240 --> 01:13:02.640
GOLDBERG: Yes, that's true. 
[LAUGHS] It is true here now,  

01:13:02.640 --> 01:13:08.480
too. But similarly, I would tell 
any loved one that they would be  

01:13:08.480 --> 01:13:15.269
well advised to keep using AI to check 
on their medical care, right. Why not?

01:13:15.269 --> 01:13:21.280
LEE: Yeah. Yeah. Last topic, just for this Episode 
4. Roxana, of course, I think really made a name  

01:13:21.280 --> 01:13:27.440
for herself in the AI era writing, actually just 
prior to ChatGPT, you know, writing some famous  

01:13:27.440 --> 01:13:34.160
papers about how computer vision systems for 
dermatology were biased against dark-skinned  

01:13:34.160 --> 01:13:44.960
people. And we did talk some about bias in these 
AI systems, but I feel like we underplayed it,  

01:13:44.960 --> 01:13:50.240
or we didn't understand the magnitude of the 
potential issues. What are your thoughts?

01:13:50.240 --> 01:13:57.920
KOHANE: OK, I want to push back, because I've 
been asked this question several times. And  

01:13:57.920 --> 01:14:04.880
so I have two comments. One is, over 
100,000 doctors practicing medicine,  

01:14:04.880 --> 01:14:09.920
I know they have biases. Some of them 
actually may be all in the same direction,  

01:14:09.920 --> 01:14:15.920
and not good. But I have no way of actually 
measuring that. With AI, I know exactly how  

01:14:15.920 --> 01:14:24.240
to measure that at scale and affordably. Number 
one. Number two, same 100,000 doctors. Let's say  

01:14:24.240 --> 01:14:31.515
I do know what their biases are. How hard is it 
for me to change that bias? It's impossible …

01:14:31.515 --> 01:14:32.468
LEE: Yeah, yeah.

01:14:32.468 --> 01:14:39.280
KOHANE: … practically speaking. Can I change the 
bias in the AI? Somewhat. Maybe some completely.

01:14:39.280 --> 01:14:41.880
I think that we're in a much better situation.

01:14:41.880 --> 01:14:42.960
GOLDBERG: Agree.

01:14:42.960 --> 01:14:48.000
LEE: I think Roxana made also the super 
interesting point that there's bias in  

01:14:48.000 --> 01:14:54.137
the whole system, not just in individuals, but, 
you know, there's structural bias, so to speak.

01:14:54.137 --> 01:14:54.155
KOHANE: There is.

01:14:54.155 --> 01:14:59.600
LEE: Yeah. Hmm. There was a super interesting 
paper that Roxana wrote not too long ago—  

01:15:00.160 --> 01:15:05.280
her and her collaborators—showing 
AI's ability to detect, to spot  

01:15:05.280 --> 01:15:10.360
bias decision-making by others. 
Are we going to see more of that?

01:15:10.360 --> 01:15:13.360
KOHANE: Oh, yeah, I was very pleased when,  

01:15:14.240 --> 01:15:17.680
in NEJM AI [New England Journal of Medicine 
Artificial Intelligence], we published a piece  

01:15:17.680 --> 01:15:27.920
with Marzyeh Ghassemi, and what they were talking 
about was actually—and these are researchers who  

01:15:27.920 --> 01:15:34.320
had published extensively on bias and threats 
from AI. And they actually, in this article,  

01:15:34.320 --> 01:15:40.080
did the flip side, which is how much better 
AI can do than human beings in this respect.

01:15:40.080 --> 01:15:47.280
And so I think that as some of these computer 
scientists enter the world of medicine,  

01:15:47.280 --> 01:15:56.000
they're becoming more and more aware of 
human foibles and can see how these systems,  

01:15:56.000 --> 01:16:01.920
which if they only looked at the pretrained state,  

01:16:01.920 --> 01:16:08.800
would have biases. But now, where we know how 
to fine-tune the de-bias in a variety of ways,  

01:16:08.800 --> 01:16:13.840
they can do a lot better and, in fact, 
I think are much more … a much greater  

01:16:13.840 --> 01:16:20.720
reason for optimism that we can change some of 
these noxious biases than in the pre-AI era.

01:16:20.720 --> 01:16:25.760
GOLDBERG: And thinking about 
Roxana's dermatological work on how  

01:16:26.640 --> 01:16:35.200
I think there wasn't sufficient work on skin 
tone as related to various growths, you know,  

01:16:35.200 --> 01:16:42.800
I think that one thing that we totally missed in 
the book was the dawn of multimodal uses, right.

01:16:42.800 --> 01:16:43.830
LEE: Yeah. Yeah, yeah.

01:16:43.830 --> 01:16:47.120
GOLDBERG: That's been truly amazing 
that in fact all of these visual  

01:16:47.120 --> 01:16:53.080
and other sorts of data can be entered 
into the models and move them forward.

01:16:53.080 --> 01:17:00.560
LEE: Yeah. Well, maybe on these slightly more 
optimistic notes, we're at time. You know,  

01:17:00.560 --> 01:17:06.960
I think ultimately, I feel pretty good 
still about what we did in our book,  

01:17:06.960 --> 01:17:10.240
although there were a lot of misses. 
[LAUGHS] I don't think any of us could  

01:17:10.240 --> 01:17:14.621
really have predicted really the 
extent of change in the world.

01:17:14.621 --> 01:17:18.880
[TRANSITION MUSIC]
So, Carey, Zak, just so much fun to

01:17:18.880 --> 01:17:23.680
do some reminiscing but also some 
reflection about what we did.

01:17:23.680 --> 01:17:31.120
[THEME MUSIC]

01:17:31.120 --> 01:17:35.360
And to our listeners, as always, thank you 
for joining us. We have some really great  

01:17:35.360 --> 01:17:39.440
guests lined up for the rest of the series, and 
they’ll help us explore a variety of relevant  

01:17:39.440 --> 01:17:44.880
topics—from AI drug discovery to what medical 
students are seeing and doing with AI and more.

01:17:44.880 --> 01:17:47.600
We hope you’ll continue to tune 
in. And if you want to catch up  

01:17:47.600 --> 01:17:50.480
on any episodes you might have 
missed, you can find them at  

01:17:50.480 --> 01:17:57.520
aka.ms/AIrevolutionPodcast or wherever 
you listen to your favorite podcasts.  

01:17:57.520 --> 01:18:05.760
Until next time. 

01:18:05.760 --> 01:18:06.728
[MUSIC FADES]