Dr Nero: Good morning and welcome to Future Pulse. I'm Dr. Tom Nero. Today, I have the pleasure of speaking to Dr. Michael Gibson. , Dr. Gibson is the CEO of the nonprofit Bain and Perfuse Research Institutes. He's a professor at Harvard Medical School and also a cardiologist and interventional cardiologist at the B.
I. Deaconess Hospital. He's also been the study chair on numerous clinical trials. , I really appreciate you spending the time with us again. and today, , we're going to be doing a little bit of a deep dive into medical statistics and trial design, maybe a bit on AI if we have the time.
And obviously these topics are really deep, so it could be multi day seminars, but we're going to try to get them done today.
[00:00:39] Dr Gibson: Great. Thanks for having me back, Tom.
[00:00:42] Dr Nero: So I'm reminded that, uh, when we're talking about statistics, my wife brought up a farsight cartoon that had the picture of hell's library and all the books are titled statistics. Uh, so, you know, I know this is one of those topics where it's so important for us to understand on the flip side, it's so boring to talk about because that's not what is important to us on a daily basis. , we want to take care of people. We don't want to be worried about the statistics.
[00:01:07] Dr Gibson: True. And we, you know, we have to make yes or no decisions, but we're dealing with a world where it's not necessarily binary. It's not black and white. It's a gray kind of bell curve. And You know, where you set that threshold for decision making is frankly hard to believe, but it's pretty arbitrary.
[00:01:29] Dr Nero: I'm reminded about something that I can't remember who had said it on one of the panels, essentially only 20 percent of the knowledge that we utilize on a daily basis is based on clinical trials.
[00:01:40] Dr Gibson: Absolutely, you know, the percent is probably 15 to 20 percent are based upon class one recommendations where we have adequate data to really make an informed choice. The rest is based upon data that may not be so good, observational data. Um, we really got led astray, for instance, in COVID where there were therapies that looked like they would work based on observational data, but did not work when you held them up to the scrutiny of a randomized trial.
In fact, most of the therapies that were proposed, , failed, , in a randomized trials. So it's really important even in the middle of an epidemic or pandemic to have the machinery in place to answer the questions. In ways that are methodologically sound.
[00:02:31] Dr Nero: , and, , going back to the guidelines, as you said, we, you know, the limitation of guidelines. And I'm always reminded that guidelines are guidelines. They're not absolutes
[00:02:40] Dr Gibson: well, guidelines are intended generally for populations. And what a lot of practicing docs say is correct. My patients don't fit into the guidelines. And , often the guidelines are based upon randomized trials, which are a select part of a population, not the entire population. So, in the future, what I think you'll see more and more of, we'll get to AI in a bit, but I think you will see more and more.
Efforts and we're making these efforts to take large data sets and individualize recommendations for individual patients based upon age, weight, kidney function, you know, all the demographics so that we truly are having shared decision making with patients. With information that says, you know, if you do this, if you lower your blood pressure this much, here's the benefit you can expect, , you, you decide how much benefit you want with how much risk you want and displaying that , in a way that is, helpful to patients.
Sometimes it's a little too complicated, but getting it down to, numbers of bodies and. , ways that people who aren't necessarily numerically, , literate is an important way of communicating. Here's what you can expect and here's the risk.
[00:04:02] Dr Nero: It will be interesting when we talk about, , sub analyses and secondary, analyses and especially with AI, which is essentially right now all going to be retrospective data is that all those things are biased. And we also have to recognize ahead of time. , that there are limitations, , but before we get to limitations, . We're going to first talk about, the standard models that we use , the composite endpoint models, Cox regression analysis, Kepler Meier curves, and things like that. What do you see as the benefits of using those statistical analysis for trials? .
[00:04:32] Dr Gibson: I, I think there are definite benefits. Um, those models have been successful for many, many years. Uh, there is though something that people aren't aware of. They assume that the benefit of a drug over placebo is constant. And when you look at the event curves, you often see that it's not constant, , for acute coronary syndromes, we often see, , very big benefits early on.
And then the relative risk reduction is actually narrows, uh, the absolute risk reduction is constant. The curves are parallel, but your relative risk reduction kind of is getting smaller and smaller. So, um, so the constancy assumption is often not met. And we often don't deal with that in traditional KM methods.
There are newer methods where you use what are called time varying covariates to adjust for the fact that the risk is changing over time. And I think that's very important. What we're doing as doctors is we have someone who comes into our office and we say, you know what, Mrs. Jones, you've done okay on that, that new anticoagulant.
You're not bleeding. So, I'm going to continue it. We are iteratively assessing risk over time. We're doing kind of multiple landmark analyses and, you know, the statistician cringes and says, Oh my God, but we just looked from the beginning, you know, and didn't do landmark analyses because you lose randomization if you look at Mrs.
Jones at three months. On the other hand, what the statisticians don't realize is Mrs. Jones is a different person at three months. She's, she passed her bleeding stress test. And her risk is now different than it was at the beginning. Uh, and intuitively, as docs, that's what we're doing. What I've been trying to do is get us as statisticians and AI people and mathematicians, is to get in sync with what we do as docs.
Let's reiteratively assess risk over time, based upon someone has, how someone has done today. One thing people don't realize, Tom, is after a year of Clopidogrel, The risk of bleeding on clopidogrel grow versus placebo is really no different. So once you pass your bleeding stress test, it's kind of all benefit from there on out, but we don't account for that in, in, in the way we interpret trial.
[00:07:03] Dr Nero: Yeah, there was a great trial that was recently released about, uh, switching over patients who are elderly from Warfarin over to DOACs. And they did worse than duax because all those patients had already passed their bleeding stress test. You know, you didn't take them at the beginning and say, okay, we're going to choose between these two. You took at the beginning of patients who were stable on one. And you know, that brings up that exact same point, . , these people have histories, they have background, they don't come in, , and yes, we randomize them into the trial, but once they're in the trial, and like you said, over time, um, they do have, they
[00:07:34] Dr Gibson: Yeah.
[00:07:35] Dr Nero: The other part about the, the regular CAM curves that we talk about is that they are really, a lot of them are based on time to first event, right? They're non hierarchical, right? You're going to, once they have an event, hospitalization, death, et cetera, um, things stop for them, right? You get that, you get that number and it's gone and you lose them essentially from, from the rest of the trial. And I think that that's also going to be one of those pieces where, you know, as we're trying to get these trials off the ground and get information for us. Um, that we're a bit limited because of that. And you know, obviously people care about death. They care about stroke. They might care about stroke more than death. Um, and I think that that's, you know, we're going to get into that until we talked about when ratio stuff. Um, but that's, again, one of those things that's a bit lost. Um, and we've been utilizing all of those things a lot as we'd say, okay, this, you know, this benefit, you know, this drug with this composite endpoint, mace plus whatever, um, is beneficial. But it's. All those things together for first events and then that's it.
[00:08:36] Dr Gibson: Yeah. Yeah. My anecdata, the data from my trials seems to indicate that that multiple event analysis or time to any event rather than time to first event analysis in my studies, my, uh, practice has been that they're more statistically powerful. The consensus among everyone is they're probably about the same power.
I've had better luck with it. On the other hand, putting statistical power aside and our ability to see a difference, , if you're a patient, you kind of are interested in the burden of all of this, , as a patient, you're really interested in two things. Am I alive? And I am I out of the hospital and, you know, that's an end point that I've been as advocating for is number of days alive and out of the hospital that takes into account all of these things, like whether it's a stroke or a, um, am I, or bleeding, you know, so in our studies, we're looking at any SAE that resulted in a hospitalization, a cardiovascular SAE that resulted in a hospitalization, Counts in that kind of analysis, are you alive and are you, are you out of the hospital, uh, free from any cardiovascular kind of event?
So from a patient perspective, I think that is a more patient centric kind of end point. That's what patients want. I agree with you, Tom, that when you look at, uh, what are called utility scores, how do people perceive. You know, all these end points as compared to death. If you put death as a one, people are surprised.
Patients will say a disabling stroke is worse than death. Uh, they'll put it up at about a 1. 1. They'll put 7, uh, and they will put, you know, bleeding down in that 0. 4, 0. 3 range. We treat all these end points as equal and they're really not equal. And this is something that's being revisited now. How can we build quantitative.
Kind of end points that fit into what you called the hierarchy of how do people value them? So I agree with you. I think it is a gross Approximation to what patients really want to know and I agree with you If you when let me know when you want to get into it when ratios and how they they get around some of these issues
[00:11:02] Dr Nero: Yeah, the, uh, the issue of game theory is you're sort of alluding to and sort of, and there've been a lot of discussion in the behavioral economics world about these kinds of things and that we really make decisions that are, that are value based and that are, uh, that are have a ratio against what we think is of a higher or lower value. We're afraid of losing money a lot more than we're favor of making, for example, even though those two things should be equal. Right? And so similar things, you know, no people, you know, my patients may be very concerned about, you know, obviously death and heart attack. They may not necessarily be quite as worried about bleeding events.
And the way we look at bleeding events includes a whole bunch of things
[00:11:43] Dr Gibson: Yeah, you know, I was asked to write an editorial for JACC recently, and I'm opposed to this blending of bleeding events and ischemic events together. Patients are, may have different perspectives on bleeding versus ischemia. Um, and if you lump them together. It's obscuring the differences, you know, in the directions and the magnitudes.
Uh, so I think most clinicians out there would agree, patients want to know, here's the upside and here's the downside. They don't want a net. They want to know the upside and the downside. They also don't want and should not receive guidance using relative risk. Uh, people really need to know the absolute risk.
And when you look at guidelines related to informing patients, it says use absolute risk. And if you have a numeric patients, put up bodies, you know, body counts. The problem with relative risk is You could say, well, it's going to cut your risk of, uh, stroke in half. Well, if the risk of stroke is 0. 2 percent and you're cutting it down to 0.
1%, you know, you're talking about vanishingly small numbers of events, uh, and it doesn't really convey that when you talk to a patient. So my practice is use absolute risk and separate it for ischemic or upside benefits and risk like bleeding.
[00:13:14] Dr Nero: , one of the other good things about the, and when we talk about these, uh, KM curves , and this analysis. Is that you are able to use net benefit and if, , a net benefit bill correctly is going to be a very appropriate way of looking at that because, you know, they don't stroke as a negative of a hemorrhagic stroke is something that is clearly, , taking away from their benefit from whatever else potentially that they're going to do of the heart attack that's avoided.
[00:13:42] Dr Gibson: . I was very critical in this editorial of lumping hemorrhagic and ischemic stroke together, , in AFib trials, because this ischemic stroke, we're trying to prevent it, right? The drugs we're using to prevent it are causing hemorrhagic strokes that are worse in terms of mortality and disability.
Then ischemic stroke. ,
[00:14:03] Dr Nero: and importantly is that sometimes the embolic event causes a
secondary hemorrhagic stroke. And so you can't really assess that as being clearly just a net negative.
, it's quite complex
[00:14:14] Dr Gibson: very complex, but what's happening right now is they're lumping them together. And as you have drugs that are more potent, uh, they may reduce some of the ischemic strokes. It's good, but they may cause obviously more hemorrhagic strokes. And on the net, it looks the same, , because they're cancelling each other out.
On the other hand, if you're causing more hemorrhagic strokes, you could have more disability. The other problem with lumping them, two of them together is if you're doing non inferiority studies. You're always driving yourself to the null, you're always driving yourself to no difference with new regimens, and it can hide differences between the strategies.
So for instance, you could use like low dose Edoxaban, which had more ischemic strokes than the high dose. But had much fewer fatal bleeds, much for your GI bleeds and actually had improved survival where the high dose didn't. So if you just focus on lumping together ischemic stroke and hemorrhagic stroke, you can miss the GI bleeds, you can miss the fatalities, , and certainly when the FDA reviewed the data, say for a drug like Edoxaban, they separated out ischemic and hemorrhagic strokes and looked at all of the different endpoints.
Separately, but the funny thing is they reached the conclusion that, yeah, we're going to give a labeling to the high dose, even though the low dose reduced mortality and reduced GI bleeding, you know, so it just points to what you said. Time used the word. It's very complex. It is very complex.
[00:15:48] Dr Nero: I also wanted to just circle back just for a second back to your, uh, comment about using SAEs and SAEs that cause hospitalizations. And there's some really nice data that's come out saying that any hospitalization for any of these things increases their mortality over time. And , you may not necessarily see that mortality within the trial, but you will see that mortality within that patient. , those things really are much greater than we think , and it's worthy, but it's different than the event that is somewhat similar that doesn't bring them into the hospital.
[00:16:22] Dr Gibson: Yeah, I agree. , a perfect example is like, um, a non fatal bleed. You know, some people have said a non fatal bleed is just a bad day. I, I agree a little bit with that, but on the other hand, if a non fatal bleed in an elderly woman, if she says, look, I don't know if it's the pink pill, the green pill or the blue pill or the white pill, I'm bleeding and I'm going to stop all four of these pills.
And what person does is they stop all the evidence based medicine. So these SAEs can trigger, particularly if they're enough to be hospitalized, they can trigger someone to stop and get off the. off the evidence based medicines that they're on. So it's a hidden kind of hazard.
, one of the nice things about those older models that we've been using for many, many years is that. , we understand them, , we understand what a P value is. , we can compare them.
[00:17:14] Dr Nero: Most of us understand the benefits of using an absolute risk reduction, I hope. Now we're going into , wind ratios and new ways of doing these analyses. And the problem with the older models is that you have to see a lot of patients to ensure that you're getting that randomization. Creates very large trials, you're waiting for events, they are time to first event driven, and they get stopped when they, once they've reached that threshold because we want to get those drugs out. , but we're losing statistical power with that. , so there's some, , trials that you've been, , working on talking about wind ratio.
So if you don't mind us sort of describing that a little bit
[00:17:49] Dr Gibson: Sure. Let me back the truck up a little bit, , and talk a little bit about the P of 0. 05. Uh, we're having a lot of discussion among trialists and statisticians about, you know, should we continue to worship at that altar? A hundred years ago, one day, this guy Fisher woke up and decided, , 5 percent was the number.
, and, , people view a P of 0. 05 , like it's a universal constant. I said earlier that people don't realize just how kind of arbitrary some of these things are. The other thing is, , if we don't, think there's any kind of inferior kind of outcome to this new drug, why are we looking at both tails of the distribution?
Why aren't we just looking at the, , one tail and using a PA 0. 05 , on the superior tail? So some of us have moved more and more in our interpretation away from just worshiping at the altar of 0. 05 to looking at the confidence intervals. So say you had a drug that had a 10 percent benefit.
Well, there's a confidence interval around it, and that confidence interval could, it could be 21 percent better, or it could be 1 percent worse. You know, now because it could be just that 1 percent worse, the hazard ratio is 1. 01 or 1. 005. People can could see it didn't work well, you know, I'm not quite sure that particularly given the a priori data that, you know, the kind of pre test probability that you bring to that question, it probably is better and we're, we're probably losing some drugs at that nominal statistical significance.
Yeah, I'm a have been a cardio renal panel member. I did approve entresto. for HFpEF. With a P value of 0. 06, , there was compelling data from other trials and animal data, et cetera, that, that made me want to do that. Let's now talk about win ratios. We rehearsed all of the potential.
downsides of some of the traditional statistics. One of which you pointed out is like an MI is treated the same as a death. Uh, it doesn't treat the data in any hierarchical manner. It only looks at what happened first. Well, the way the win ratio works is it gets around that. It puts a hierarchy to all these endpoints.
So death is number one. Uh, you could put stroke or MI is number two and then stent thrombosis or something like that is number four. And then you could have. An outcome like, , infarct size as a continuous variable, so the way you win ratio works, it's fascinating. , it's good because you test for death first. That's the most important end point. That's great. Now, if we had a hundred patients in a trial, patient number one, And the treated arm would , compete against all 100 patients in the placebo arm.
And you'd say, did they win on death against this patient or not? And, um, you would count up the numbers of wins and losses. Okay. You would hope you would win more often than you would lose, right? Now, on the other hand, if it was a tie, say neither of them had death. Then they progress to the next endpoint, say that stroke, and then that patient one competes against all 100 patients on stroke.
If neither of them had a stroke, then you progress to test MI, and you go on and on and on. Then finally at the bottom. At the bottom of the testing, everyone had an infarct size measured. And you then compete on the size of the infarct size. So there aren't going to be any ties. There's going to be a winner and a loser.
And you see how often you win on something like infarct size. Well, the practical outcome of this in, say, STEMI trials, is we've been able to get the sample size down from 8 to 10, 000 patients for death in a my two down to 2, 500 for patients with death and my stroke and, , stent thrombosis and infarct size, the peak troponin.
And, , that has obviously massive implications for trial conduct. You can get the trials done more quickly, more efficiently. , at the end of the day, though, then the question becomes. , what does it mean to have won, , 15 percent more often in the end point? It's not necessarily as intuitive as some of the numbers we have, like the event rate, right?
The other question becomes like, well, okay, so there was really no difference in death and my stroke, how often you won, but it was all driven by the infarct size, the last thing you tested. , and that's not great either, because, , you're really not seeing anything going on in the harder endpoints.
So when you present a win ratio, and this hasn't been done well in the past with some of the studies, it's important to show for every single endpoint, like death, stroke, MI, show the hazard ratio. Where does it lie with respect to line of unity? And the big question at the end becomes. Is there a consistent benefit across death, stroke, mi, and in FARC size?
Is it consistent now, even though you're not powered on death, even though you're not powered on stroke? Or am I, does the magnitude or the relative risk reduction parallel what you saw for the powered endpoint? And if it does, great, but if Deaths go in the wrong direction, strokes the wrong direction, or am I all those if they're going in the wrong direction?
I think you're going to have a hard time with clinicians accepting it and with regulators accepting it. ), I think it is good. It is clever. It is more efficient, but it has to be interpreted, , rigorously.
You know, how a patient views their, how they feel is an important outcome. Um, I mean We have two goals. One is to add years to patients lives, but we also want to be adding life to patients years.
[00:24:01] Dr Nero: The, , issue with win ratios , when you're looking at continuous variable, like KCCQ scores, et cetera, you win if you're just a little bit better. , and it's not really comparing those things in a realistic way is a win, no win tie situation. And so it does potentially magnify. a difference there when it's not necessarily there. And it's hard to walk that back and read that out of the results of those trials.
[00:24:30] Dr Gibson: Yeah, you know, it's a little bit like, , what we do in our elections, right? When , you say, they won the state, well, actually, they took the state by a thousand votes or something, or they could have taken the state by millions of votes, you know, uh, the, the electoral college and blurs, the differences.
That's why, again, I think we should have some rules of the road that people need to present things like, you know, what was the KCC score in this group versus that group? , The other thing people often don't understand when you're using scores is they think of them as, , numbers instead of categories.
, A score of, say, class example for me. Timmy flow grade. Drives me nuts when I'm the reviewer. Someone will say, well, the Timmy flow grade was 2. 23. Well, no, , that it's, that's like saying, you know, if you put blue, yellow, red together, the score was brown. I mean, these are, these are categories, they're names.
They're not numbers. So a three is not three times better than a one. And the way to compare them statistically is not to lump them all together is averaging them up. It's to look at the statistically. Ratios of all the threes, the twos, the ones, the zeros. So people often, miss, they use the wrong statistics in analyzing some of this data.
And that's another problem. There's pseudo, pseudo precision to kind of averaging these categories,
[00:26:09] Dr Nero: And it brings it a little bit of bias into the way that we're adjudicating events already with that score. You know, you're putting these things that you think are, , that are bound together and giving them a hierarchy that they don't necessarily deserve , and the scores may not be necessarily linear in outcomes either.
[00:26:26] Dr Gibson: right? There may not be linear in outcomes and they, that, that is assumed they're linear. And that's, that's not good mathematically.
[00:26:34] Dr Nero: One of the final problems with the win ratios is that they don't always use an unmatched variable. So when, what you were describing where you're compared to everyone else is a much, much more complicated computational event than what they tend to do, which is a matched control. And so they'll match them out one to another. Which, unfortunately, again, includes a lot of bias into your sample. Uh, you really have to identify what, how you're going to do those match controls ahead of time, uh, so that you understand who's being compared to who, uh, it is clearly much better to do a true, you know, a, a, full Monte Carlo simulation of this, where you're looking at every patient versus
every other possibility.
[00:27:21] Dr Gibson: Well, the other, the other problem is say you had a hundred patients say in any term, say they tied 90 percent of the time, but the treatment in those remaining 10 patients, it won, , say it won, uh, seven times, uh, in the other strategy, the control strategy, one, three. Well, you would say.
They would claim, well, seven divided by three, the win ratio was over two. It won twice as often as the placebo arm. And but that discounts the 90 cases where there was a tie. So one of the other problems in presenting this data is they're not presenting the massive numbers of ties. And that is hidden within the recitation of the results.
[00:28:11] Dr Nero: Yeah, that what to do with the ties is an important piece to that. And what if you tie at
the end, right? What if, do those patients just fall out, you know, essentially
[00:28:20] Dr Gibson: They do.
[00:28:21] Dr Nero: right. And now you've just eliminated a whole group of people on your way of analyzing it. And you can't necessarily then say about proportional benefit that a patient is going to
[00:28:32] Dr Gibson: Right. And so there are some emerging ways to include the ties in the calculation to make it a fair comparison, which I support.
[00:28:42] Dr Nero: So I want to quickly dive into AI, which is obviously anytime we talk about this, it's, it's going to be a bit controversial. , I will start with my own bias is that I hate the term AI. , I think it includes a lot of things that are not intelligent. , at least as far as the computers being intelligent, , large language models are not necessarily intelligent. Um, they're just spitting back out to us what we already know or , the data that's already there , generative AI is not clearly intelligent. It is a little bit more interesting. , but when we talk about, , using these large databases to determine, , potential benefits of outcomes, I think right now really it's hypothesis generating more than anything else.
[00:29:23] Dr Gibson: I think it is. , the thing that bothers me about AI is it is such a black box. We don't understand even, even the world's experts. They can't really articulate for you how it's working. That makes me very nervous. Uh, one of my favorite examples is when we hit AI. Tools to predict mortality from melanoma and, uh, the conclusion was, yes, , this AI stuff can tell you whether the person's gonna live or die.
But when they looked inside the black box, they found that the whole association with mortality was driven by the presence of a ruler. The presence of a ruler on the image. People who had a ruler on the image died. That's because it was probably some great big thing, right? And, , that's a stark reminder of the black box nature of all this.
I, I was a big fan at the beginning, but we tried hard in several scenarios to, uh, improve upon logistic regression in predicting death of my stroke on the arterial side and venous side. We succeeded. We had a higher area under the curve with AI, but, but it was modest. It was very, very modest. It's, , we're talking about a.
0. 03 difference. You know, we're talking about moving from 0. 65 to 0. 68. You know, we're not talking about big, big changes. , it was statistically significant, but I would say it wasn't all that didn't knock my feet off as a clinician. , I do think AI has made strides in the imaging areas. , that may be where It has some of its biggest punch, although, , be careful what you call AI.
I mean, sometimes it's called AI, but it's not really AI. It's not really a black box AI figuring out where the border of an echo is, but it's just a refined kind of edge detection algorithm that we already use. So I think some things are being mislabeled as AI that are not AI. They're not black box AI.
So be careful in the labeling. But, , I'm not quite sure we're there yet in terms of predicting outcomes., We continue to work on it. We're continuing to use large data sets to try and do better for individual patient prediction. Um, the calibration, I will say the calibration of AI is better than logistic regression.
In other words, if you applied this, this question to a separate data set. You tend to get a more consistent answer with AI, which I think is fascinating. But, , I think it's been overhyped a bit, but we're continuing to try and learn.
[00:31:56] Dr Nero: Yeah. I'm intrigued by the, the, the bias, you know, sort of the ruler bias , that you described. , there was a trial that looked at, trying to determine who is going to benefit from a high risk program. It was a cost benefit analysis, and what they found was that the people that we needed to focus all this, all of our efforts on were , middle class white people. , and the racial bias behind that was built into the database, right? So we were taking out people because they didn't utilize healthcare dollars because we didn't give them access to healthcare. , and what it did is it gave us an, a completely nonsensical answer. And when you look at something like the UK biobank, which I think is spectacular and interesting and wonderful in many, many ways, but every patient that has an outcome in that biobank had an outcome because there was a physician and a patient who were involved making a decision ahead of time and then they had their outcome. And so in order to try to get to the solution from that or to , the end point. Then you have to sort of understand that there's those biases there. And I know that some of the people that I've talked to on AI will say, well, if you have to build a large enough database, , it's going to eliminate a lot of these biases. And I'm not. convinced. And even if you can identify the biases, then what do you do about them?
[00:33:13] Dr Gibson: Yeah. I'm a big fan of the UK biobank and, um, anytime I say something, my son, his quantitative genomicist just shakes his head and walks me over to the computer and does some analysis on the UK biobank to try and prove me wrong. So, you know, the good, good thing about it is it does have a pretty accurate representation of the majority of people.
You know, uh, like the Swedes also have a very good database, everyone in the country's in the database that gets rid of a lot of bias. I don't care how many people you have, if you don't have adequate numbers of underrepresented people, they're underrepresented by definition. Um, having, , big numbers doesn't overcome the underrepresentation in many ways.
So on the front end, I think the data is probably better in the Swedish and UK databanks. I think that's great. , in terms of how often someone accesses health care, there's a bias there because even though everyone's in the database, someone who can't get off work because they don't go to work, they, they're going to not be able to pay their bills.
It's not going to access health care as much as someone who can pay their bills. So even though they're in there, they're not accessing health care the same way.
[00:34:29] Dr Nero: It clearly are our databases that we have here in the United States , the effect upon that is going to be enormous. We don't have data on something like 34 states are essentially eliminated when we do any kind of large national databases. And that's, that's just not going to give us the answer that we need. And like you said, it's going to eliminate a whole groups of patients
[00:34:49] Dr Gibson: well, I'll be honest, I'm a trialist, but we're doing a horrible job in trials. I mean, we, we are not able to adequately enroll, uh, women in underserved populations for a lot of reasons. Not because we're not trying. We're trying. We're desperately trying. It's just, it's very hard. You have to, we expect people to come to the trial, but we have to do a better job of bringing the trial to the people and we're just, we're not there yet.
Right.
[00:35:18] Dr Nero: about pragmatic clinical trials and the complexity of trying to pull one of those off. You know, and even with , what we're trying to do right now, it does require things like patients having access to the internet, you know, uh, and we don't really, we don't really have that.
And have them then. Yeah. give you data and even if they're not coming to the clinic, um, we're not necessarily getting all that information
that we really want to get out of it. So it's much, much more complicated than I think that people realize when they start, when they talk about the, just the
results. Well, I know we're, we're running out of time here and , at some point we should probably just do an entire hour on AI. , but, , I want to thank you again. This has been wonderful. And, , anytime that you want to talk about anything, I'm, I'm open. Uh, but this has been great.
Thank you again. Dr.
[00:36:05] Dr Gibson: Thanks, Tom.
I. Deaconess Hospital. He's also been the study chair on numerous clinical trials. , I really appreciate you spending the time with us again. and today, , we're going to be doing a little bit of a deep dive into medical statistics and trial design, maybe a bit on AI if we have the time.
And obviously these topics are really deep, so it could be multi day seminars, but we're going to try to get them done today.
[00:00:39] Dr Gibson: Great. Thanks for having me back, Tom.
[00:00:42] Dr Nero: So I'm reminded that, uh, when we're talking about statistics, my wife brought up a farsight cartoon that had the picture of hell's library and all the books are titled statistics. Uh, so, you know, I know this is one of those topics where it's so important for us to understand on the flip side, it's so boring to talk about because that's not what is important to us on a daily basis. , we want to take care of people. We don't want to be worried about the statistics.
[00:01:07] Dr Gibson: True. And we, you know, we have to make yes or no decisions, but we're dealing with a world where it's not necessarily binary. It's not black and white. It's a gray kind of bell curve. And You know, where you set that threshold for decision making is frankly hard to believe, but it's pretty arbitrary.
[00:01:29] Dr Nero: I'm reminded about something that I can't remember who had said it on one of the panels, essentially only 20 percent of the knowledge that we utilize on a daily basis is based on clinical trials.
[00:01:40] Dr Gibson: Absolutely, you know, the percent is probably 15 to 20 percent are based upon class one recommendations where we have adequate data to really make an informed choice. The rest is based upon data that may not be so good, observational data. Um, we really got led astray, for instance, in COVID where there were therapies that looked like they would work based on observational data, but did not work when you held them up to the scrutiny of a randomized trial.
In fact, most of the therapies that were proposed, , failed, , in a randomized trials. So it's really important even in the middle of an epidemic or pandemic to have the machinery in place to answer the questions. In ways that are methodologically sound.
[00:02:31] Dr Nero: , and, , going back to the guidelines, as you said, we, you know, the limitation of guidelines. And I'm always reminded that guidelines are guidelines. They're not absolutes
[00:02:40] Dr Gibson: well, guidelines are intended generally for populations. And what a lot of practicing docs say is correct. My patients don't fit into the guidelines. And , often the guidelines are based upon randomized trials, which are a select part of a population, not the entire population. So, in the future, what I think you'll see more and more of, we'll get to AI in a bit, but I think you will see more and more.
Efforts and we're making these efforts to take large data sets and individualize recommendations for individual patients based upon age, weight, kidney function, you know, all the demographics so that we truly are having shared decision making with patients. With information that says, you know, if you do this, if you lower your blood pressure this much, here's the benefit you can expect, , you, you decide how much benefit you want with how much risk you want and displaying that , in a way that is, helpful to patients.
Sometimes it's a little too complicated, but getting it down to, numbers of bodies and. , ways that people who aren't necessarily numerically, , literate is an important way of communicating. Here's what you can expect and here's the risk.
[00:04:02] Dr Nero: It will be interesting when we talk about, , sub analyses and secondary, analyses and especially with AI, which is essentially right now all going to be retrospective data is that all those things are biased. And we also have to recognize ahead of time. , that there are limitations, , but before we get to limitations, . We're going to first talk about, the standard models that we use , the composite endpoint models, Cox regression analysis, Kepler Meier curves, and things like that. What do you see as the benefits of using those statistical analysis for trials? .
[00:04:32] Dr Gibson: I, I think there are definite benefits. Um, those models have been successful for many, many years. Uh, there is though something that people aren't aware of. They assume that the benefit of a drug over placebo is constant. And when you look at the event curves, you often see that it's not constant, , for acute coronary syndromes, we often see, , very big benefits early on.
And then the relative risk reduction is actually narrows, uh, the absolute risk reduction is constant. The curves are parallel, but your relative risk reduction kind of is getting smaller and smaller. So, um, so the constancy assumption is often not met. And we often don't deal with that in traditional KM methods.
There are newer methods where you use what are called time varying covariates to adjust for the fact that the risk is changing over time. And I think that's very important. What we're doing as doctors is we have someone who comes into our office and we say, you know what, Mrs. Jones, you've done okay on that, that new anticoagulant.
You're not bleeding. So, I'm going to continue it. We are iteratively assessing risk over time. We're doing kind of multiple landmark analyses and, you know, the statistician cringes and says, Oh my God, but we just looked from the beginning, you know, and didn't do landmark analyses because you lose randomization if you look at Mrs.
Jones at three months. On the other hand, what the statisticians don't realize is Mrs. Jones is a different person at three months. She's, she passed her bleeding stress test. And her risk is now different than it was at the beginning. Uh, and intuitively, as docs, that's what we're doing. What I've been trying to do is get us as statisticians and AI people and mathematicians, is to get in sync with what we do as docs.
Let's reiteratively assess risk over time, based upon someone has, how someone has done today. One thing people don't realize, Tom, is after a year of Clopidogrel, The risk of bleeding on clopidogrel grow versus placebo is really no different. So once you pass your bleeding stress test, it's kind of all benefit from there on out, but we don't account for that in, in, in the way we interpret trial.
[00:07:03] Dr Nero: Yeah, there was a great trial that was recently released about, uh, switching over patients who are elderly from Warfarin over to DOACs. And they did worse than duax because all those patients had already passed their bleeding stress test. You know, you didn't take them at the beginning and say, okay, we're going to choose between these two. You took at the beginning of patients who were stable on one. And you know, that brings up that exact same point, . , these people have histories, they have background, they don't come in, , and yes, we randomize them into the trial, but once they're in the trial, and like you said, over time, um, they do have, they
[00:07:34] Dr Gibson: Yeah.
[00:07:35] Dr Nero: The other part about the, the regular CAM curves that we talk about is that they are really, a lot of them are based on time to first event, right? They're non hierarchical, right? You're going to, once they have an event, hospitalization, death, et cetera, um, things stop for them, right? You get that, you get that number and it's gone and you lose them essentially from, from the rest of the trial. And I think that that's also going to be one of those pieces where, you know, as we're trying to get these trials off the ground and get information for us. Um, that we're a bit limited because of that. And you know, obviously people care about death. They care about stroke. They might care about stroke more than death. Um, and I think that that's, you know, we're going to get into that until we talked about when ratio stuff. Um, but that's, again, one of those things that's a bit lost. Um, and we've been utilizing all of those things a lot as we'd say, okay, this, you know, this benefit, you know, this drug with this composite endpoint, mace plus whatever, um, is beneficial. But it's. All those things together for first events and then that's it.
[00:08:36] Dr Gibson: Yeah. Yeah. My anecdata, the data from my trials seems to indicate that that multiple event analysis or time to any event rather than time to first event analysis in my studies, my, uh, practice has been that they're more statistically powerful. The consensus among everyone is they're probably about the same power.
I've had better luck with it. On the other hand, putting statistical power aside and our ability to see a difference, , if you're a patient, you kind of are interested in the burden of all of this, , as a patient, you're really interested in two things. Am I alive? And I am I out of the hospital and, you know, that's an end point that I've been as advocating for is number of days alive and out of the hospital that takes into account all of these things, like whether it's a stroke or a, um, am I, or bleeding, you know, so in our studies, we're looking at any SAE that resulted in a hospitalization, a cardiovascular SAE that resulted in a hospitalization, Counts in that kind of analysis, are you alive and are you, are you out of the hospital, uh, free from any cardiovascular kind of event?
So from a patient perspective, I think that is a more patient centric kind of end point. That's what patients want. I agree with you, Tom, that when you look at, uh, what are called utility scores, how do people perceive. You know, all these end points as compared to death. If you put death as a one, people are surprised.
Patients will say a disabling stroke is worse than death. Uh, they'll put it up at about a 1. 1. They'll put 7, uh, and they will put, you know, bleeding down in that 0. 4, 0. 3 range. We treat all these end points as equal and they're really not equal. And this is something that's being revisited now. How can we build quantitative.
Kind of end points that fit into what you called the hierarchy of how do people value them? So I agree with you. I think it is a gross Approximation to what patients really want to know and I agree with you If you when let me know when you want to get into it when ratios and how they they get around some of these issues
[00:11:02] Dr Nero: Yeah, the, uh, the issue of game theory is you're sort of alluding to and sort of, and there've been a lot of discussion in the behavioral economics world about these kinds of things and that we really make decisions that are, that are value based and that are, uh, that are have a ratio against what we think is of a higher or lower value. We're afraid of losing money a lot more than we're favor of making, for example, even though those two things should be equal. Right? And so similar things, you know, no people, you know, my patients may be very concerned about, you know, obviously death and heart attack. They may not necessarily be quite as worried about bleeding events.
And the way we look at bleeding events includes a whole bunch of things
[00:11:43] Dr Gibson: Yeah, you know, I was asked to write an editorial for JACC recently, and I'm opposed to this blending of bleeding events and ischemic events together. Patients are, may have different perspectives on bleeding versus ischemia. Um, and if you lump them together. It's obscuring the differences, you know, in the directions and the magnitudes.
Uh, so I think most clinicians out there would agree, patients want to know, here's the upside and here's the downside. They don't want a net. They want to know the upside and the downside. They also don't want and should not receive guidance using relative risk. Uh, people really need to know the absolute risk.
And when you look at guidelines related to informing patients, it says use absolute risk. And if you have a numeric patients, put up bodies, you know, body counts. The problem with relative risk is You could say, well, it's going to cut your risk of, uh, stroke in half. Well, if the risk of stroke is 0. 2 percent and you're cutting it down to 0.
1%, you know, you're talking about vanishingly small numbers of events, uh, and it doesn't really convey that when you talk to a patient. So my practice is use absolute risk and separate it for ischemic or upside benefits and risk like bleeding.
[00:13:14] Dr Nero: , one of the other good things about the, and when we talk about these, uh, KM curves , and this analysis. Is that you are able to use net benefit and if, , a net benefit bill correctly is going to be a very appropriate way of looking at that because, you know, they don't stroke as a negative of a hemorrhagic stroke is something that is clearly, , taking away from their benefit from whatever else potentially that they're going to do of the heart attack that's avoided.
[00:13:42] Dr Gibson: . I was very critical in this editorial of lumping hemorrhagic and ischemic stroke together, , in AFib trials, because this ischemic stroke, we're trying to prevent it, right? The drugs we're using to prevent it are causing hemorrhagic strokes that are worse in terms of mortality and disability.
Then ischemic stroke. ,
[00:14:03] Dr Nero: and importantly is that sometimes the embolic event causes a
secondary hemorrhagic stroke. And so you can't really assess that as being clearly just a net negative.
, it's quite complex
[00:14:14] Dr Gibson: very complex, but what's happening right now is they're lumping them together. And as you have drugs that are more potent, uh, they may reduce some of the ischemic strokes. It's good, but they may cause obviously more hemorrhagic strokes. And on the net, it looks the same, , because they're cancelling each other out.
On the other hand, if you're causing more hemorrhagic strokes, you could have more disability. The other problem with lumping them, two of them together is if you're doing non inferiority studies. You're always driving yourself to the null, you're always driving yourself to no difference with new regimens, and it can hide differences between the strategies.
So for instance, you could use like low dose Edoxaban, which had more ischemic strokes than the high dose. But had much fewer fatal bleeds, much for your GI bleeds and actually had improved survival where the high dose didn't. So if you just focus on lumping together ischemic stroke and hemorrhagic stroke, you can miss the GI bleeds, you can miss the fatalities, , and certainly when the FDA reviewed the data, say for a drug like Edoxaban, they separated out ischemic and hemorrhagic strokes and looked at all of the different endpoints.
Separately, but the funny thing is they reached the conclusion that, yeah, we're going to give a labeling to the high dose, even though the low dose reduced mortality and reduced GI bleeding, you know, so it just points to what you said. Time used the word. It's very complex. It is very complex.
[00:15:48] Dr Nero: I also wanted to just circle back just for a second back to your, uh, comment about using SAEs and SAEs that cause hospitalizations. And there's some really nice data that's come out saying that any hospitalization for any of these things increases their mortality over time. And , you may not necessarily see that mortality within the trial, but you will see that mortality within that patient. , those things really are much greater than we think , and it's worthy, but it's different than the event that is somewhat similar that doesn't bring them into the hospital.
[00:16:22] Dr Gibson: Yeah, I agree. , a perfect example is like, um, a non fatal bleed. You know, some people have said a non fatal bleed is just a bad day. I, I agree a little bit with that, but on the other hand, if a non fatal bleed in an elderly woman, if she says, look, I don't know if it's the pink pill, the green pill or the blue pill or the white pill, I'm bleeding and I'm going to stop all four of these pills.
And what person does is they stop all the evidence based medicine. So these SAEs can trigger, particularly if they're enough to be hospitalized, they can trigger someone to stop and get off the. off the evidence based medicines that they're on. So it's a hidden kind of hazard.
, one of the nice things about those older models that we've been using for many, many years is that. , we understand them, , we understand what a P value is. , we can compare them.
[00:17:14] Dr Nero: Most of us understand the benefits of using an absolute risk reduction, I hope. Now we're going into , wind ratios and new ways of doing these analyses. And the problem with the older models is that you have to see a lot of patients to ensure that you're getting that randomization. Creates very large trials, you're waiting for events, they are time to first event driven, and they get stopped when they, once they've reached that threshold because we want to get those drugs out. , but we're losing statistical power with that. , so there's some, , trials that you've been, , working on talking about wind ratio.
So if you don't mind us sort of describing that a little bit
[00:17:49] Dr Gibson: Sure. Let me back the truck up a little bit, , and talk a little bit about the P of 0. 05. Uh, we're having a lot of discussion among trialists and statisticians about, you know, should we continue to worship at that altar? A hundred years ago, one day, this guy Fisher woke up and decided, , 5 percent was the number.
, and, , people view a P of 0. 05 , like it's a universal constant. I said earlier that people don't realize just how kind of arbitrary some of these things are. The other thing is, , if we don't, think there's any kind of inferior kind of outcome to this new drug, why are we looking at both tails of the distribution?
Why aren't we just looking at the, , one tail and using a PA 0. 05 , on the superior tail? So some of us have moved more and more in our interpretation away from just worshiping at the altar of 0. 05 to looking at the confidence intervals. So say you had a drug that had a 10 percent benefit.
Well, there's a confidence interval around it, and that confidence interval could, it could be 21 percent better, or it could be 1 percent worse. You know, now because it could be just that 1 percent worse, the hazard ratio is 1. 01 or 1. 005. People can could see it didn't work well, you know, I'm not quite sure that particularly given the a priori data that, you know, the kind of pre test probability that you bring to that question, it probably is better and we're, we're probably losing some drugs at that nominal statistical significance.
Yeah, I'm a have been a cardio renal panel member. I did approve entresto. for HFpEF. With a P value of 0. 06, , there was compelling data from other trials and animal data, et cetera, that, that made me want to do that. Let's now talk about win ratios. We rehearsed all of the potential.
downsides of some of the traditional statistics. One of which you pointed out is like an MI is treated the same as a death. Uh, it doesn't treat the data in any hierarchical manner. It only looks at what happened first. Well, the way the win ratio works is it gets around that. It puts a hierarchy to all these endpoints.
So death is number one. Uh, you could put stroke or MI is number two and then stent thrombosis or something like that is number four. And then you could have. An outcome like, , infarct size as a continuous variable, so the way you win ratio works, it's fascinating. , it's good because you test for death first. That's the most important end point. That's great. Now, if we had a hundred patients in a trial, patient number one, And the treated arm would , compete against all 100 patients in the placebo arm.
And you'd say, did they win on death against this patient or not? And, um, you would count up the numbers of wins and losses. Okay. You would hope you would win more often than you would lose, right? Now, on the other hand, if it was a tie, say neither of them had death. Then they progress to the next endpoint, say that stroke, and then that patient one competes against all 100 patients on stroke.
If neither of them had a stroke, then you progress to test MI, and you go on and on and on. Then finally at the bottom. At the bottom of the testing, everyone had an infarct size measured. And you then compete on the size of the infarct size. So there aren't going to be any ties. There's going to be a winner and a loser.
And you see how often you win on something like infarct size. Well, the practical outcome of this in, say, STEMI trials, is we've been able to get the sample size down from 8 to 10, 000 patients for death in a my two down to 2, 500 for patients with death and my stroke and, , stent thrombosis and infarct size, the peak troponin.
And, , that has obviously massive implications for trial conduct. You can get the trials done more quickly, more efficiently. , at the end of the day, though, then the question becomes. , what does it mean to have won, , 15 percent more often in the end point? It's not necessarily as intuitive as some of the numbers we have, like the event rate, right?
The other question becomes like, well, okay, so there was really no difference in death and my stroke, how often you won, but it was all driven by the infarct size, the last thing you tested. , and that's not great either, because, , you're really not seeing anything going on in the harder endpoints.
So when you present a win ratio, and this hasn't been done well in the past with some of the studies, it's important to show for every single endpoint, like death, stroke, MI, show the hazard ratio. Where does it lie with respect to line of unity? And the big question at the end becomes. Is there a consistent benefit across death, stroke, mi, and in FARC size?
Is it consistent now, even though you're not powered on death, even though you're not powered on stroke? Or am I, does the magnitude or the relative risk reduction parallel what you saw for the powered endpoint? And if it does, great, but if Deaths go in the wrong direction, strokes the wrong direction, or am I all those if they're going in the wrong direction?
I think you're going to have a hard time with clinicians accepting it and with regulators accepting it. ), I think it is good. It is clever. It is more efficient, but it has to be interpreted, , rigorously.
You know, how a patient views their, how they feel is an important outcome. Um, I mean We have two goals. One is to add years to patients lives, but we also want to be adding life to patients years.
[00:24:01] Dr Nero: The, , issue with win ratios , when you're looking at continuous variable, like KCCQ scores, et cetera, you win if you're just a little bit better. , and it's not really comparing those things in a realistic way is a win, no win tie situation. And so it does potentially magnify. a difference there when it's not necessarily there. And it's hard to walk that back and read that out of the results of those trials.
[00:24:30] Dr Gibson: Yeah, you know, it's a little bit like, , what we do in our elections, right? When , you say, they won the state, well, actually, they took the state by a thousand votes or something, or they could have taken the state by millions of votes, you know, uh, the, the electoral college and blurs, the differences.
That's why, again, I think we should have some rules of the road that people need to present things like, you know, what was the KCC score in this group versus that group? , The other thing people often don't understand when you're using scores is they think of them as, , numbers instead of categories.
, A score of, say, class example for me. Timmy flow grade. Drives me nuts when I'm the reviewer. Someone will say, well, the Timmy flow grade was 2. 23. Well, no, , that it's, that's like saying, you know, if you put blue, yellow, red together, the score was brown. I mean, these are, these are categories, they're names.
They're not numbers. So a three is not three times better than a one. And the way to compare them statistically is not to lump them all together is averaging them up. It's to look at the statistically. Ratios of all the threes, the twos, the ones, the zeros. So people often, miss, they use the wrong statistics in analyzing some of this data.
And that's another problem. There's pseudo, pseudo precision to kind of averaging these categories,
[00:26:09] Dr Nero: And it brings it a little bit of bias into the way that we're adjudicating events already with that score. You know, you're putting these things that you think are, , that are bound together and giving them a hierarchy that they don't necessarily deserve , and the scores may not be necessarily linear in outcomes either.
[00:26:26] Dr Gibson: right? There may not be linear in outcomes and they, that, that is assumed they're linear. And that's, that's not good mathematically.
[00:26:34] Dr Nero: One of the final problems with the win ratios is that they don't always use an unmatched variable. So when, what you were describing where you're compared to everyone else is a much, much more complicated computational event than what they tend to do, which is a matched control. And so they'll match them out one to another. Which, unfortunately, again, includes a lot of bias into your sample. Uh, you really have to identify what, how you're going to do those match controls ahead of time, uh, so that you understand who's being compared to who, uh, it is clearly much better to do a true, you know, a, a, full Monte Carlo simulation of this, where you're looking at every patient versus
every other possibility.
[00:27:21] Dr Gibson: Well, the other, the other problem is say you had a hundred patients say in any term, say they tied 90 percent of the time, but the treatment in those remaining 10 patients, it won, , say it won, uh, seven times, uh, in the other strategy, the control strategy, one, three. Well, you would say.
They would claim, well, seven divided by three, the win ratio was over two. It won twice as often as the placebo arm. And but that discounts the 90 cases where there was a tie. So one of the other problems in presenting this data is they're not presenting the massive numbers of ties. And that is hidden within the recitation of the results.
[00:28:11] Dr Nero: Yeah, that what to do with the ties is an important piece to that. And what if you tie at
the end, right? What if, do those patients just fall out, you know, essentially
[00:28:20] Dr Gibson: They do.
[00:28:21] Dr Nero: right. And now you've just eliminated a whole group of people on your way of analyzing it. And you can't necessarily then say about proportional benefit that a patient is going to
[00:28:32] Dr Gibson: Right. And so there are some emerging ways to include the ties in the calculation to make it a fair comparison, which I support.
[00:28:42] Dr Nero: So I want to quickly dive into AI, which is obviously anytime we talk about this, it's, it's going to be a bit controversial. , I will start with my own bias is that I hate the term AI. , I think it includes a lot of things that are not intelligent. , at least as far as the computers being intelligent, , large language models are not necessarily intelligent. Um, they're just spitting back out to us what we already know or , the data that's already there , generative AI is not clearly intelligent. It is a little bit more interesting. , but when we talk about, , using these large databases to determine, , potential benefits of outcomes, I think right now really it's hypothesis generating more than anything else.
[00:29:23] Dr Gibson: I think it is. , the thing that bothers me about AI is it is such a black box. We don't understand even, even the world's experts. They can't really articulate for you how it's working. That makes me very nervous. Uh, one of my favorite examples is when we hit AI. Tools to predict mortality from melanoma and, uh, the conclusion was, yes, , this AI stuff can tell you whether the person's gonna live or die.
But when they looked inside the black box, they found that the whole association with mortality was driven by the presence of a ruler. The presence of a ruler on the image. People who had a ruler on the image died. That's because it was probably some great big thing, right? And, , that's a stark reminder of the black box nature of all this.
I, I was a big fan at the beginning, but we tried hard in several scenarios to, uh, improve upon logistic regression in predicting death of my stroke on the arterial side and venous side. We succeeded. We had a higher area under the curve with AI, but, but it was modest. It was very, very modest. It's, , we're talking about a.
0. 03 difference. You know, we're talking about moving from 0. 65 to 0. 68. You know, we're not talking about big, big changes. , it was statistically significant, but I would say it wasn't all that didn't knock my feet off as a clinician. , I do think AI has made strides in the imaging areas. , that may be where It has some of its biggest punch, although, , be careful what you call AI.
I mean, sometimes it's called AI, but it's not really AI. It's not really a black box AI figuring out where the border of an echo is, but it's just a refined kind of edge detection algorithm that we already use. So I think some things are being mislabeled as AI that are not AI. They're not black box AI.
So be careful in the labeling. But, , I'm not quite sure we're there yet in terms of predicting outcomes., We continue to work on it. We're continuing to use large data sets to try and do better for individual patient prediction. Um, the calibration, I will say the calibration of AI is better than logistic regression.
In other words, if you applied this, this question to a separate data set. You tend to get a more consistent answer with AI, which I think is fascinating. But, , I think it's been overhyped a bit, but we're continuing to try and learn.
[00:31:56] Dr Nero: Yeah. I'm intrigued by the, the, the bias, you know, sort of the ruler bias , that you described. , there was a trial that looked at, trying to determine who is going to benefit from a high risk program. It was a cost benefit analysis, and what they found was that the people that we needed to focus all this, all of our efforts on were , middle class white people. , and the racial bias behind that was built into the database, right? So we were taking out people because they didn't utilize healthcare dollars because we didn't give them access to healthcare. , and what it did is it gave us an, a completely nonsensical answer. And when you look at something like the UK biobank, which I think is spectacular and interesting and wonderful in many, many ways, but every patient that has an outcome in that biobank had an outcome because there was a physician and a patient who were involved making a decision ahead of time and then they had their outcome. And so in order to try to get to the solution from that or to , the end point. Then you have to sort of understand that there's those biases there. And I know that some of the people that I've talked to on AI will say, well, if you have to build a large enough database, , it's going to eliminate a lot of these biases. And I'm not. convinced. And even if you can identify the biases, then what do you do about them?
[00:33:13] Dr Gibson: Yeah. I'm a big fan of the UK biobank and, um, anytime I say something, my son, his quantitative genomicist just shakes his head and walks me over to the computer and does some analysis on the UK biobank to try and prove me wrong. So, you know, the good, good thing about it is it does have a pretty accurate representation of the majority of people.
You know, uh, like the Swedes also have a very good database, everyone in the country's in the database that gets rid of a lot of bias. I don't care how many people you have, if you don't have adequate numbers of underrepresented people, they're underrepresented by definition. Um, having, , big numbers doesn't overcome the underrepresentation in many ways.
So on the front end, I think the data is probably better in the Swedish and UK databanks. I think that's great. , in terms of how often someone accesses health care, there's a bias there because even though everyone's in the database, someone who can't get off work because they don't go to work, they, they're going to not be able to pay their bills.
It's not going to access health care as much as someone who can pay their bills. So even though they're in there, they're not accessing health care the same way.
[00:34:29] Dr Nero: It clearly are our databases that we have here in the United States , the effect upon that is going to be enormous. We don't have data on something like 34 states are essentially eliminated when we do any kind of large national databases. And that's, that's just not going to give us the answer that we need. And like you said, it's going to eliminate a whole groups of patients
[00:34:49] Dr Gibson: well, I'll be honest, I'm a trialist, but we're doing a horrible job in trials. I mean, we, we are not able to adequately enroll, uh, women in underserved populations for a lot of reasons. Not because we're not trying. We're trying. We're desperately trying. It's just, it's very hard. You have to, we expect people to come to the trial, but we have to do a better job of bringing the trial to the people and we're just, we're not there yet.
Right.
[00:35:18] Dr Nero: about pragmatic clinical trials and the complexity of trying to pull one of those off. You know, and even with , what we're trying to do right now, it does require things like patients having access to the internet, you know, uh, and we don't really, we don't really have that.
And have them then. Yeah. give you data and even if they're not coming to the clinic, um, we're not necessarily getting all that information
that we really want to get out of it. So it's much, much more complicated than I think that people realize when they start, when they talk about the, just the
results. Well, I know we're, we're running out of time here and , at some point we should probably just do an entire hour on AI. , but, , I want to thank you again. This has been wonderful. And, , anytime that you want to talk about anything, I'm, I'm open. Uh, but this has been great.
Thank you again. Dr.
[00:36:05] Dr Gibson: Thanks, Tom.
