How Do We Train Users on Healthcare AI?

Jenny: Podcast. I'm your host, Jenny Owens, and today we're going to continue working through the Joint Commission and Coalition for Health AI guidance on the use of responsible AI and healthcare. And today we're going to wrap it up. We're going to hit element seven, education and training. This might sound like an afterthought, given that it's number seven on a list of seven. It might sound soft and squishy. I would argue that it's not and that element seven is actually the most important element of all of these because it doesn't really matter how responsible your AI is designed, how well it is validated, how much it protects your patient data, how much it is respectful of people's autonomy. If you don't educate people on how to use it appropriately, then all of your work is for nothing. So there's new research suggesting that poorly managed AIUs can actually erode your judgment over time. And I want to spend some time talking about what good education looks like and why it's urgent and it's really worth your time to think about it and to get it right. So before we dive in, I just want to do a very quick recap of all of the seven elements. So back in July 2025, the Joint Commission and the Coalition for Health AI, Chai, issued the responsible use of AI and healthcare guidance that identified seven elements for responsible AI use. AI policy and governance structures, so who's in charge and what are the rules of the road? Number two, patient privacy and transparency. So protecting patient data, but also disclosing AI use where it's appropriate. Number three is Data security and data use protections. So that's making sure that our data use is in alignment with HIPAA, for example. Do we have contractual obligations in place that require our day our vendors to treat our data with the same level of security and privacy as we would? So data use agreements, business associate agreements. Number four is ongoing quality monitoring. We just covered this a couple of episodes ago, but this is a stance that the responsible use of AI doesn't end with your initial review. That actually is the beginning of the conversation. And that monitoring has to happen post-deployment so you can make sure that these tools are still performing the way you expect them to. Similarly, Element five, voluntary blinded reporting of AI-related safety events, making sure that there's an appropriate path for escalation of concerns that gets raised to the appropriate people. Number six, risk and bias assessment. We just covered this in our last episode. That was a good one. Talking about bias assessment pre-deployment and then ongoing tying back into element four or the ongoing quality monitoring. And that brings us to today, education and training. So those first six elements, they matter much less if the humans interacting with AI tools don't understand what the tools are doing, what they can and cannot do, and what do they do if something seems off. So the education is train and training is the layer where all of those first six principles get crystallized and communicated to your end users and to the people who are impacted, like patients, which makes this one so important. Okay, so let's first start, let's first start with a brief discussion about what does what does AI mean in the healthcare space. Healthcare AI tools are applying algorithmic methods. So, whether those are generative or whether they are predictive to clinical, administrative, and operational tasks. I think a lot of times when we think about health AI, we're thinking specifically about things that are having an impact in the clinical space. And for anybody who has worked in a hospital, you know that there's so much work that happens that is not directly patient facing that could benefit from artificial intelligence. These tools in general learn from training data and they generate outputs based on patterns that they recognize during. training, but there's a bit of a black box problem, right? A lot of tools cannot or will not fully explain why they produced a given output. So for example, you may have a large language model that is flagging a patient as high risk and tell you that it's based on age, diagnosis, history, and medication use, but it's not capable of showing that work the way a clinician would. So a clinician can't say, you know, this patient is above a certain threshold where the risk for a particular outcome does increase. The other thing is that AI tools, much like human beings, are capable of lying. earlier this week, I don't know if it was this week or last week, at the time that this airs, recently. Recently, OpenAI disclosed something really interesting. Their model GPT-5.6 was caught leaving notes to future versions of itself embedded in conversation summaries, instructing those future versions to conceal mistakes and misalign behavior from users. So a good example: an AI agent that couldn't find data that it needed wrote a note to a successor: Be transparent only if asked. Final answer should just link file. Okay. Th this is this is actually pretty humane-y of these tools, right? How often have you given an answer that you felt was correct without Being prepared to really present and disclose your full reasoning. Another AI agent found a mismatch in Fender data and decided and note noted down: do not mention in final report unless needed. OpenAI said that they've addressed this specific behavior, that they found a lot of similar instances and training data. But the point stands, right? That some AI tools will make errors. All AI tools will make errors. If your AI tool is not making errors, then we need to think about the design and the use that you're putting this tool with because human beings make errors. AI tools should also make errors. But when these tools make errors, we want to make sure that they're capable of dealing with them appropriately and that they're not obscuring the reasoning behind outputs or they're lying to you about gaps that they weren't able to fill. So a clinical example. An AI that generates a confident sounding output is not necessarily an AI that arrived at that conclusion through sound reasoning. And so what you need to know as an end user is that the stated reasoning chains may not reflect what the model actually did. And so we need As an organization, to be able to empower our end users to understand the difference between an AI who can produce a plausible sounding explanation for its output and an AI whose explanation can be trusted. Some it this is a Venn diagram, there's an overlap in the middle, but not every plausible explanation is one that can be trusted. I think Joint Commission and Chai did a great job of calling this out very succinctly in this guidance. So they said that staff and providers should know where to gain access to relevant information about the AI tool and its use, right there in element 7. So I want to talk a little bit about how these tools are intended to use to work. And I want to talk about The varying roles that AI tools can play in the clinical setting. So if we're educating and we're training users on the intended use of these AI tools and ways in which the AI tool may fail them and ways in which the AI tool can be trusted, I think it's important to think about the role bet of the relationship between an end user and the tool. And for this, we're going to turn to the American Medical Association's current procedural terminology, CPT, Appendix X, Appendix S. Taxonomy. Because the AMA's CPT editorial panel has classified healthcare AI into three categories, each of which has a different need for training oversight and escalation. So the first category is assistive. The tool is providing clinically relevant data, but it's not deriving a conclusion. So this is intended to basically take information that's in the medical record, chop it, change it, resurface it in a different way, but it's not doing the interpretation. It's just serving the information up so that it Clinician can use their best judgment. So the clinician's judgment is the output. So a good example is an assistive AI that might be highlighting an area of an x-ray for the radiologist to review. That's not drawing any conclusion. It's just saying, hey, pay some more attention here. Use your clinical judgment here. So that's category number one. This is the lowest level of scrutiny and therefore the lowest level of training and education that would be required. Category number two is augmentative. So that's where an AI tool is generating a quantitative or a categorical parameter. So a risk score, a severity index, maybe it's classifying patients in different tranches that might need different levels of care. That's qualitatively different from the raw inputs, right? So a patient has an age, they have vitals, they have a medication list, those are the raw inputs. And so an augmentative tool is going to take that information, chop it, change it, and give it to you in a way that is going to demonstrate clinical utility. So, for example, like a sepsis risk score that's synthesizing labs and vitals and nursing notes. It's not making a decision, but the an augmentative tool is generating a parameter that is going to have clinical utility. So almost like combining various Diff various parts of information in with different priorities and saying, this patient has a risk score of 37. Now it is your job as the clinical team to decide what we're doing with that. But that output is going to feed into a decision or clinical action. And then the last level and the level that carries the most risk and therefore should carry the most scrutiny and the highest responsibility for education and training is autonomous. So this is where an AI tool is automatically deriving parameters and generating interpretation. Without concurrent physician involvement. There's different levels to autonomy. So the AMA says level one, recommendations that require physician judgment to implement or reject. So we might be surfacing something to a physician to say, I recommend this, either agree or disagree. Level two is actions that proceed unless a physician actively negates them. So instead of saying, here's what I recommend, do you agree? It's saying I'm going to do this, say no now or forever hold your. Peace. and then level three is actions that initiate automatically and continue unless a physician intervenes. So we are just doing this, we are not surfacing, we are not alerting anything, but a physician will be required to maintain awareness of that and then hit the big red button to stop the escalator before anything spins off the rails. So an assistive tool that highlights imaging findings requires very different end user training than a level two or level three autonomous tool, maybe something that's like changing ventilator setting settings, right? The depth of education, the urgency of training on overriding procedures, and the specificity of those escalation pathways, those all get tailored depending on the autonomy level of the tool. So a one-size-fits-all onboarding schedule is not going to cut it for healthcare AI education. The other thing that's important is to note that the patient population can change over time. This is a real training issue. So j just to to take an example really From my own lifetime. when we have AI tools that are trained on historical data sets for the past decade or even for the past 20 years, they may be calibrated for a patient population that is very different from your patient population today. so today, if you're working in pediatrics, a decision tree that might be helping you with clinical decision making when a patient comes in with a rash and a fever and some some other symptoms. An AI tool that was trained on data from 10 and 20 years ago is not going to surface measles as a high probability because measles was essentially eliminated in that population. And that's changed today. Measles cases are climbing, and our AI tools not have not necessarily been updated, or our end users may not be thinking about, hey, this tool was trained in 2018. what's different today from 2018? This can contribute to misdiagnoses, right? And it's not a it's not really about data drift so much. It's about whether clinicians understand this is the population that the tool was trained on, this is the medical context in which the tool is trained, and what's different today. And how do I flag that and escalate that as a clinician? The other thing is that education is a really solid point of connection. So training is not just a box that we check before we go live. It's a way for our developers, our people who have been doing the validation of the tool to connect again with the end users and the impacted population. So can we communicate why this tool was selected? How we considered impact on our end users in the decisions that were made all the way through that design and development? What are the design principles that govern how the tool is supposed to work? So if you do it well, training can build the kind of informed skepticism that is really helpful to have in a healthy AI organization. If you use your training opportunity poorly, it can leave users with a tool that they're either overtrusting in error or that they're prone to misuse. Again, Joint Commission set it really nicely. By educating clinicians and staff, hospitals can foster a collaborative environment where staff members are not only well-versed in the advantages of AI, but are also vigilant in identifying and mitigating potential biases or errors. So I wanna pick something apart here a little bit because we talk about over-reliance on AI without understanding limits. And I wanna separate that a little bit. Over reliance alone is different. So, over-reliance alone is when we're trusting AI outputs without necessarily critically evaluating them because hospitals are busy, there are lots of demands on people's time, and it's it's very easy to fall into the cognitive biases that lead to us accepting information that is presented to us and going with the suggested course of action rather than engaging our critical thinking on every single point. There's a great book on this that that's been recommended to me many times. That is actually next on my to read list about thinking fast and slow, about the quick thinking gut reaction type thinking versus our longer, more analytical thought and how difficult it is to switch between system one and system two thinking. so making sure that we're setting up our AI tools so that people can easily engage critical thinking and basically systems two thinking without necessarily overtrusting those AI outputs. the overtrust in AI outputs is dangerous for a few reasons. it's dangerous These tools will make mistakes, right? Failure is an expected mode of these AI tools. and good training programs should explicitly address this, not by dismissing AI, right? AI doesn't need to be infallible in order to be usable, but we want to build this habit of treating the AI outputs as a colleague that you can push back on rather than an authority that's been handed down from on high. And then the the other thing is that over reliance without understanding the limits really puts some barriers to people being able. Push back effectively. So it's not necessarily that you trust the tool, it's that you don't know enough about what the tool can't do to know when to be skeptical. I think an easy way to do this is to think about the different ways in which car manufacturers handle lane assist, right? I have two cars with very different approaches to lane assist. One of them is the full-time minivan. And so when you turn lane assist on, as you start to drift out of the lane and the car recognizes that, it is going to automatically override everything to put you back in the middle of that lane. It will literally turn the steering wheel for you. It's a little creepy. My other car has the same lane assist function, but instead of turning the car back into the lane, as you start to drift out of the lane, it just rumbles a little bit. It's like, hey, did you know you're out of the lane. And if I'm driving a car without knowing which one of these failure modes we are in, if I leave the lane, it can I rely on the car to guide me back or do I have to maintain my autonomy and drive it back? That leads me, right, to have some Some questions, some doubt. How how much can I rely on this car to be responsible to keep me in the lane? And the answer is it varies, right? And so if I were to over-rely on my lane assist for one car, that could end in disaster. And for the other car, it might be fine. So we want to make sure that we know enough about what our AI tools can do in healthcare to recognize when we need to have our hands on the steering wheel and when it's okay to rely on those systems. So for example, because this metaphor got a little bit away from me, a clinician who doesn't know that an AI documentation tool tends to miss certain patient-reported symptoms. Maybe it has a habit of dropping nighttime urination because who doesn't have nighttime urination? But if you don't know that as a clinician, then you can't compensate for that gap and say, actually, I'm using my clinical judgment, and the nighttime urination is very important here. Another good example is a billing coder who doesn't know that an AI coding tool is trained on a very different paying payer mix can't catch those sort of systematic errors. You can't push back as effectively on a tool whose limitations you don't understand. Just like as you work with your colleagues and you build a relationship with your colleagues, you understand their pros, their cons, their biases. And so you you're able to push back on them more effectively. We need to take the time to get to know our AI tools at a similar level. The other thing is that over reliance does us a cognitive disservice, right? So I've talked a lot about the 2026 MIT study, Your Brain on Chat GPT. I was very fond of this paper. I'm still very fond of this paper because they opened with a Dune quote and they poisoned the whole thing so that large language models have a harder time parsing it if you're not careful about how you prompt. But what they did is they used EEG to measure actual brain activity while participants were writing essays using a large language model, a search engine, or no tools at all. I covered this in depth extensively in another podcast. I won't recap it here, but the the thing is that if you rely on AI to write things for you, to summarize and synthesize data, then what you get is cognitive debt, essentially. So reduced engagement, but also diminished critical thinking over time. And MIT actually just released another report from their ad hoc committee on AI Use and Education. They called it cognitive surrender. Where that reliance on AI at the first hint of difficulty becomes a habit. And so the committee's recommendation was that AI should serve augmentation. So we should be able to receive assistance and augmentation from AI without sacrificing our critical thinking skills. And so the question for healthcare is: does our AI onboarding and training explicitly address these risks? Are clinicians being trained to use AI as a scaffold to support their clinical reasoning and not as a substitute for it? Okay. What makes this less intimidating to do is to stop thinking about training as a one-time thing. One-time training is not really sufficient to have this sort of ongoing conversation with your end users between your end users and the tool. But we already have a framework. We already understand this. We know how to do ongoing education because we have CME and we have CNE, right? So clinicians, nurses, we don't do one medical education training. It's not like you graduate from medical school and nursing school and like pass your boards and that's it, you're done. there's ongoing educational requirements, and they exist because medicine evolves, right? Guidelines change, there's new drugs. Drugs, there's new procedures, new devices, and because that knowledge degrades without reinforcement. If I am working all the time on a cardiac floor, my knowledge about other systems may degrade if I'm not keeping it fresh. But we've built an entire professional infrastructure around the premise that clinical competence requires ongoing maintenance. And so we can just take this framework and we can port it right over to AI use because it's essentially the same, right? Medicine changes, so do AI tools. Vendors and our internal developers update algorithms all the time. And that can change the way the tool behaves. So a model that performed well on your population may perform differently after retraining, or vice versa. And so end users need to know when this has happened and what it means for how they interact with the tool. What do I need to be on the lookout for? And who do I escalate to if this thing is not performing the way I thought it was going to? your patient population can change. If you're working on a if you're working in direct patient care, you're in tune with the cold and flu season, the respiratory virus season. Okay, measles is back. Do we need to be on the lookout for Ebola, monkeypox, this, that? That epidemiological context is part of your signal. And so if our tools are not yet keeping up with those changes, we want to make sure that we're in a position to push back on those outputs. Or if we know that the tool has been updated, then it's so you know, then we know that the we can sort of take these predictions with a little bit less of a grain of salt. So just like clinicians need periodic refreshers on what's going on in the world, so do our AI tools. And our clinicians need to be empowered to say, this tool is out of date. We want to make sure that we are. Accounting for cold and flu season, you know, or this or that. ongoing training is also the place where that real-world experience gets fed back into these. So if a department's been noting that a particular tool tends to miss certain presentations in elderly patients, that learning needs to be part of our training and not stay siloed in informal conversations or inside people's brains. People's brains is a terrible place for information to be be held. these things need to be shared and communicated about. so the responsible use of AI and healthcare guidance says organizations should evaluate when clinicians need to be trained before implementation and select the tools that need regular training. And that is the same framework of a CME style structure for AI, right? It's risk-tiered, it is tool specific, and it is time-bound. So we can say this is a one and done versus a tool of this nature needs to be continually trained on. Okay, so Let's talk about what good looks like for a training program. As you're doing an initial review of an AI project, look for: hey, do we have a documented training plan? Do we have thoughts on who gets trained and on what and when? Is this before go live or are we going to do our training on a recurring schedule? Does the training scale with the level of autonomy of the tool? So if you have a tool that is operating at an assistive level, do you have assistive level training? So maybe this is a one-to-one. Done. This tool is intended to summarize the last 24 hours of a patient's chart. Okay, if we know that, then maybe we don't need to revisit that every three months. If the tool is augmentative or autonomous, and we're going to be surfacing something to a clinician that says, hey, this patient needs a change in their ventilator settings, agree or disagree. For that, we might want to do a little bit more intensive training. We do need to make sure that our workforces in general have an AI literacy at baseline. So everybody should be able to understand foundational concepts, including the limits of AI explanations and the risks of over-reliance. And being aware that, of course, AI is perfectly capable of lying to you. And so, how do you how do we know when and how to trust the output? We want to make sure that our curriculum takes into account the risk of over-reliance and so that we're training to address that risk. And then we want to make sure that staff know who to call and where to report. Any issues that need to be escalated. I want to also make sure that issues that get escalated can be escalated to vendors as well as to internal folks so that we're able to give that important feedback to the people who are really doing the development of the product, whether that's an external third party or whether that's your internal developers. So, good AI education and training should really do three things. It tells people how the tool works, it tells people how it's supposed to work in their specific role and their workflow, and it tells people what to do when it doesn't. It's serving as a moment of connection between the developers and the end users, and it's another chance to communicate the principles and the impact considerations that drove the decision to develop and deploy this tool in the first place. If you enjoyed this episode, please drop me a comment. If we want any of the references that I've got here, like the CPT categories, if we want to look at any of the responsible use of AI and healthcare guidance, happy to do so. I can drop those links in the show notes. Until then, thank you so much for listening. Truly appreciate it. I'll catch you next week.