What does Privacy and Transparency Mean Anyway?

Jennifer Owens: Hello. Welcome the Health Data Ethics Podcast. I'm your host, Jenny Owens, and I'm really glad to have you back with me. Today we're going to continue our walk through the Joint Commission and Coalition for Health AI Guidance on the Responsible Use of AI in Health Care. In today's episode, we're going to be focusing on item number two, patient privacy and transparency. So over the last couple of weeks, we covered AI policy. Turns out I had a lot to say about that. This will not be a surprise. I also have a lot to say about patient privacy and transparency. In how you can use transparency to build an atmosphere of trust and trustworthiness in your organization. want to start with what the Joint Commission guidance ⁓ directly. This is a direct quote. Health care organizations should have policies in place regarding data access, use, and protection, ⁓ well as consumer transparency disclosures or education regarding AI-enabled tools. Furthermore, To build consumer confidence, support and adoption of AI tools, organizations should develop a mechanism to disclose and educate patients and their families on the use and benefit of these tools. Hospital AI policies, we talked about that, address transparency for both hospital staff and patients, including how AI tools are used. When appropriate, patients should be notified when AI directly impacts their care and how their data may be used in the context of AI. Where and when relevant consent should be obtained. And rationale for this is very straightforward as well. ⁓ I'm going to continue quoting directly. Hospitals and health systems have a wealth of healthcare data. AI tools may rely on patient data for training data sets, produce entirely new data sets built on patient data, or provide information in a patient's data record. To protect patient privacy and ensure consumer confidence, healthcare organizations must ensure the data privacy and the transparency of AI use to patients and staff. This guidance is really eminently reasonable. There's a few things that I really like in here, and one of them is that we have all affected parties at the table. Although the guidance really focuses on transparency to patients and their families, caregivers, employees of hospitals are also in this guidance. Since hospital AI policy should address transparency for both hospital staff and patients, including how AI tools are used. So I like that we have all impacted parties of an artificial intelligence tool at the table here. guidance is really reasonable. The gap is in how do you implement this? Truly, how do you today create an atmosphere of transparency for both your employees and your patients? And is disclosure all you need? I think the answer is no, especially if you end up disclosing artificial intelligence that people may have opinions or feelings about that they cannot reasonably opt out of. And we'll talk about this in detail a little bit later in the episode. So the first thing I wanna start with is a disclosure question, because this is something that has come up in a lot of pilots that we have run. So when you are using artificial intelligence in healthcare, it is very rare that a patient will have a single AI encounter, ideally will be integrated where it makes sense all along their patient journey. ⁓ may have a scribe in their ambulatory encounter. They may have inpatient predictive tools. They may have ⁓ planning models, et cetera. ⁓ ⁓ You... Although it really doesn't seem feasible to get an individual consent every single time a model or a tool is used on a patient, that's probably not the right model, right? If I'm in bed, if I'm unconscious being monitored for sepsis, I don't need my family notified to say, hey, by the way, we're using AI to make sure that we know if she gets sepsis, that is not information that I need. On the other hand, if we go for a blanket privacy tool like the Notice of Privacy Practices, that may... a little less nuance to the situation. ⁓ notice of privacy practices is your broad notice ⁓ what privacy you can expect when you're being treated at an organization. I believe we sign them once a year. I know my poor friends in compliance, ⁓ I read every year and I send them a little email. like, look, I see we put AI in here this year. So many health systems are looking at integrating artificial intelligence into the notice of privacy practices and Cleveland Clinic has done this as well. ⁓ This is awesome, is very helpful to say upfront, hey, if you were walking through our doors to receive care, we are going to use technology to help provide you the best care that we feel is possible, and that may include artificial intelligence. However, as I often say, the number one most told lie today is I have read and agreed to the terms and conditions. A lot of people are not going to read the Notice of Privacy Practices. If they do read the Notice of Privacy Practices, they're not going to be retaining it and thinking about the ways that it applies to their care. So that broad disclosure is honest and it is helpful and it is amazing first step. But that is not the only thing that will make up a meaningful, transparent relationship with most of your patients. The other really important thing to note is that a Notice of Privacy Practices goes to your patients. It doesn't go to your caregivers. It doesn't go to your employees. So we've left an important person out of this relationship. Ideally, what I would love to see is a multi-layered approach to transparency. So yes, a broad disclosure, AI may be used in your care. And then when there are places where you can meaningfully opt out. educate the patient at that point and allow them the opportunity to do so. A great example is ambulatory scribe. It is not actually required that an ambulatory scribe be used in your ambulatory appointment. You have the opportunity to opt out. And if your doctor says, actually, this is required for you to receive care from me, then you have the opportunity to take your medical concerns to another practice that will not require you to participate. an opt out that is executable. Right. So your doctor has the ability not to use the scribe. ⁓ have the ability not to to use that doctor, there are meaningful ways for you to not have an ambulatory scribe involved in your care at all. we offer an opt out for services that can't be honored, that's where things get really tricky and it starts to feel a little bit performative. So when we're thinking about ⁓ models, for example, going back to the situation where I'm inpatient, I'm being monitored for sepsis. ⁓ It is not possible. is not technically feasible for me to opt out of that. There is no hospital level dashboard where we can flip the sepsis detection on and off per bed, nor should we. If the sepsis detection has made its way through validation, it has demonstrated that it is offering benefit that outweighs the risks to the patients of having their data disclosed. It would be irresponsible of us not to the best care possible to our patients using all of the tools that make the best care possible possible. And so in that case, we would not want to offer a patient an opt out because it is both ⁓ actually technically feasible, but also even if you were able to turn it off, let's just not look at the alert for the patient in that bed. The patient data is already there. It is really difficult to deconvolute the patient data from the model that is training all the time. And so if you offer a patient an opt-out, they have a reasonable expectation that they've said no, right? This goes all the way back to my time in clinical research. ⁓ you're doing research that requires a written consent, If a patient opts out or if they revoke that consent, not only do you remove them from the study going forward, but you go back in and you take their data out. You mark that the patient ⁓ would actually remove themselves from the study and they end up not only not participating, but that revocation goes backwards in time. So their data is not retained. And this matters a lot because if you're a patient and you're exercising your ability to revoke an informed consent, you want to make sure that you understand what this actually means. If the opt-out is not technically feasible, if we cannot offer a patient a true clinical journey that is devoid of the tool that they're trying to opt out of, ⁓ we're really offering them kind of opt-out theater or even like consent theater. And that's going to erode trust faster than any other. disclosure of limitation. The thing is is that ⁓ And I wanna be really clear and open and honest. I'm using hypothetical examples here because these are things that I hear from not only my own health system, but also from my friends at other health systems. It is very difficult to know when you are a clinician giving care, engaged in direct patient care, do you know which bucket your vendor tools fall into? If a patient says, no, thank you, I don't want AI involved in my care today, do you know which can be turned off meaningfully and which cannot? That's really going to impact your approach to provide that trustworthy relationship with your patient. brings me to my next point. There's a clinician transparency gap as well. ⁓ the Joint Commission on CHI Guidance addresses that staff transparency. It's really tempting to treat that as a training checkbox, right? You've been trained on the use of the tool. You know how to use it operationally. How do you activate it? How do you read the output? How do you it into your workflow? But that training doesn't ⁓ you a good sense of the model behavior or its limitations or its failure modes. ⁓ if we as health systems are not transparent with our clinicians ⁓ provide that explainability at the training level, clinicians are going to lack that foundation to recognize when a model's output should be questioned or when it a patient's circumstances make them a bad fit for what the model is supposed to be doing. And so I might counsel that patient to opt out of using this model because I know that it's not going to treat them, it's not going to treat their data fairly. So have a problem ⁓ on both ends, right? If we have a clinician who doesn't ⁓ the model itself, can they counsel a patient on whether the patient should opt in or opt out? And if the patient does opt in or opt out, is that a meaningful choice or have they technically been opted in? The thing is that the clinicians bear the accountability for those decisions. They need to know when AI is in the loop, what it can and cannot do, and where it might be wrong for the patient so that they can exercise their human in the loop clinical judgment. And a governance framework that doesn't distinguish that patient facing transparency from the clinician facing transparency. and a framework that won't go beyond operational training to really true transparency about the model itself, that gap is going to build, right? And that technical debt that we carry with us, when a pilot goes in and then we ramp it up across multiple locations without thinking about what does it mean if a patient wants to opt out, that technical debt then becomes an ethical debt because we have failed to recognize an opportunity to offer our patients a chance to exercise their agency. So this is really a seed that needs to be planted all the way back in the procurement and the integration design stage. You can remediate it post-deployment, but it is tremendously, it is a tremendously heavy lift to do so. So when you're doing your initial governance review, if the use case is one that patients might reasonably want to opt out of, that opt out capability needs to be a requirement, both in the contracting language and in the design, the implement, like the integration design from the start. It can't be a future roadmap. We can't say, ⁓ we're gonna enroll everybody de facto today and let them opt out at some point in the future. That's not gonna work. We need to build the opt out in from the beginning. And if you are a health system and you're asking vendors questions as you're testing out a pilot for feasibility, can you exclude a patient from this model? How would you do that? Are you able to deconvolute and pull their data out if they say, no, I don't want to do it in two weeks? path is not clean and it is not easy, but it is really honest. The health systems that get this right are going to build an atmosphere of trustworthiness. They will truly be worthy of their patients and their clinicians trust. The ones that don't are gonna feel that trust erode. In our next episode, we're gonna continue this series. We're gonna continue to tackle the Joint Commission and CHI Guidance. I really appreciate you all coming on this journey with me. I'm really enjoying this. It's a nice chance to revisit and to do a little bit of a deeper dive. If you have questions, if you have thoughts on transparency, on disclosure to clinicians or patients, please feel free, drop me a comment. would love to hear from you. Thanks.