speaker-0: Biopharma is rapidly evolving, but the evidence decision makers need most can often be the hardest to generate. Rare diseases, emerging therapies, changing treatment landscapes, and gaps in real-world evidence can all create uncertainty. This is Pharma Unscripted from RTI Health Solutions, where we bring forward important conversations about biopharma, evidence generation, and access strategy. I'm Peter Wirth, Senior Director of Business Development and Growth Strategy at RTI Health Solutions. Today, I'm joined by Senior Medical Affairs Lead Eshwati Menon and Head of Expert Insights Mark Price to discuss Delphi methodology, what it is, when to use it, and how a well-designed expert consensus process can turn incomplete evidence into credible, publishable, and actionable. Insights will explore why expert consensus is more than a fallback when clinical trial or real-world evidence is limited, how rigorous Delphi studies are designed and reported, and what pharmaceutical and biotech teams should consider when applying the findings to publications, regulatory submissions, HTA dossiers, and market access strategy. All of which has been summarized in our newly released practical guidebook. on Delphi Methodology live on RTIHS website right now. Enjoy my conversation with Ash and Mark. So Ash to start, can you give us a brief overview of Delphi methodology and why it matters when traditional evidence sources like RCT or long-term real world data are limited or unavailable?
speaker-1: Sure, Peter. ⁓ so as you correctly identified earlier, high quality data are essential for making informed decisions and developing successful scientific pipelines. But what happens when the data you need simply aren't available or are difficult and time consuming to collect? What if you need to validate assumptions for an early economic model or there's no clear agreement on a disease definition or treatment pathway? This is where Delphi panels can be really valuable. They provide actionable expert insights without the timelines or of a traditional research study. The modern Dovy method was developed back in the nineteen fifties by researchers at Rand to forecast the potential impact of technology on Warfreat, you know? ⁓ and basically what they tried to do was answer this tricky question of how to systematically gather expert opinion when there isn't necessarily one right answer. Their solution was pretty clever. Instead of putting a group of experts in a room and letting the most confident or loudest voice take over, they asked experts to give their opinions independently. Those responses were then shared back anonymously with the group, giving everyone a chance to see what their peers thought, reflect on the feedback, and if they wanted to, reconsider their own answers. This structured process was repeated over several rounds with the goal of moving from a collection of individual opinions towards a clearer picture where experts agree, where they disagree, and where certainty remains. And yes, the name does actually go all the way back to the ancient Greeks, right? And the famous Oracle of Delphi. So people obviously, ⁓ you know, at the Oracle of Delphi sought ⁓ expert guidance for uncertain questions. But fast forward to today, we're not seeking prophecies. We're looking for evidence informed expert perspectives. And that's exactly why Delphi panels have become so useful in healthcare policy and strategy. Evidence is constantly evolving, clinical practice can vary across geographies and healthcare systems. And there are often situations where expert consensus or guidance is needed before definitive evidence is available.
speaker-2: Thank you, Peter. Thank you, Ash. Delphi methodology offers a structured, transparent solution. It's a multiple round, iterative process that transforms fragmented expert opinion into defensible publication quality consensus. When it's designed and executed through a cross functional lens and with methodological rigor, including adherence to reporting standards, Delphi studies produce findings that withstand peer review. satisfy payer and regulatory scrutiny and generate actionable insights across the evidence spectrum. I've been doing this this work for nearly 23 years and during my first 12 to 15 years I would say at RTI Health Solutions, ⁓ we received maybe five requests or RFPs for Delphi studies. And in the last five to eight years, we've received somewhere around a hundred requests for Delphi, either formally or informally. So the technique has really taken hold in a EOR in support of product development and market access.
speaker-0: Great. Thanks, Mark. ⁓ so let's talk about expert consensus and we know that it's not a fallback when evidence is incomplete. So how should teams think about where Delphi fits in the evidence hierarchy?
speaker-2: That's a great question, Peter. ⁓ when we think about the evidence hierarchy, we think about that being a foundational framework in health economics and outcomes research and clinical research. And at the top in HEOR sits systematic reviews and meta-analyses of RCTs, and at the base sits expert opinion and case reports. So this hierarchy reflects the relative acceptability of different study designs to bias. But it doesn't mean that the lower tier evidence is without value. The critical distinction is between evidence that's unavailable because a study has not been conducted and evidence that's unavailable because a study cannot feasibly be conducted. For a lot of ATR questions, particularly in rare disease, ⁓ pediatric populations or areas of rapid therapeutic change, the latter applies. In these contexts, expert consensus methodology is not a compromise, but it is in fact the most rigorous approach available rather than a low tier approach.
speaker-0: Thanks, Mark. So we've talked a little about where they potentially fit, but Ash, maybe tell us a little bit about what are the most common evidence gaps where a Delphi study is especially useful.
speaker-1: Yeah. And you know, as Mark was discussing, I think a Delphi panel can be particularly useful in situations where traditional evidence is limited, emerging, difficult to generalize, or doesn't fully answer the practical question at hand. And these include rare diseases, you know, where small patient populations, strict ethical considerations, and long recruitment timelines can make conventional trials challenging. What about ⁓ Urgent public health questions, right? There simply isn't time to wait for a traditional study in novel or emerging therapies where clinical experience is developing faster than the evidence base. These are just a few examples where Delphi studies can really help address the gaps in real-world evidence. Also, you know, in in in the real-world evidence base, ⁓ Delphi panels can also address gaps in databases where They like the granularity or the endpoints needed to answer specific questions. Another important area is variation and clinical practice across geographies and healthcare systems. A Delphi panel can bring together experts from different regions to identify both differences of practice and also areas of common ground. They can be a valuable tool for HTA and peer modeling, right? head-to-head evidence may be unavailable and sometimes expert input needed to address these gaps and inform assumptions around treatment pathways, resource use, and also outcomes. Also finally, I think Delphi can support clinical guideline development, particularly when the available evidence is incomplete or conflicting. In these situations, the strength of the approach is not simply that it provides expert opinion, but that it systematically captures tests and documents expert judgment. Producing a transparent and defensible consensus that can be published in peer reviewed journals, used as a call for action to create, modify or expand clinical guidance for medical conditions. So, broadly speaking, Delphi is most useful when the evidence exists but doesn't quite answer the question, we need to make a decision.
speaker-0: Right. Thanks, Ash. So Mark, we've talked a little bit about what a Delphi is and why we do it, but tell us a little bit about how Delphi studies are done. What what does a well designed Delphi study look like in practice? What's the scoping, protocol development to panel selection, questionnaire design, iterative rounds and reporting? Just kind of walk us through how it's changed over time versus what we're doing today.
speaker-2: Yeah, absolutely. so Delphi is is really just a rigorous methodological overlay onto classic research that we might do in ⁓ qualitative topics. So the methodological rigor starts well before the first data collection, which is usually a a first Delphi survey questionnaire. You really want to design the study with the end goal in mind. very carefully about what question or questions that we're trying to answer and where will those findings ultimately be used, whether that's to go into a publication, a regulatory submission, a payer dossier, or even to develop clinical tools or influence clinical guidelines. From there, you define the evidence gap through literature or landscape review, and you develop a clear protocol up front, including who should be on the panel. How many rounds you're going to run of data collection, what actually should be defined or what constitutes consensus, and when you're going to stop data collection. Then comes what's probably one of the most important pieces: selecting the right experts to service panelists. Delphi panels are often smaller than people expect. They're typically anywhere from, say, four or five to up to 20 participants. But it's the quality and relevance of the experts that really matters, not simply the number and the sample size. You want the panel to reflect a right mix of medical specialties, geographies, care settings, and clinical experience while also having transparent processes for managing conflicts of interest. And finally, you need ⁓ well-designed questionnaires. The questions ought to be grounded in the evidence, clearly worded. Tested for content validity, including rating scales that are complemented by opportunities for experts to explain their reasoning and rationale. So essentially a a rigorous Delphi is not just about asking experts questions. Every phase of the process has to be carefully designed, documented, and reported to ensure that the final output meets publication and submission standards. Yeah.
speaker-1: Thanks, Mark. And you know, I think once you have that perfect questionnaire designed, ⁓ it's also important how you administer them and how you collect the data, analyze the results, right? So typically for a traditional Delphi panel, we take these questionnaires and they're administered to the panelists in at least two iterative rounds. In the first round, panelists independently rate the questions or statements and We analyze these responses to understand both the overall level of agreement and how much opinions vary. Then comes the really important part. Between rounds, we share the anonymized group feedback, things like medium response, level of agreement, selected comments, and share it with the entire panel. We do so so that panelists can see where they stand compared with the wider group and decide if they want to revise their responses. We then repeat this process in round two, and sometimes additional rounds are needed for more complex questions or when consensus is so low. More importantly, the consensus thresholds are defined upfront rather than adjusted to fit the study results. This provides an objective framework which allows us to look at whether responses are becoming more stable between rounds and also explore differences across specialties, regions, or practice settings. And if the panel does not reach consensus after several rounds, that's not a failure. That's a finding in itself, right? So a Delphi panel has ⁓ several outputs of agreement, disagreement, and uncertainty. And when we analyze that data, once the analysis is complete, these findings can be translated into different outputs from peer-reviewed publication to conference presentation to regulatory submission, or ⁓ HTA does here. using the same underlying evidence, but tailored specifically to the strategic needs and formats of the appropriate audience and question. Over to you, Peter.
speaker-0: So clearly a very rigorous process. Thanks both for that detail in terms of how the Delphi are developed and operationalized. Let's shift gears a little bit and talk about ⁓ quality standards ⁓ like creeds and accord. Why are they so important for publication, regulatory credibility and HTA or payer use? Ash, what are your thoughts?
speaker-1: Right. I think that's ⁓ I think one of my favorite questions ⁓ because it's it's very important because when it comes to these Delphi studies, how you report the study is just as important as how you conduct it, right? Particularly if you're aiming for publications, regulatory use or an HTA submission. So two important consensus reporting frameworks come to mind. ⁓ they're specially adapted for use in the life sciences and those are CREATS or the guidance for conducting and reporting Delphi studies and a CORD, the accurate consensus reporting document. So let's let's do a quick ⁓ review of both of these. So CREETS provides a framework for transparently reporting the full Delphi process, from why a Delphi is needed, how experts are selected, quested air development, consensus threshold, how to analyze run by run results, and also how to discuss study limitations. So provides you with guidance. For a full Delphi from start to finish. Accord does the same, but I think it's even more tailored for biomedicine and provides additional guidance for reporting elements such as pre-specified items, facilitated discussion, and other methodological adaptations. Accord is also useful specifically for market access applications because it also includes a structured framework that satisfies documentation requirements for most HTA and regulatory bodies. So Ultimately, you know, we have tools available, we have these frameworks available, and you know, they really helped make sure that the readers can see not just that consensus was reached, but how we got there, how we got the findings that were observed and how transparent, reproducible and credible conclusions for peer reviewers, regulators and HTA bodies were assembled.
speaker-0: Great. Thanks, Ash. Clearly, very important to understand the implications in those quality standards and how to again apply those throughout the research process. So this is really a great buildup to this question that I've been thinking about. Maybe Mark, you can jump into us. You know, we've talked about why, how, where, the quality standards, all those things, but now it comes to let's hear some real examples. Could you give us a couple of client examples of how Delphi was used to inform? HTA and regulatory submissions so our audience can f kinda get that sense of how Delphi has been beneficial.
speaker-2: Absolutely, Peter. It's one of my favorite things to talk about because it allows me the opportunity to talk about how Delphi ⁓ research results translated into actionable insights for some of our clients. So I've got a couple of really good examples that come to mind. ⁓ we had one manufacturing client ⁓ come to us with a need to pressure test the inputs, assumptions, and costs included in a draft ⁓ cost effectiveness economic model. For an ultra rare disease that affects toddl young children soon after birth and through their development. The sequela of this particular disease are devastating and they affect mobility, articulation, fine motor skills, vision, you name it. And the impact on the appearance of these kids is also devastating because they develop these deformations from highly visible growths and tumors around the spine. and inside and behind the eye. The treatment being developed ⁓ was for this orphan indication that I just described. And HTA authorities, what was then CAD in Canada a few years ago, now it's the CMA AMC. It's the Canadian HTA body. ⁓ accepted the results of a Delphi that we designed and executed with the few expert clinicians who follow children with this disease in Canada. And they issued a positive opinion that is the the Cadith ⁓ HTA body that was used to influence reimbursement of the treatment in each of the provinces of Canada. I've got another example. There was a manufacturing client developing a treatment for an ultra-rare disease that strikes neuromuscular systems. It occurs Right in middle age, usually ⁓ around the time that people are looking at retirement, it causes a gradual loss of muscle strength and articulation. And the symptoms will start distally, like in the tips of your fingers and then your hands and your toes and then your feet. And then the condition works its way towards the body's core. And while there were no data because of the ultra rarity of this condition on mortality due to the condition, the client And we had been working with ⁓ some global expert clinicians who follow these patients. And they suspected that these patients don't live out their full life expectancy because as it works to the body's core, it starts to impact bulb or dysfunction, swallowing, it starts to impact the diaphragm, and it has quite a ⁓ quite an effect on your ability to move your hips and your legs. All of that becomes implicated. And you can guess. that with all of those things, there's an increased risk of aspiration pneumonia if you can't swallow correctly, breathing issues if your diaphragm is is ⁓ implicated. And of course, as we get older, falls with ⁓ hip and leg fractures become more and more serious ⁓ in that patient population. So we designed and conducted a a Delphi with 13 expert clinicians in eight countries. And they all agreed that mortality does in fact come earlier on the whole for these patients. So even though that that mortality data had not yet been completely codified, the results were submitted to FDA and they gave accelerated approval pathway for this particular treatment. So you can see those are two powerful examples of how Delphi data can influence HTA and regulatory decision making.
speaker-0: Right. Thanks for sharing those, Mark. And definitely you can see the impact of Delphi in supporting these assets and ultimately the patients that are receiving these important treatments. So thanks for sharing those. So wrapping all these pieces up as we kind of come down towards the end of our time together, I'll let both of you jump in here, maybe a Ash first. What are some key quick tips, five tips for our pharma biotech teams that are considering Delphi study. What what should they be thinking about or what advice can we give them to take away from the podcast today?
speaker-1: Great, you're putting me on the spot, Peter, but here we will try and give it a go. So if we have to give Pharma or Biotech teams, I guess, five quick tips, I think I'd start with use the frameworks, right? We have creeds and a chord and they provide a really useful checklist for good research practice. You know, they help you make sure that you've thought through every aspect of the study from start to finish. And ⁓ I think it's worthwhile investing time using that as a good starting point. So that's number one. Secondly, I think it's very, very important to recruit the like right panelists. It's it's not about having the biggest panel. It's more about having the right mix of expertise, geography, clinical experience, and capturing the perspectives that will genuinely help you answer the question you're trying to address. Mark, what do you think? Do you have some tips for us?
speaker-2: Having a background in in survey development, I would recommend to design your questionnaires well. Make sure to use the right type of question for the topic. And design questions that don't just ⁓ seek a dichotomous affirmative or negative, but really get at qualitative nuance and rationale for why a clinician would take the action that they say they would take. Another favorite type Of survey question that I like to embed in these Delphi questionnaires are patient vignettes. We like to create hypothetical but plausible in real life patient vignettes that describe an individual patient's demographics, access to treatment resources, how far away they live from those treatment resources, what sort of ⁓ personal care network do they have? All of that in addition to consideration of the disease and the treatment characteristics for that patient. And then Ask the expert clinician who's serving on the Delphi panel, how would you choose or modify a treatment for that particular patient relative to other patients that may have different characteristics? And then the other thing I might recommend is to think about the end product from the beginning. The very best Delphi publications result in a practical tool that can be used by clinicians in real world settings to help guide their clinical judgment as they triage a given patient. We've had these tools printed out as posters. or on long skinny laminated cards that can be slipped into a lab coat pocket for easy reference. Also the best publications, if if you have a very well-chosen sample on your Delphi panel and you have these influential clinician authors who participated in the study on ⁓ the manuscript, you have the opportunity to make a powerful call for professional societies to revisit or create new guidelines for the diagnosis, monitoring, and treatment of patients. ⁓ when the treatment landscape continues to evolve and better treatments for a disease become available, or even when an initial orphan treatment in a rare disease becomes available because those guidelines don't yet exist since care has been palliative up to the point of there being an orphan treatment.
speaker-1: Yeah, absolutely agree, Mark. And I think that's a good segue to, you know, the last take home message that wanted to provide. And, you know, that's very simple and premise. And it's don't stop when that Delphi paper is published, right? Publication is just the first step to scientific dissemination. The real value of a Delphi comes from translating the findings into broader evidence generation and evidence communication strategies. So expert led opinion pieces, commentaries. Plain language summaries, patient facing materials, Congress presentations, symposia, visual infographics, clinical algorithms, decision aids, practical tools for the clinic. These are just some examples of how we can use Delphi insights and bring them to life, you know, practical tools and guidelines. We can also use findings from the Delphi studies to inform medical education programs, change initiatives, digital content, follow-up. evidence generation activities depending on the audience and the evidence gap we're trying to address. I mean it is our collective responsibility as scientists to make sure that sci scientific findings reach the right audience at the right time, in the right format and in the right context, not only so that they can be leveraged to inform clinical decision making, but most importantly so that they can inform better patient care. And these are just some of the tips that we can give you To make your Delphi the best out there.
speaker-0: Thanks for listening to Pharma Unscripted from RTI Health Solutions, where knowledge leads to understanding and strategy drives impact. Today, we explored how well-designed Delphi studies can turn expert judgment into credible, actionable evidence when traditional evidence is limited or still emerging. For a practical step-by-step approach to designing and conducting a rigorous Delphi study, download our Delphi methodology guide, or better yet, Reach out to Ash or Mark to discuss where Delphi may fit in your market access strategy. Join us next time as we continue exploring the evidence and access questions shaping biopharma.
speaker-1: Sure, Peter. ⁓ so as you correctly identified earlier, high quality data are essential for making informed decisions and developing successful scientific pipelines. But what happens when the data you need simply aren't available or are difficult and time consuming to collect? What if you need to validate assumptions for an early economic model or there's no clear agreement on a disease definition or treatment pathway? This is where Delphi panels can be really valuable. They provide actionable expert insights without the timelines or of a traditional research study. The modern Dovy method was developed back in the nineteen fifties by researchers at Rand to forecast the potential impact of technology on Warfreat, you know? ⁓ and basically what they tried to do was answer this tricky question of how to systematically gather expert opinion when there isn't necessarily one right answer. Their solution was pretty clever. Instead of putting a group of experts in a room and letting the most confident or loudest voice take over, they asked experts to give their opinions independently. Those responses were then shared back anonymously with the group, giving everyone a chance to see what their peers thought, reflect on the feedback, and if they wanted to, reconsider their own answers. This structured process was repeated over several rounds with the goal of moving from a collection of individual opinions towards a clearer picture where experts agree, where they disagree, and where certainty remains. And yes, the name does actually go all the way back to the ancient Greeks, right? And the famous Oracle of Delphi. So people obviously, ⁓ you know, at the Oracle of Delphi sought ⁓ expert guidance for uncertain questions. But fast forward to today, we're not seeking prophecies. We're looking for evidence informed expert perspectives. And that's exactly why Delphi panels have become so useful in healthcare policy and strategy. Evidence is constantly evolving, clinical practice can vary across geographies and healthcare systems. And there are often situations where expert consensus or guidance is needed before definitive evidence is available.
speaker-2: Thank you, Peter. Thank you, Ash. Delphi methodology offers a structured, transparent solution. It's a multiple round, iterative process that transforms fragmented expert opinion into defensible publication quality consensus. When it's designed and executed through a cross functional lens and with methodological rigor, including adherence to reporting standards, Delphi studies produce findings that withstand peer review. satisfy payer and regulatory scrutiny and generate actionable insights across the evidence spectrum. I've been doing this this work for nearly 23 years and during my first 12 to 15 years I would say at RTI Health Solutions, ⁓ we received maybe five requests or RFPs for Delphi studies. And in the last five to eight years, we've received somewhere around a hundred requests for Delphi, either formally or informally. So the technique has really taken hold in a EOR in support of product development and market access.
speaker-0: Great. Thanks, Mark. ⁓ so let's talk about expert consensus and we know that it's not a fallback when evidence is incomplete. So how should teams think about where Delphi fits in the evidence hierarchy?
speaker-2: That's a great question, Peter. ⁓ when we think about the evidence hierarchy, we think about that being a foundational framework in health economics and outcomes research and clinical research. And at the top in HEOR sits systematic reviews and meta-analyses of RCTs, and at the base sits expert opinion and case reports. So this hierarchy reflects the relative acceptability of different study designs to bias. But it doesn't mean that the lower tier evidence is without value. The critical distinction is between evidence that's unavailable because a study has not been conducted and evidence that's unavailable because a study cannot feasibly be conducted. For a lot of ATR questions, particularly in rare disease, ⁓ pediatric populations or areas of rapid therapeutic change, the latter applies. In these contexts, expert consensus methodology is not a compromise, but it is in fact the most rigorous approach available rather than a low tier approach.
speaker-0: Thanks, Mark. So we've talked a little about where they potentially fit, but Ash, maybe tell us a little bit about what are the most common evidence gaps where a Delphi study is especially useful.
speaker-1: Yeah. And you know, as Mark was discussing, I think a Delphi panel can be particularly useful in situations where traditional evidence is limited, emerging, difficult to generalize, or doesn't fully answer the practical question at hand. And these include rare diseases, you know, where small patient populations, strict ethical considerations, and long recruitment timelines can make conventional trials challenging. What about ⁓ Urgent public health questions, right? There simply isn't time to wait for a traditional study in novel or emerging therapies where clinical experience is developing faster than the evidence base. These are just a few examples where Delphi studies can really help address the gaps in real-world evidence. Also, you know, in in in the real-world evidence base, ⁓ Delphi panels can also address gaps in databases where They like the granularity or the endpoints needed to answer specific questions. Another important area is variation and clinical practice across geographies and healthcare systems. A Delphi panel can bring together experts from different regions to identify both differences of practice and also areas of common ground. They can be a valuable tool for HTA and peer modeling, right? head-to-head evidence may be unavailable and sometimes expert input needed to address these gaps and inform assumptions around treatment pathways, resource use, and also outcomes. Also finally, I think Delphi can support clinical guideline development, particularly when the available evidence is incomplete or conflicting. In these situations, the strength of the approach is not simply that it provides expert opinion, but that it systematically captures tests and documents expert judgment. Producing a transparent and defensible consensus that can be published in peer reviewed journals, used as a call for action to create, modify or expand clinical guidance for medical conditions. So, broadly speaking, Delphi is most useful when the evidence exists but doesn't quite answer the question, we need to make a decision.
speaker-0: Right. Thanks, Ash. So Mark, we've talked a little bit about what a Delphi is and why we do it, but tell us a little bit about how Delphi studies are done. What what does a well designed Delphi study look like in practice? What's the scoping, protocol development to panel selection, questionnaire design, iterative rounds and reporting? Just kind of walk us through how it's changed over time versus what we're doing today.
speaker-2: Yeah, absolutely. so Delphi is is really just a rigorous methodological overlay onto classic research that we might do in ⁓ qualitative topics. So the methodological rigor starts well before the first data collection, which is usually a a first Delphi survey questionnaire. You really want to design the study with the end goal in mind. very carefully about what question or questions that we're trying to answer and where will those findings ultimately be used, whether that's to go into a publication, a regulatory submission, a payer dossier, or even to develop clinical tools or influence clinical guidelines. From there, you define the evidence gap through literature or landscape review, and you develop a clear protocol up front, including who should be on the panel. How many rounds you're going to run of data collection, what actually should be defined or what constitutes consensus, and when you're going to stop data collection. Then comes what's probably one of the most important pieces: selecting the right experts to service panelists. Delphi panels are often smaller than people expect. They're typically anywhere from, say, four or five to up to 20 participants. But it's the quality and relevance of the experts that really matters, not simply the number and the sample size. You want the panel to reflect a right mix of medical specialties, geographies, care settings, and clinical experience while also having transparent processes for managing conflicts of interest. And finally, you need ⁓ well-designed questionnaires. The questions ought to be grounded in the evidence, clearly worded. Tested for content validity, including rating scales that are complemented by opportunities for experts to explain their reasoning and rationale. So essentially a a rigorous Delphi is not just about asking experts questions. Every phase of the process has to be carefully designed, documented, and reported to ensure that the final output meets publication and submission standards. Yeah.
speaker-1: Thanks, Mark. And you know, I think once you have that perfect questionnaire designed, ⁓ it's also important how you administer them and how you collect the data, analyze the results, right? So typically for a traditional Delphi panel, we take these questionnaires and they're administered to the panelists in at least two iterative rounds. In the first round, panelists independently rate the questions or statements and We analyze these responses to understand both the overall level of agreement and how much opinions vary. Then comes the really important part. Between rounds, we share the anonymized group feedback, things like medium response, level of agreement, selected comments, and share it with the entire panel. We do so so that panelists can see where they stand compared with the wider group and decide if they want to revise their responses. We then repeat this process in round two, and sometimes additional rounds are needed for more complex questions or when consensus is so low. More importantly, the consensus thresholds are defined upfront rather than adjusted to fit the study results. This provides an objective framework which allows us to look at whether responses are becoming more stable between rounds and also explore differences across specialties, regions, or practice settings. And if the panel does not reach consensus after several rounds, that's not a failure. That's a finding in itself, right? So a Delphi panel has ⁓ several outputs of agreement, disagreement, and uncertainty. And when we analyze that data, once the analysis is complete, these findings can be translated into different outputs from peer-reviewed publication to conference presentation to regulatory submission, or ⁓ HTA does here. using the same underlying evidence, but tailored specifically to the strategic needs and formats of the appropriate audience and question. Over to you, Peter.
speaker-0: So clearly a very rigorous process. Thanks both for that detail in terms of how the Delphi are developed and operationalized. Let's shift gears a little bit and talk about ⁓ quality standards ⁓ like creeds and accord. Why are they so important for publication, regulatory credibility and HTA or payer use? Ash, what are your thoughts?
speaker-1: Right. I think that's ⁓ I think one of my favorite questions ⁓ because it's it's very important because when it comes to these Delphi studies, how you report the study is just as important as how you conduct it, right? Particularly if you're aiming for publications, regulatory use or an HTA submission. So two important consensus reporting frameworks come to mind. ⁓ they're specially adapted for use in the life sciences and those are CREATS or the guidance for conducting and reporting Delphi studies and a CORD, the accurate consensus reporting document. So let's let's do a quick ⁓ review of both of these. So CREETS provides a framework for transparently reporting the full Delphi process, from why a Delphi is needed, how experts are selected, quested air development, consensus threshold, how to analyze run by run results, and also how to discuss study limitations. So provides you with guidance. For a full Delphi from start to finish. Accord does the same, but I think it's even more tailored for biomedicine and provides additional guidance for reporting elements such as pre-specified items, facilitated discussion, and other methodological adaptations. Accord is also useful specifically for market access applications because it also includes a structured framework that satisfies documentation requirements for most HTA and regulatory bodies. So Ultimately, you know, we have tools available, we have these frameworks available, and you know, they really helped make sure that the readers can see not just that consensus was reached, but how we got there, how we got the findings that were observed and how transparent, reproducible and credible conclusions for peer reviewers, regulators and HTA bodies were assembled.
speaker-0: Great. Thanks, Ash. Clearly, very important to understand the implications in those quality standards and how to again apply those throughout the research process. So this is really a great buildup to this question that I've been thinking about. Maybe Mark, you can jump into us. You know, we've talked about why, how, where, the quality standards, all those things, but now it comes to let's hear some real examples. Could you give us a couple of client examples of how Delphi was used to inform? HTA and regulatory submissions so our audience can f kinda get that sense of how Delphi has been beneficial.
speaker-2: Absolutely, Peter. It's one of my favorite things to talk about because it allows me the opportunity to talk about how Delphi ⁓ research results translated into actionable insights for some of our clients. So I've got a couple of really good examples that come to mind. ⁓ we had one manufacturing client ⁓ come to us with a need to pressure test the inputs, assumptions, and costs included in a draft ⁓ cost effectiveness economic model. For an ultra rare disease that affects toddl young children soon after birth and through their development. The sequela of this particular disease are devastating and they affect mobility, articulation, fine motor skills, vision, you name it. And the impact on the appearance of these kids is also devastating because they develop these deformations from highly visible growths and tumors around the spine. and inside and behind the eye. The treatment being developed ⁓ was for this orphan indication that I just described. And HTA authorities, what was then CAD in Canada a few years ago, now it's the CMA AMC. It's the Canadian HTA body. ⁓ accepted the results of a Delphi that we designed and executed with the few expert clinicians who follow children with this disease in Canada. And they issued a positive opinion that is the the Cadith ⁓ HTA body that was used to influence reimbursement of the treatment in each of the provinces of Canada. I've got another example. There was a manufacturing client developing a treatment for an ultra-rare disease that strikes neuromuscular systems. It occurs Right in middle age, usually ⁓ around the time that people are looking at retirement, it causes a gradual loss of muscle strength and articulation. And the symptoms will start distally, like in the tips of your fingers and then your hands and your toes and then your feet. And then the condition works its way towards the body's core. And while there were no data because of the ultra rarity of this condition on mortality due to the condition, the client And we had been working with ⁓ some global expert clinicians who follow these patients. And they suspected that these patients don't live out their full life expectancy because as it works to the body's core, it starts to impact bulb or dysfunction, swallowing, it starts to impact the diaphragm, and it has quite a ⁓ quite an effect on your ability to move your hips and your legs. All of that becomes implicated. And you can guess. that with all of those things, there's an increased risk of aspiration pneumonia if you can't swallow correctly, breathing issues if your diaphragm is is ⁓ implicated. And of course, as we get older, falls with ⁓ hip and leg fractures become more and more serious ⁓ in that patient population. So we designed and conducted a a Delphi with 13 expert clinicians in eight countries. And they all agreed that mortality does in fact come earlier on the whole for these patients. So even though that that mortality data had not yet been completely codified, the results were submitted to FDA and they gave accelerated approval pathway for this particular treatment. So you can see those are two powerful examples of how Delphi data can influence HTA and regulatory decision making.
speaker-0: Right. Thanks for sharing those, Mark. And definitely you can see the impact of Delphi in supporting these assets and ultimately the patients that are receiving these important treatments. So thanks for sharing those. So wrapping all these pieces up as we kind of come down towards the end of our time together, I'll let both of you jump in here, maybe a Ash first. What are some key quick tips, five tips for our pharma biotech teams that are considering Delphi study. What what should they be thinking about or what advice can we give them to take away from the podcast today?
speaker-1: Great, you're putting me on the spot, Peter, but here we will try and give it a go. So if we have to give Pharma or Biotech teams, I guess, five quick tips, I think I'd start with use the frameworks, right? We have creeds and a chord and they provide a really useful checklist for good research practice. You know, they help you make sure that you've thought through every aspect of the study from start to finish. And ⁓ I think it's worthwhile investing time using that as a good starting point. So that's number one. Secondly, I think it's very, very important to recruit the like right panelists. It's it's not about having the biggest panel. It's more about having the right mix of expertise, geography, clinical experience, and capturing the perspectives that will genuinely help you answer the question you're trying to address. Mark, what do you think? Do you have some tips for us?
speaker-2: Having a background in in survey development, I would recommend to design your questionnaires well. Make sure to use the right type of question for the topic. And design questions that don't just ⁓ seek a dichotomous affirmative or negative, but really get at qualitative nuance and rationale for why a clinician would take the action that they say they would take. Another favorite type Of survey question that I like to embed in these Delphi questionnaires are patient vignettes. We like to create hypothetical but plausible in real life patient vignettes that describe an individual patient's demographics, access to treatment resources, how far away they live from those treatment resources, what sort of ⁓ personal care network do they have? All of that in addition to consideration of the disease and the treatment characteristics for that patient. And then Ask the expert clinician who's serving on the Delphi panel, how would you choose or modify a treatment for that particular patient relative to other patients that may have different characteristics? And then the other thing I might recommend is to think about the end product from the beginning. The very best Delphi publications result in a practical tool that can be used by clinicians in real world settings to help guide their clinical judgment as they triage a given patient. We've had these tools printed out as posters. or on long skinny laminated cards that can be slipped into a lab coat pocket for easy reference. Also the best publications, if if you have a very well-chosen sample on your Delphi panel and you have these influential clinician authors who participated in the study on ⁓ the manuscript, you have the opportunity to make a powerful call for professional societies to revisit or create new guidelines for the diagnosis, monitoring, and treatment of patients. ⁓ when the treatment landscape continues to evolve and better treatments for a disease become available, or even when an initial orphan treatment in a rare disease becomes available because those guidelines don't yet exist since care has been palliative up to the point of there being an orphan treatment.
speaker-1: Yeah, absolutely agree, Mark. And I think that's a good segue to, you know, the last take home message that wanted to provide. And, you know, that's very simple and premise. And it's don't stop when that Delphi paper is published, right? Publication is just the first step to scientific dissemination. The real value of a Delphi comes from translating the findings into broader evidence generation and evidence communication strategies. So expert led opinion pieces, commentaries. Plain language summaries, patient facing materials, Congress presentations, symposia, visual infographics, clinical algorithms, decision aids, practical tools for the clinic. These are just some examples of how we can use Delphi insights and bring them to life, you know, practical tools and guidelines. We can also use findings from the Delphi studies to inform medical education programs, change initiatives, digital content, follow-up. evidence generation activities depending on the audience and the evidence gap we're trying to address. I mean it is our collective responsibility as scientists to make sure that sci scientific findings reach the right audience at the right time, in the right format and in the right context, not only so that they can be leveraged to inform clinical decision making, but most importantly so that they can inform better patient care. And these are just some of the tips that we can give you To make your Delphi the best out there.
speaker-0: Thanks for listening to Pharma Unscripted from RTI Health Solutions, where knowledge leads to understanding and strategy drives impact. Today, we explored how well-designed Delphi studies can turn expert judgment into credible, actionable evidence when traditional evidence is limited or still emerging. For a practical step-by-step approach to designing and conducting a rigorous Delphi study, download our Delphi methodology guide, or better yet, Reach out to Ash or Mark to discuss where Delphi may fit in your market access strategy. Join us next time as we continue exploring the evidence and access questions shaping biopharma.
