Speaker 0: Everyone. This is Molly Gammel, Becker's Healthcare. Thank you so much for tuning in to the Becker's Healthcare podcast series. Today, we're going to talk about agentic orchestration, a new approach to enterprise AI. Joining me for today's discussion is Munjal Shah, cofounder and CEO of Hippocratic AI.
Munjal, thanks for joining me today.
Speaker 1: Thanks for having me, Molly.
Speaker 0: So agentic orchestration, Munjal, this is a a new term to me, but it's something you and the Hippocratic AI team have a a lot of depth in, and you're also launching or or agentic orchestrators or this orchestration layer that moves from individual AI agents who handle discrete tasks to coordinated teams of agents managing more complex workflows. What does that evolution look like in practice, and and what challenges are you seeing in health care that really require a shift like this?
Speaker 1: Yeah. You know, I think when we started the company, we thought we were gonna build an LLM. And then we all realized that nobody wants to take an LLM and and have to turn it into something you use for your health care, tasks. And so we actually were one of the first to say we're gonna build an agent. And to do that, we had to fine tune an LOM to be able to provide the underlying the underpinnings.
We had to we then announced a set of agentic skills. We said, oh, you know, you need a skill to be able to talk to patients that are post stroke and and have slurred speech, and you need a skill to be able to deal with patients that have mild cognitive impairment and need a lot of reminding. And so we then announced that we were building a bunch of skills, and we built hundreds of them, frankly, to deal with things we saw in real life, you know, in the I think we now have 250 some odd million patient interactions, and and we just saw a lot of problems that had to be solved. But as we did that, we realized that isn't quite how health care like, it's not enough to just give you an agent to do x because then you gotta say, okay. Run this agent now.
Oh, wait. I want another agent. So let's take something like readmission prevention. You know, it isn't just one agent that does a discharge call. It might be an agent inpatient that is educating you before you leave.
It might be one you know, we're actually working on putting our our agents into a robot that might bring meds to beds for you and actually deliver the medication to you, the first dose or the second dose, which is common. Right? Because that helps to ensure patients will start taking their medication. We then will build another agent that'll call you, you know, at some regular interval that your care plan says. And so it says, you know, on day three, you're supposed to change your band, and it's really important that you're you know, so you don't get infected.
Like, our AI will make sure we call. Now what is all that? That is a group of agents designed and focused on creating an outcome. And that is what we really call an orchestrator now, which is that we think that, you know, we're not here just to deliver agents that do tasks that may or may not deliver the outcome. You know, health care wants outcomes.
We want to change patient lives, and we want to improve KPIs on which we measure how well we're delivering the quality of care. And it's time to align this AI world with that.
Speaker 0: So many health systems have already seen success using AI for individual workflows, but orchestration, patient journeys. If I'm in a CEO or COO shoes and looking through their perspective, Munjal, what are some new operational opportunities that creates that weren't possible with stand alone agents?
Speaker 1: You know, I think, you know, COO or CEO, you love to manage by objective. Right? Please lower the readmission rate. Please get my loss to follow-up closed and and help us increase better care for these patients. Please avoid leakage.
And I think that for the first time, orchestrators offer a solution based approach to using AgenTic AI. You just indicate the objective you want and the solution you're really after, the area of focus, and and you do that. So we're now providing orchestrators that help you improve all of your onboarding, improve all of your leakage, improve all of your loss to follow-up, improve your pharmacy, business, improve your inpatient experience, your ambulatory experience, reduce your readmissions, and improve your heated scores and your stars. And so, you know, if you think about it, it allows you to operate on an objective level rather than just on a task level. And I think until now, we really couldn't really ask our AI to achieve an objective and know it would go and figure out all the things needed to to do that.
There's actually three systems where they had us do discharge calls and try to avoid, readmissions. And we realized how many different agents we had to build to be able to achieve the goal. And we were in in actually, two of the systems, both of them, we ended up lowering readmission rates by 4%. One was 18 to 14, which if you know anything about readmission rate reduction was so big a change, nobody believed it. Like, we didn't believe it.
Our client health system client didn't believe it. And in fact, they made us rerun the number seven times on our side. We can we actually even we reran them because we're like, this can't be true. This has gotta be impossible. And it turned out that I mean, after seven times, we all realized, wow.
It it was really true. And then we did it again in another health system and saw another four point drop. And that's when we realized, oh, wow. Like, this is actually working.
Speaker 0: Absolutely. A a range of of settings and needs that you just highlighted there. And I'm I'm putting myself in the in the position of our our audience who I think hearing some of these things, Minjal, they might share that early sense of of disbelief or just like they're we need to reorient ourselves thinking about this in terms of what orchestration can do. You've shared an example. You talked about some of the discharge calls, but another type of call I've heard about is this example of an academic medical center where your agents contacted more than 1,700 patients who had open CT scan orders in three days, and the result was two hundred and forty nine scheduled lung cancer screenings.
Speaker 1: So that was
Speaker 0: that was one agent in one workflow. What becomes possible when the suites of agents are working together across an entire enterprise?
Speaker 1: Yeah. Let's let's take that as, like we call that the kinda lost to follow-up orchestrator. You know, that was one list. By the way, of those 250 folks who did this, scheduled it, a 180 showed up. We found nine significant findings.
In fact, there's probably gonna be three cancers found out of that. Oh, I know there was one where they just had to they caught it so early, they were able to just do a resection and didn't need to actually do, more, invasive, and, you know, chemo and other things. So it was great, that we caught it so early. So, I mean, you know, we're this is delivering real good. But our goal isn't just to do it for CT lung cancer screenings.
We should do it for everything. Right? Lost a follow-up on colonoscopies. Lost a follow-up on mammograms. You know, some of those are care gaps, but also lost a follow-up on many there's many things that patients don't get done.
And now some of that isn't just about calling the patient and persuading them and convincing him to do it. You know, that that would be fine. We have an orchestrator above of, like, five. But now think about this. What if the patient has two things that they have to do?
You don't wanna call them twice with two different agents. You wanna avoid the abrasion and call them once. Second, what if, the order is expired? Now you wanna follow-up with the doctor and say, you know, it expired, but I don't think it expired because you meant for it to expire. It just expired because the guy didn't get it done in so long.
Do you really think that mister Shaw still needs to get that done? And we could then activate and get a, additional kind of just refresh of the order. And then on top of that, there's another agent that we will build that'll keep track of all of your capacity. Okay. We're not gonna call the lung cancer folks anymore this week.
We're gonna wait a couple weeks. Why? Because there's no slots. Mhmm. Mhmm.
You're you're full. So there's no point in calling. We're not gonna be able to find any slots. And so but, oh, but our our CT machine or our other machine, our MRI is available, but not our CT or vice versa, or our bone density machine is available. Right?
Our DXA scan's available. So let's go and follow-up with all of the women who had had hip fractures who we now wanna make sure they've come back in and got another bone density scan. In fact, we're doing this use case for a customer of ours right now, going after, low bone density in women in particular and trying to encourage them to get a bone density scan. And so so imagine now you have a capacity or agent combined with each of the individual loss to follow-up agents combined with the consolidation agent that's putting it all together, to avoid abrasion. And and then, obviously, you have a scheduling agent that'll do all of the actual scheduling of the thing, plus one that follow-up with the actual physician if we need to change the order in some way or get it refreshed.
And so that is an orchestrator. We're trying to get to the outcome.
Speaker 0: And and this is work that is much more complex, conditional. It's it's multipart. It's open to interpretation in some ways like you just highlighted. So thanks for those examples. I think it's really helpful to start to understand how this differs from the agents that health systems might be used to in current day.
Another element of all this work and and scaling AI across an organization, it just naturally raises questions about governance, about trust. Hippocratic AI's Polaris uses a primary model alongside more than 30 supervising models that are validated by thousands of clinicians. So this is clearly something you and your company take quite seriously. I guess, Munjal, at a broader level, when you look around in the environment health care leaders are operating within today, what is a a common misconception or something that we might overlook about AI safety that warrants a closer look?
Speaker 1: So I said two things on that. One is, you know, trust is is, compounding. You know? So we didn't ask everybody to trust our orchestrators that obviously are a level of abstraction and independence that's that's a little bit more, right, than you've seen before. We first said trust our agent.
Now that you trust our agents, now you can trust when we combine our agents. So it's almost like I'm not gonna give you the molecule without first you trusting every atom that's going in the molecule. And so if you think about, like, we we and then and then we told you how we built the atoms themselves. We're like, look. We tested them.
We did this. We did that. So I I think that you couldn't have gotten here till now. You had to start with, alright. Here's how we took the LOM.
Here's how we train the LOM. Here's how we then built it into an agent. Here's how we validated the agent. Here's how we now run a bunch of you know, I think we've now done we've done this 250,000,000 patient interactions without one safety incident. Because we do escalate to humans when necessary, we've escalated to humans 38,000 times.
Speaker 0: Wow.
Speaker 1: So we know what happens when it happens, how to ensure, you know, and we've avoided any significant safety issues. So it's like that's that piece. But I think that I think when people are having conceptions about safety, I I think it's in the abstract mostly. It's like you've never actually done the thing, and you're worried about its safety or not. But once you do it, you realize, a, it's not it's not a worry in some things you were worried about.
And b, it is actually so you have to be, you know, watchful in this other area, but you didn't know it because you've never done it. Mhmm. Like, think about when you before you ever went skiing, would you have known, like, what you gotta worry about when you ski to not get injured? Right? Like, you know, now whenever I ski, by the way, like, I I'm always off the slope by, like, 01:00, 02:00 at the moment.
Right? Because I always feel like you get injured in the last, like, hour of your skis thing when the snow is slushy and it's horrible, at least here in California. And so but I wouldn't have known that if I never skied. So first, I I think you need to focus on that. In terms of the actual things to kinda look out for, I think there's a siren song right now around smaller models.
And the siren song says, I can give you a cheaper AI because I'm using smaller models. I'm calling them SDMs or SLMs, small language models. Okay. That's not a thing. The reason this was a big innovation in the world was it's called a large language model.
Mhmm. It's the large part that made it smart. Actually, very directly, the size of the model correlates to its intelligence. Like, everybody doesn't realize this. Like, the breakthrough for CHAT GPT came because we crossed 30,000,000,000 parameters.
What's a parameter? It's the number of connections between the neurons and the neural network. It's literally like the brain. The more connections a brain has, the smarter the brain is. So I'm like, great.
You wanna downgrade to a mouse brain? Because that's kinda what you just did, literally. You literally just downgraded to a mouse brain when you moved to an SLM. And that may work fine, but you know what it doesn't do? It doesn't catch all the corner cases, and medicine is a giant game of corner cases.
Mhmm. Mhmm. And, like, you know, like, diagnosing UTIs in women, if you you know, we don't do diagnosis. But if you did, yeah. Could a little rule system get you there eighty percent of the time?
Sure. But it's that twenty percent of the time, five percent of the time, that's when, you know, when the UTI is a possible infection risk to the kidney. And, I mean, that that that's when it gets complicated.
Speaker 0: Yeah.
Speaker 1: And so I think that, anyways, there's there's a, one of the things I think people don't realize is safety and size of model are directly related.
Speaker 0: Mhmm. Mhmm. Thank you for expanding on that. I think also just for highlighting some of the the human biases we may bring to to the unknown as these are now this technology is capable of doing things that six months, eighteen months ago just was not in existence. So I think it's really helpful to to keep that in mind, Munjal.
As we wind down here, in addition to agentic orchestration, you also rolled out what I understand to be a self-service agent builder. Can you just explain what this is for one second and also what it means for the ability of health care leaders to get to market and production more quickly?
Speaker 1: Yeah. So the the innovation of our agent builder is actually not the agent builder, but it's actually the validation. So we developed this unique way to validate agents that, you know, I haven't seen anybody else do, which is we hired 7,000 now 7,750 licensed US clinicians. We gave them, like, you know, what we call our gig app. They actually sign on on their iPhone every, you know, morning, and they see if there's tasks for them to do, and we pay them for task.
You know? And the tasks are usually act as a patient, call this new agent we're building, and or this new orchestrator, and tell us if it's making mistakes and mark every error. And one out of 50 times, they got a human on the other line, so then we got to calibrate how good the error rate was compared to the kind of, human error rate. We used to build all of our models that way. They would all get validated to this process, and we've now done 700,000 I see almost 800,000 test calls with their simulations.
And we wouldn't release a a model until it had been validated as as safer than the human equivalent. We wouldn't validate we wouldn't release a use case or an agent until we had done that, you know, and now we won't release an an orchestrator until we've done that. But we had to do that. If you we had a little agent builder where you could build your own agent, but then we would, by hand, take it through this validation system. But now what we do is we link the two.
So you can build your own agent. You can send it. It'll automatically send it for validation. It'll validate it and give you back the feedback. But then we said, wait a minute.
We don't even just have to give you back the feedback. We'll run another agent to rewrite your prompts and your scripts based upon the feedback. We'll send it for validation again, and we'll do this loop for a week without you even being involved and then say, hey. Here's what we think is the right one. Now you try it again and tell us where it is, and then we'll validate it one more time.
Speaker 0: Wow.
Speaker 1: And so the power is really in the combination of the agent builder feeding into it, but then making it a continuous improving loop that actually is somewhat autonomous. And that's the power of this self-service is is you know, we probably should have come up with a better name for it because it's not just self-service. It's our self validating way to kind of develop agents and and orchestrators around that. And so, you know, I I think that's what I'm most excited about is is we are now exporting the ability for you to make safe agents, not just you to make agents.
Speaker 0: I'm really glad you expand on that. I thought it was more of, like, just the access of building an agent was gonna be the value proposition there, but it's actually the validation and your homegrown process of safety assurance now being accessible to the end user, like, in in such a way.
Speaker 1: Right. Exactly. Exactly.
Speaker 0: Manjal, you you and Hippocratic AI just continue to be I I don't even know how many steps ahead. I I just wanna thank you so much for your time and your insights today. This was a really interesting conversation. Thanks for joining me.
Speaker 1: Yeah. Thanks for having me, Molly. It's always a pleasure.
Speaker 0: And to our audience, thank you so much for tuning in. You can tune in to more podcasts from Becker's Healthcare by visiting our podcast page at beckershospitalreview.com.
Munjal, thanks for joining me today.
Speaker 1: Thanks for having me, Molly.
Speaker 0: So agentic orchestration, Munjal, this is a a new term to me, but it's something you and the Hippocratic AI team have a a lot of depth in, and you're also launching or or agentic orchestrators or this orchestration layer that moves from individual AI agents who handle discrete tasks to coordinated teams of agents managing more complex workflows. What does that evolution look like in practice, and and what challenges are you seeing in health care that really require a shift like this?
Speaker 1: Yeah. You know, I think when we started the company, we thought we were gonna build an LLM. And then we all realized that nobody wants to take an LLM and and have to turn it into something you use for your health care, tasks. And so we actually were one of the first to say we're gonna build an agent. And to do that, we had to fine tune an LOM to be able to provide the underlying the underpinnings.
We had to we then announced a set of agentic skills. We said, oh, you know, you need a skill to be able to talk to patients that are post stroke and and have slurred speech, and you need a skill to be able to deal with patients that have mild cognitive impairment and need a lot of reminding. And so we then announced that we were building a bunch of skills, and we built hundreds of them, frankly, to deal with things we saw in real life, you know, in the I think we now have 250 some odd million patient interactions, and and we just saw a lot of problems that had to be solved. But as we did that, we realized that isn't quite how health care like, it's not enough to just give you an agent to do x because then you gotta say, okay. Run this agent now.
Oh, wait. I want another agent. So let's take something like readmission prevention. You know, it isn't just one agent that does a discharge call. It might be an agent inpatient that is educating you before you leave.
It might be one you know, we're actually working on putting our our agents into a robot that might bring meds to beds for you and actually deliver the medication to you, the first dose or the second dose, which is common. Right? Because that helps to ensure patients will start taking their medication. We then will build another agent that'll call you, you know, at some regular interval that your care plan says. And so it says, you know, on day three, you're supposed to change your band, and it's really important that you're you know, so you don't get infected.
Like, our AI will make sure we call. Now what is all that? That is a group of agents designed and focused on creating an outcome. And that is what we really call an orchestrator now, which is that we think that, you know, we're not here just to deliver agents that do tasks that may or may not deliver the outcome. You know, health care wants outcomes.
We want to change patient lives, and we want to improve KPIs on which we measure how well we're delivering the quality of care. And it's time to align this AI world with that.
Speaker 0: So many health systems have already seen success using AI for individual workflows, but orchestration, patient journeys. If I'm in a CEO or COO shoes and looking through their perspective, Munjal, what are some new operational opportunities that creates that weren't possible with stand alone agents?
Speaker 1: You know, I think, you know, COO or CEO, you love to manage by objective. Right? Please lower the readmission rate. Please get my loss to follow-up closed and and help us increase better care for these patients. Please avoid leakage.
And I think that for the first time, orchestrators offer a solution based approach to using AgenTic AI. You just indicate the objective you want and the solution you're really after, the area of focus, and and you do that. So we're now providing orchestrators that help you improve all of your onboarding, improve all of your leakage, improve all of your loss to follow-up, improve your pharmacy, business, improve your inpatient experience, your ambulatory experience, reduce your readmissions, and improve your heated scores and your stars. And so, you know, if you think about it, it allows you to operate on an objective level rather than just on a task level. And I think until now, we really couldn't really ask our AI to achieve an objective and know it would go and figure out all the things needed to to do that.
There's actually three systems where they had us do discharge calls and try to avoid, readmissions. And we realized how many different agents we had to build to be able to achieve the goal. And we were in in actually, two of the systems, both of them, we ended up lowering readmission rates by 4%. One was 18 to 14, which if you know anything about readmission rate reduction was so big a change, nobody believed it. Like, we didn't believe it.
Our client health system client didn't believe it. And in fact, they made us rerun the number seven times on our side. We can we actually even we reran them because we're like, this can't be true. This has gotta be impossible. And it turned out that I mean, after seven times, we all realized, wow.
It it was really true. And then we did it again in another health system and saw another four point drop. And that's when we realized, oh, wow. Like, this is actually working.
Speaker 0: Absolutely. A a range of of settings and needs that you just highlighted there. And I'm I'm putting myself in the in the position of our our audience who I think hearing some of these things, Minjal, they might share that early sense of of disbelief or just like they're we need to reorient ourselves thinking about this in terms of what orchestration can do. You've shared an example. You talked about some of the discharge calls, but another type of call I've heard about is this example of an academic medical center where your agents contacted more than 1,700 patients who had open CT scan orders in three days, and the result was two hundred and forty nine scheduled lung cancer screenings.
Speaker 1: So that was
Speaker 0: that was one agent in one workflow. What becomes possible when the suites of agents are working together across an entire enterprise?
Speaker 1: Yeah. Let's let's take that as, like we call that the kinda lost to follow-up orchestrator. You know, that was one list. By the way, of those 250 folks who did this, scheduled it, a 180 showed up. We found nine significant findings.
In fact, there's probably gonna be three cancers found out of that. Oh, I know there was one where they just had to they caught it so early, they were able to just do a resection and didn't need to actually do, more, invasive, and, you know, chemo and other things. So it was great, that we caught it so early. So, I mean, you know, we're this is delivering real good. But our goal isn't just to do it for CT lung cancer screenings.
We should do it for everything. Right? Lost a follow-up on colonoscopies. Lost a follow-up on mammograms. You know, some of those are care gaps, but also lost a follow-up on many there's many things that patients don't get done.
And now some of that isn't just about calling the patient and persuading them and convincing him to do it. You know, that that would be fine. We have an orchestrator above of, like, five. But now think about this. What if the patient has two things that they have to do?
You don't wanna call them twice with two different agents. You wanna avoid the abrasion and call them once. Second, what if, the order is expired? Now you wanna follow-up with the doctor and say, you know, it expired, but I don't think it expired because you meant for it to expire. It just expired because the guy didn't get it done in so long.
Do you really think that mister Shaw still needs to get that done? And we could then activate and get a, additional kind of just refresh of the order. And then on top of that, there's another agent that we will build that'll keep track of all of your capacity. Okay. We're not gonna call the lung cancer folks anymore this week.
We're gonna wait a couple weeks. Why? Because there's no slots. Mhmm. Mhmm.
You're you're full. So there's no point in calling. We're not gonna be able to find any slots. And so but, oh, but our our CT machine or our other machine, our MRI is available, but not our CT or vice versa, or our bone density machine is available. Right?
Our DXA scan's available. So let's go and follow-up with all of the women who had had hip fractures who we now wanna make sure they've come back in and got another bone density scan. In fact, we're doing this use case for a customer of ours right now, going after, low bone density in women in particular and trying to encourage them to get a bone density scan. And so so imagine now you have a capacity or agent combined with each of the individual loss to follow-up agents combined with the consolidation agent that's putting it all together, to avoid abrasion. And and then, obviously, you have a scheduling agent that'll do all of the actual scheduling of the thing, plus one that follow-up with the actual physician if we need to change the order in some way or get it refreshed.
And so that is an orchestrator. We're trying to get to the outcome.
Speaker 0: And and this is work that is much more complex, conditional. It's it's multipart. It's open to interpretation in some ways like you just highlighted. So thanks for those examples. I think it's really helpful to start to understand how this differs from the agents that health systems might be used to in current day.
Another element of all this work and and scaling AI across an organization, it just naturally raises questions about governance, about trust. Hippocratic AI's Polaris uses a primary model alongside more than 30 supervising models that are validated by thousands of clinicians. So this is clearly something you and your company take quite seriously. I guess, Munjal, at a broader level, when you look around in the environment health care leaders are operating within today, what is a a common misconception or something that we might overlook about AI safety that warrants a closer look?
Speaker 1: So I said two things on that. One is, you know, trust is is, compounding. You know? So we didn't ask everybody to trust our orchestrators that obviously are a level of abstraction and independence that's that's a little bit more, right, than you've seen before. We first said trust our agent.
Now that you trust our agents, now you can trust when we combine our agents. So it's almost like I'm not gonna give you the molecule without first you trusting every atom that's going in the molecule. And so if you think about, like, we we and then and then we told you how we built the atoms themselves. We're like, look. We tested them.
We did this. We did that. So I I think that you couldn't have gotten here till now. You had to start with, alright. Here's how we took the LOM.
Here's how we train the LOM. Here's how we then built it into an agent. Here's how we validated the agent. Here's how we now run a bunch of you know, I think we've now done we've done this 250,000,000 patient interactions without one safety incident. Because we do escalate to humans when necessary, we've escalated to humans 38,000 times.
Speaker 0: Wow.
Speaker 1: So we know what happens when it happens, how to ensure, you know, and we've avoided any significant safety issues. So it's like that's that piece. But I think that I think when people are having conceptions about safety, I I think it's in the abstract mostly. It's like you've never actually done the thing, and you're worried about its safety or not. But once you do it, you realize, a, it's not it's not a worry in some things you were worried about.
And b, it is actually so you have to be, you know, watchful in this other area, but you didn't know it because you've never done it. Mhmm. Like, think about when you before you ever went skiing, would you have known, like, what you gotta worry about when you ski to not get injured? Right? Like, you know, now whenever I ski, by the way, like, I I'm always off the slope by, like, 01:00, 02:00 at the moment.
Right? Because I always feel like you get injured in the last, like, hour of your skis thing when the snow is slushy and it's horrible, at least here in California. And so but I wouldn't have known that if I never skied. So first, I I think you need to focus on that. In terms of the actual things to kinda look out for, I think there's a siren song right now around smaller models.
And the siren song says, I can give you a cheaper AI because I'm using smaller models. I'm calling them SDMs or SLMs, small language models. Okay. That's not a thing. The reason this was a big innovation in the world was it's called a large language model.
Mhmm. It's the large part that made it smart. Actually, very directly, the size of the model correlates to its intelligence. Like, everybody doesn't realize this. Like, the breakthrough for CHAT GPT came because we crossed 30,000,000,000 parameters.
What's a parameter? It's the number of connections between the neurons and the neural network. It's literally like the brain. The more connections a brain has, the smarter the brain is. So I'm like, great.
You wanna downgrade to a mouse brain? Because that's kinda what you just did, literally. You literally just downgraded to a mouse brain when you moved to an SLM. And that may work fine, but you know what it doesn't do? It doesn't catch all the corner cases, and medicine is a giant game of corner cases.
Mhmm. Mhmm. And, like, you know, like, diagnosing UTIs in women, if you you know, we don't do diagnosis. But if you did, yeah. Could a little rule system get you there eighty percent of the time?
Sure. But it's that twenty percent of the time, five percent of the time, that's when, you know, when the UTI is a possible infection risk to the kidney. And, I mean, that that that's when it gets complicated.
Speaker 0: Yeah.
Speaker 1: And so I think that, anyways, there's there's a, one of the things I think people don't realize is safety and size of model are directly related.
Speaker 0: Mhmm. Mhmm. Thank you for expanding on that. I think also just for highlighting some of the the human biases we may bring to to the unknown as these are now this technology is capable of doing things that six months, eighteen months ago just was not in existence. So I think it's really helpful to to keep that in mind, Munjal.
As we wind down here, in addition to agentic orchestration, you also rolled out what I understand to be a self-service agent builder. Can you just explain what this is for one second and also what it means for the ability of health care leaders to get to market and production more quickly?
Speaker 1: Yeah. So the the innovation of our agent builder is actually not the agent builder, but it's actually the validation. So we developed this unique way to validate agents that, you know, I haven't seen anybody else do, which is we hired 7,000 now 7,750 licensed US clinicians. We gave them, like, you know, what we call our gig app. They actually sign on on their iPhone every, you know, morning, and they see if there's tasks for them to do, and we pay them for task.
You know? And the tasks are usually act as a patient, call this new agent we're building, and or this new orchestrator, and tell us if it's making mistakes and mark every error. And one out of 50 times, they got a human on the other line, so then we got to calibrate how good the error rate was compared to the kind of, human error rate. We used to build all of our models that way. They would all get validated to this process, and we've now done 700,000 I see almost 800,000 test calls with their simulations.
And we wouldn't release a a model until it had been validated as as safer than the human equivalent. We wouldn't validate we wouldn't release a use case or an agent until we had done that, you know, and now we won't release an an orchestrator until we've done that. But we had to do that. If you we had a little agent builder where you could build your own agent, but then we would, by hand, take it through this validation system. But now what we do is we link the two.
So you can build your own agent. You can send it. It'll automatically send it for validation. It'll validate it and give you back the feedback. But then we said, wait a minute.
We don't even just have to give you back the feedback. We'll run another agent to rewrite your prompts and your scripts based upon the feedback. We'll send it for validation again, and we'll do this loop for a week without you even being involved and then say, hey. Here's what we think is the right one. Now you try it again and tell us where it is, and then we'll validate it one more time.
Speaker 0: Wow.
Speaker 1: And so the power is really in the combination of the agent builder feeding into it, but then making it a continuous improving loop that actually is somewhat autonomous. And that's the power of this self-service is is you know, we probably should have come up with a better name for it because it's not just self-service. It's our self validating way to kind of develop agents and and orchestrators around that. And so, you know, I I think that's what I'm most excited about is is we are now exporting the ability for you to make safe agents, not just you to make agents.
Speaker 0: I'm really glad you expand on that. I thought it was more of, like, just the access of building an agent was gonna be the value proposition there, but it's actually the validation and your homegrown process of safety assurance now being accessible to the end user, like, in in such a way.
Speaker 1: Right. Exactly. Exactly.
Speaker 0: Manjal, you you and Hippocratic AI just continue to be I I don't even know how many steps ahead. I I just wanna thank you so much for your time and your insights today. This was a really interesting conversation. Thanks for joining me.
Speaker 1: Yeah. Thanks for having me, Molly. It's always a pleasure.
Speaker 0: And to our audience, thank you so much for tuning in. You can tune in to more podcasts from Becker's Healthcare by visiting our podcast page at beckershospitalreview.com.
