speaker-1: Hello, hello, hello friends. Welcome to Human of the Loop once again. we have a very, very stacked show today, lots of really funky stuff coming your way, and a very special guest. I'll allow her to intro herself very soon. But as usual you have Ioana and I. Ioana's the brains and I'm the one that just yaps. So what we're gonna do is we're gonna kick off and Mehrnoosh would you mind doing a very quick intro of yourself, say hi to everyone and tell us a bit about yourself?
speaker-0: Yeah, absolutely. Hello, everyone. It's a pleasure to be here. I'm Mehrnoosh Sameki I'm currently with Microsoft Core AI and I lead the developer experiences and tools within the Response. Personally and as a team, we're on a mission to make sure that this buzzword of trustworthy AI and responsible AI is something that can be very easily operationalized in the machine learning and generative AI lifecycle for any developer building with AI.
speaker-2: gonna jump in here because I'm getting excited. Hey folks and hi friends. thank you for tuning in today. Mehrnoosh I was smiling as you're talking about trustworthy AI and responsible AI and kind of getting it more than to be a buzzword and that's something that Chris and I talk a lot about of hey this is actually not just something to flag of yeah it's there somewhere in the background and forget about, it's critical. can you give us because I know you've been in this space for before it was cool. can you give us some insights into just a differentiator from that perspective? Because we know we should not have technology without those ingredients. But I'm curious of your take and where you're seeing kind of the real life impact of if you break it it's really gonna be broken.
speaker-0: Absolutely. So when I started in the space of trustworth AI responsible AI, we were dealing more with traditional machine learning, talking about classifiers, regression models, time series, et cetera. Back then the concern was very much around explainability of these machine learning models based on what features, what factors they're making their predictions. Or sometimes it was also about the fairness of the prediction. Let's say if it has classified someone to have a high chance of paying off the loan and hence that person might get the loan because of that prediction. We wanted to know if the machine learning model is systematically giving a a more favorable outcome to people of certain ethnicities, age group, gender groups, et cetera, et cetera. So and at some point we shifted to more of an error analysis, trying to understand how errors are embedded in the machine learning responses, if they're systematically a higher Or groups that are getting systematically higher error rates than other groups, and how can we fix it? When the generative AI boom happened, and now with AI agents, the conversation completely shifted to a new set of concerns that we have. As you know, with AI agents, they're incredibly autonomous. They can take actions on your behalf, they can interact with databases, interact with a bunch of internal and external tools. They sometimes have access to data sets that only us as employees or as certain high privileged users might have. And so it started becoming incredibly important to make sure we understand how these agents are navigating what they have access to and the tools that they have access to and all these different actions that they're making. What are these actions? What is allowed and what is not allowed? how can we give them a little bit of a identity so we can stop them, quarantine them, and and maybe restart them and kind of control them as a virtual entity. And then the other thing that was incredibly important is now with this huge autonomous action, a range of autonomous actions that they can make, can we really understand and measure with regards to certain regulations in mind or requirements of our company in mind? How are they really doing the job? Are they really adhering to the task that they're supposed to do? Are they doing any prohibited actions? That is absolutely no no. So with more power of these tools and agents and generative AI applications comes a lot of new, I think, heightened risks that we need to really make sure we're putting the right evaluation and also protection in place.
speaker-1: That's wild. So I mean, it it is a little scary, right? Like I see, you know, Ioana and I have had some interesting experiences with some deployments. We'll leave it at that. But I guess the fear the fear that I have is that like we as a society haven't quite grasped like what what agents actually can do, you know, from an end to end perspective. Like I think we're still trying to we're still thinking that they're these little automations that kick around in the background and do stuff. But actually, I feel like when I look at what companies are doing and Microsoft partners and the the ecosystem, I'm like, we're still very much like on the like five to ten percent margin of what these things can really truly do. Right. Do you see like in your role, do you see that or do you see like agents being used for a lot more wider tasks and things?
speaker-0: Yeah, so first of all, I see that folks are still very slow to take those agents into production. So everyone is being really careful about the use of agents at the moment. They are very much in the stage of development pre production. And that's where a a lot of like tools and and infrastructure that, for example, my team builds come handy. But you're you're absolutely right. I think People know this is a very incredibly powerful tool. And they have started experimenting with it. They have their doubts, not in terms of the technology capability, but in doubts of what could go wrong. And they're taking their time at this stage in order to more characterize what those potential risks could be before putting it in production. And a lot of folks have started from I'd say internal processes. So a lot of customers that I talk to are building agents for productivity of their own employees. Like if it's a healthcare productivity of their own doctors, nurses, medical staff. If it's a HR department or HR company, the productivity of their employees in order to maybe service the other employees better with questions regarding their benefits or other more personal things that often we reach out to HR for. So it's it's a lot of companies have started experimenting how can we make our own processes more efficient with that. And then I think the risks is a little bit more contained because it's of course it many things could go wrong there too that that's why we're gonna talk about the tools. But then a lot of folks also have started thinking about it just they can deploy externally. To their audience, but they are taking their time of taking it to production. And that's where we engage quite a lot with them. So it's actually quite nice to see people are excited, but more than they were with traditional machine learning, they're cautious because they know that that power could also kind of like kick back with some risks. And that's where I think we engage the most with customers.
speaker-2: That is fascinating. I love your insights in terms of how you're seeing people deploy and the focus, you know, internal versus external. And I would say that caution is positive, right? We should be cautious. Absolutely. And I also validate the fact that unless you're an expert in the space, it can be a new topic for a lot of folks who are like, hey, this sounds really cool, but we don't know what we don't know. and this is where I really value the work that you and your team does because you have shared knowledge and resources and tools that everybody can use and get them to like, hey, this is a new space, but we have the tools in order to be able to understand like what is our risk profile? How can we measure? So I'm gonna steer us a little bit towards areas of in around measurement and also mitigation. I know that you announced and built I say you, your team announced and built ASSERT which is a new framework that Anybody out there can can use. Can you tell us a little bit more about ASSERT and how people should think about what it is?
speaker-0: Absolutely. Absolutely. So the to the point that you were making, fear often comes from unpreparedness. And as I mentioned, my team is always on a mission to make sure we give you the developers, the tools that could be incorporated seamlessly as a part of your generative AI application or a genetic AI systems lifecycle. So you don't feel like it's an afterthought that, gosh, now I have to go and find all these different things. And Go and measure risks, there's something that is very much part of your CICD to measure and mitigate. So the very first tool that we introduced in the space of measurement was ASSERT as you mentioned. It can be accessed via github.com / responsible AI / assert. And the reason behind this tool or the mission behind this tool is to help you assess your agents, your generative A applications with regards to the niche requirements that you have in mind. Originally when I started in the space of measurements, my team put up a lot of different safety measurements like violence defectorate, harm fair unfairness defect rate, or sexual defectorate, self-harm, this type of thing, jailbreak defectorate. So to what extent application is generating this type of content or is falling for jailbreak attempts. And those generic evaluations were Really useful if you want to get a sense of how your application is doing in production. those are often some very popular dimensions that we see customers care about. like in healthcare, they really want to make sure that there is no response that is suggesting any self-harm to the patient or in is working with an education institutes, they deeply cared about violence or sexual content not being present in the responses. So These measurements help with some generic dimensions. But with ASSERT, one thing that we noticed is people come and say, but I have either some regulations that are very specific or I have some requirements because of the my domain, like I do not want my application to generate legal advice or healthcare institutes would say, I do not want my generative AI applications to generate any medical dosage recommendation. So these type of niche things, we wanted to create a tool that takes this as a human language, natural language, simple words, and turns it into both policies of based on this, what is allowed, what is not allowed. So like in case of a medical dosage, discuss this with your medical professional is allowed, or generically talking about the side effects of a drug is allowed, but get this dose of hyperbrufen is not allowed. So Turning it into permissible and impermissible clauses and policies, generating a data set very specific to the context of your application that now stress tests for this permissible and impermissible policies and passes it to your application, get the responses of your application, and then scores to what extent you kind of complied with that permissible clauses and impermissible clauses and give you a scorecard. So that's pretty much how it works. We collaborated with Microsoft Research and our own core AI research team, generated this pipeline that takes the natural language from you, a requirement from you, turns it into behaviors and policies, measure generates the initial test data, and then passes it to your application, get the responses and score the responses. So your hands-on work with it is very little. Like you bring your requirement, you bring your application, and then you Can check the permissible, impermissible policies and modify them if you want to add to them, if you want to edit what we have generated. And then the rest of it is automated to give you the scorecard. And the good thing about these things is once it's in front of you, you can't miss it. Now you have to think about really fixing it. And and that also gives you a lot more hints of how to do more focused mitigation in a way that it's effective rather than generic trying to improve your application.
speaker-1: Love that. I think something that you said there, which like triggered my brain, is you said the context of your app. Right. And I find this a lot. Okay. So I think I think potentially we have to qualify that properly for the public. And the reason I say that is because I don't think a lot of people who build these things actually truly understand the context of what they're making. I think that like the medical one is interesting. So I actually will tell you a story. I worked with an organization in The UK. Let's put it like this it was more of a test proof of concept type thing, right? So, like I've got to qualify this. And we were building an agent or a set of agents that was potentially going to help people with information on how to come off narcotics. Right. So person is a narcotics user, they need to come off. You can't just do it. It doesn't just happen, right? Like you don't just go, I quit. Because if you know anything about the narcotics industry and like medical around that, people can die from just doing that, right? So there's guidance around like how to do that. But as you know, like that is super sensitive information. You can't just be flinging out medical advice. So there's a bunch of context that needs to be put in there about the person, medical records. There's they're called it a a pa a PIM patient information system. Like there's tons of stuff, right? So all that context is like rolled back up into that thing. We'll call it an app. That then helps provide that person or that group information. And I was like, when you were talking, I'm like, you know what? This could be endless. Like the number of things you have to test and ask it are endless. So I I guess I don't know if there's an answer to this. So please, I promise I'm trying to, I'm not like trying to landmine you here. But like, how do you test for that? How do you know? I mean, it's genius what you guys have done, but how do you know the depth of the context that the thing should have? 'Cause in my brain it's Yeah. It's wild.
speaker-0: Absolutely. And and e exactly when I was mentioning the generic evaluators at the very beginning and the simulators that we had created in order to generate some appropriate test data to measure then for violence or other things, that's what that's where they fell short. That let's say I have a healthcare scenario and I keep asking questions, simulated questions of generic, I don't know. like questions about violence or sexual content or try to jailbreak it, it's n really not gonna probably identify the gaps of that application the way that it should in your case with the UK healthcare system that you were working. So with something like ASSERT, one good thing is you can pass a lot of so there's a YAML file where you can configure and say what you want to measure, but in that YAML file, you also pass the configuration of what that agent is about to begin with. What it is supposed to do, what it's not supposed to do, what tools it has access to, what knowledge base it has access to. So you there there's a little framework that we have created where you pass information about that agent to. That helps us on some fronts, many fronts, in fact. One, to generate test sets. That are very specific to your use case. In fact, they're trying to make your agent call those tools or do manipulations on that data in a way that we would like for you to then realistically test it. And then the content of the test cases that are generated are also gonna be relevant to the context of what you have passed to us and what tools it has access to, what knowledge base in that description. And one thing that we're working on and we're gonna put it in in public hands very soon is the notion of sandboxing. So how to make sure then as a part of your evaluation, things don't go wrong. Because of course part of the evaluation, especially sometimes adversarial evaluation, is for you to stress test things in a negative way. But because you're doing it as the creator, it should be a safe space. And last thing that you want is as a part of your testing mass email is sent or your knowledge base is deleted and wiped without no way back. So We're very conscious of not just bringing the access or the architecture and context of your agents into the testing, but also doing it in a sandboxed environment. So the the harm that can go wrong is reversible and is contained only to that testing scenario and it's not gonna create something in the real world. So what you're saying is absolutely real, that's how you do it with ASSERT and that's why we're working on sandboxing to make sure everything is contained.
speaker-2: Love that. And I would add another flavor to it as well, which I think is a secondary benefit, but one that is also very important. What I love about the sandboxing as well, and with adversarial testing, you're creating the environment where somebody who has to do this testing can do it. And also that gives the organization who's deploying it the contextual this person is doing this because it's part of their role to do the adversarial testing. And the reason I bring it up. Chris and I got into some hot waters with some of the testing that we were doing. It wasn't deployed, but it it still got flagged by different IT systems going like, Hey, what are you doing to this system? 'Cause it's being recorded that your adversary already in testing. It's actually breaking the terms of use. Which correct. That is the point of the testing. So having that sandbox, that controlled environment where you're s this is the role that I play, this is what I'm meant to do and can do it safely, I think is also very important.
speaker-0: That's absolutely right. In fact, some of the customers reached out because we also have an AI red teaming agent that is more like a service deployed inside foundry that folks could use. And that tests for a lot of precanned adversarial attacks across a lot of dimensions and we're in the process of bringing more nuanced ASSERTs like custom requirements to it. And so one of the things that early on we had to do is First of all, make sure that the adversarial model that is being used as a part of that service itself is not getting flagged. Cause there there needs to be a model that is generating a lot of these attack like things. and it's very contained behind the service. No one can access, you know, it's very but like for the customer running it. In a lot of cases, that that itself was getting flagged and we fixed that. And then now this exactly to your point of how to make sure that it's it's very obvious that it's a test environment with the right sandboxing still gives you everything that could go wrong, but without exposing you to any dangers or reputational risks or anything that is irreversible and and and terrible to happen. And also creates this culture of this is part of testing and it's content, don't worry about it. so the technology should definitely follow that.
speaker-2: No, that's fantastic. I love it. and I know we're talking a little bit about the the measurement and custom the customs use cases. I think this is where I get really excited because to your point, i it was fantastic when we introduced like, hey, this is the baseline of if you're gonna release something, these are the type of risks you should be thinking of, you know, prompt injections and and so forth. But as agents evolve, as they get more sophisticated, obviously the the amount of measurement has to be tweaked. So this is really cool. So we let's say we're doing things correctly. I'm an organization. I'm deploying my agents. They're customized. I'm using uncertainty in order in order to measure what are my type of harms, what is my defect rate, you know, what do I need to tweak? How do I mitigate? What do I do next once I have the data?
speaker-0: Yeah, th this is another very great top of mind for folks. Sometimes they come and they tell me, listen, we have run all these measurements and these numbers, for the most part, sometimes they don't have a meaning unless, first of all, they're relative to some tests that you've run before and you can compare. But also, people are very confused of how to fix something that the generative AI application is saying. So that's one challenge. And then the second challenge is. Let's say you have deployed certain mitigations for your agents based on the best practices of your organization. It cannot be easily transferred to my agent. So if a couple of things that we're seeing, folks use meta prompt or system prompt improvements in their agents. Sometimes they add filters and classifiers to their input and output to maybe filter for let's say sexual self-harm, hate, violence, jailbreak, these type of things. they have PII filters. In fact, as a part of Azure AI contest safety, we release a lot of those things that people are using. But we also know that folks are having fleet of agents. So many different agents they're developing. And if I have to talk to Chris, to you, to Ana, to everyone in order to compare notes and Pass the mitigations, it's gonna take a long while and it's also inefficient. There's logic everywhere. It's not translatable. So that's where one of the things that we released in the open source, also at Bield, together with ASSERT, was called agent control spec. The reason why this agent control spec was released was with ASSERT, essentially you can understand what the risks are, but then you can also see you want to see, all right. How can I fix it at scale across my fleet, but also what's happening when my agent is running? Do I have some level of protection on its runtime in order to enforce certain best practices more systematically? So we released agent control spec or ACS as a part to have a portable way of describing your runtime policy. So think about it this way. ACS is having an SDK that gets attached to eight different intervention points of your agent. It could be agent startup, agent shutdown, model input, model output, agent input, agent output, free tool call, or the the thing that goes into the tool call and the output of the tool response. And so what it does is it enables you to apply certain policies against those different intervention points with your known Policy languages like CEDAR or or Rego. So you identify that for all of my agent inputs, I would like to apply these classifier endpoints, these filters. For my tool response, I also want to apply PII removal. Or you can also bring regex, just like freeform deterministic rules that I really want if there is an interaction happening or an attempt to interction, financial interaction of more than $1,000 bring human into the loop. Or it could be also LLM judge. You can apply LLM judge to any of these intervention points and task it with any prompt. Now the point of this is this policy gets compiled and then gets reinforced against your agent. And and so the part the huge part of it is that the host's runtime reinforces these policies in a deterministic way rather than leaving it to the chance of these policies to apply or not. And then you can carry this YAML file, this policy against other agents. So suddenly you can have these YAML files applied to all the agents in your ecosystem. You can also have a hierarchy of YAML files. Maybe one agent has different requirements that some others don't have. So we just wanted to give folks a really light open source, maybe portable package that is not another guardrail classifier, but 'Cause there are so many guardrail systems out there that is really great. But more like a control layer that gives those system a common contract, for the lack of better word, that determines where and when those policies should be invoked and what happens when when those actions happen and and fix it or raise it to the human or transform it or warn people or block it. So it has a lot of kind of like veradics of what it follows in the host runtime. So that's what I really like to talk to people about. That I think sit down with regards to the evaluation results, see what is available to you. It could be some meta prompts that you would like to apply to your agent. It could be some classifiers and endpoints and filters you would like to apply to your agent input or agent output or tool input or tool output or model input, model output. It might be some deterministic rules you want to create because you notice that something is systematically going wrong. It could be that you want to help of LLM judge at a certain point. And then create this portable spec and see apply this portable stick to step spec to as many agents that you feel like is relevant. And then from now on you can have a level of peace of mind that at least there is governance happening systematically in your runtime. And you can build on it by hierarchy of other requirements. So that's something that I think they could find helpful if they're about to go to production and they would like to have a level of protection that cannot be jailbroken or cannot be you know, just like removed or anything in any form or shape.
speaker-1: Yeah, this is wild.
speaker-2: You know, we know we have lost Chris to a mental to a mental rabbit hole when she's like, I am so lost in this in the most beautiful way. I can see you thinking.
speaker-1: Yeah, it's because it's because okay, I'll tell you why I'm thinking. It's because I feel like when I look at the way when people start deploying like multi agent systems, all right, they it's normally within a singular ecosystem that they do this. So that and when I say singular ecosystem, I don't mean a tenants. I mean the agent itself is one ecosystem. So what I mean by that is I can go, Okay, I'm gonna put that agent into production. It's one little agent, right? So you can test that. It's easy to test it because you're not dealing with I mean it's not easy, but you're dealing with far less complexities, right? But when you have agent A talking to agent B talking to agent C, so like take that little trifecta, you have to test incoming, outgoing, on tool. Like there's a bunch of things that you have to understand. And it's not always just as simple as like, this agent's gonna access an MCP server. Like what are the outputs when it does that? How do you manage that whole set of guardrails and give the have the control plane to do that? So this is like this is something that blows my mind because I think people think that like red teaming is the whole thing when it comes down to, I'm just gonna test my agents. But actually, you're testing at like an interaction layer, which is way more difficult. And that's why I was thinking I'm like picturing this whole thing in my head of like how I would I need I need the understanding for me. Like I want to see it. And I love I love it. Okay, let me just give you an example. The first time I ever saw agent to agent working, like one agent doing something and physically passing to another agent, I was like, I get this stuff now. Like this makes sense to me. Then all that went through my head, and this was three years ago. I'm like, How are we gonna govern this? And it sounds like you have the tools to do that. Like it sounds like you have the framework, you have the tools.
speaker-0: Yeah, it this is actually a really, really great point because wha when we were talking about ACS, one thing that we were very, very conscious was we don't believe that every single company has to invent their own governance ecosystem and mechanism. This is just not gonna scale. And the same way that A2A and MCP standardize how agents talk to each other or how agents talk and interact with tools. We wanted to have a standard out there that enables you to kind of like express and enforce your controls across the ecosystem. Now, to your point, when like when when you define something with agent control spec, the actual policy interventions that it has, it can apply to when the agent is receiving the input. It can apply before it calls the MCP tool. after it receives the response from the MCP tool, so that the tool response, it can govern the final output of your agent. If there's a model, which always there's a model in between as a part of that agent ecosystem, the input of it, the output of it. So we really wanted to have a story of assert helps me understand and identify risks of before I put it out there, Does this agent behave the way that I expect with my niche requirements? And then agent control spec helps you answer. Now, as my agent ecosystem scales and grows, how do I consistently control what all of these agents are allowed to do? And then have a way, which is we're in the process of bringing it into the foundry so that you can also have an ecosystem of or a dashboard of. Kind of understanding, these are all my agents in my fleet, which we have as a part of foundry control plane. You can visualize that. But then these are all the policies that I have applied to these agents. This is how each agent is doing with regards to this policy production. And this is now let's think about assert post-production. I keep continuously monitoring that agent or based on any evaluation post-production. And I can get alerts of if something is going off the rail, I can jump on it and quickly take an action before it creates a disaster. So I really love to think about this ecosystem and see how can we help people who are building soon gonna build hundred agents, thousands of agents. And all to your point, all these agents in track with each other. How can we help them scale without being also vendor locked, honestly? That's why we build build both assert and Agent control spec as a part of agent governance toolkit in the open for folks to be able to see exactly what logic went into it, create their own PRs and forks of it for their organization and give us feedback.
speaker-2: I love it. I think that's so smart. There's two things there. I I love the and I didn't put together the end-to-end thread that you just explained. And now that you say it, it makes so much sense because it is that connective, like, how do I monitor, how do I measure, how do I mitigate, how do I make sure that I can intervene where I need to intervene? and that continuous monitoring and testing, which honestly, bias view, but everybody should do because nothing is ever static, especially in this world that we live in and with technology evolving. So I love that connected issue of this is how you think of it and this is how you think of sidetrack. Chris and I almost had this conversation of like the new role is managing the agent. If you're like the manager of the agents, this is exactly in your toolkit of being able to do that at scale with irrespective of how many hundreds, thousands of agents your organization has. The other thing that I find super, super interesting is the I don't want to say model because it's not even model, it's the infrastructure agnostic, right? Of you can be running whatever you're running. And more so we're we're seeing it with clients and people have multiple things running in their in their infrastructure organization. It's not like one vendor that they're running across their tech stack or whatever it is. It's increasingly multi vendor. especially when it comes to agents. So how do you how do you able to operate across all of them without having to go like Well for this vendor I have to do this, for this other vendor I have to do this. It's it's just not scalable. So I I love I love
speaker-0: I love that you say that because originally when we were building our evaluations and mitigations a lot in inside the foundry, of course, we always empower our foundry users, those who are building agents, to have access to the best and latest and greatest evaluations and also mitigations. But a lot of these things started getting enabled as a service, starting with our Azure AI content safety and then now AI Red Teaming agent. And then at some point we were like, okay. With ASSERT and ACS, we would love to have as many converters as possible to support different agentic frameworks. So right off the bat, it both of them support pretty much every single agent government that you can imagine, ASSERT, including custom ones. So I'm working with customers all over. They're using Langchain and and of those who are on Foundry and those who are using custom agentic framework infrastructure. So the beauty of it is it can apply to all. And I feel like that's how we make progress going to the earlier part of this conversation that these things are new and with novelty sometimes comes fear, but fear is often lack of preparedness. So so that's exactly how we wanted to make sure that that fear goes away with access, regardless of where you're building and how you're building.
speaker-2: gonna steal that regarding fear and how
speaker-0: Yeah.
speaker-2: we think of deployments and how can we think of safety doing it. and it what you just said Manush it also made me think back to the point of governance and that's how we enable governments, true governance, where you have those tools and you have them clearly communicated and populated through the through the likes of your organization. And I think what I extra love about I'm gonna call it policy. Let's say that we with the framework is like, hey, you know, you have different policies. We also know policy is not static. So being able to kind of trigger those changes in terms of like, well, this is the policy for my agents, but I need to update that policy at scale, this is where I think the the control comes in and it's super helpful because policy evolves. And I use that loosely. I don't just mean from a legislation perspective. I also mean that organization perspective, your rules in engagement change. Absolutely. Before I go to my last my my usual homework question, Chris. Anything you wanted to ask or add?
speaker-1: So the way I see it, it's like trustworthy AI is the intersection of le legal standing, okay, human ethics and technical best practice. And I feel like the things you were talking about involve all of those, right? Like you're talking about how do you ethically help somebody through the red teaming phase. You're talking about like the legalities around like procedure and process. And then you've got the technical best practice around the tools that you all have built that help kind of wrap this together. And I feel like in my brain, I want I want to get my hands on what you guys have built, ASAP, because I believe that as somebody who really enjoys building out these tools and built and working in the AI space, like, I believe it'll help me immediately. So selfishly, I I really want to see what you guys have got.
speaker-0: Would love for you to try it. Yes. And for anyone who is listening to this or watching this, please get your hands on both the ASSERT and Agent Control Spec. reach out to me on LinkedIn if I can be of any help. and and send us PRs, your your issues and anything with it, or feedback. we would love to grow it with you as a part of the community and and we would love to also make sure that this works perfectly for you. So if that's not it, I would love to know. what else we can do and add as the next phase of planning for this product. So thank you for letting me speak about them and I'm really, really grateful I had this very thoughtful conversation with you.
speaker-2: my goodness, the pleasure was ours, Mehrnoosh. we learned so much. I always learn listening to you. So thank you for being so generous with your time and expertise. And folks, if you if you don't follow Mehrnoosh on LinkedIn, please do. because I can promise you post all the time some very interesting links and articles. So thank you for that. And folks, thank you so much for joining us for another episode in Human in the Loop. We'll see you next time.
speaker-0: Appreciate it. Thank you.
speaker-0: Yeah, absolutely. Hello, everyone. It's a pleasure to be here. I'm Mehrnoosh Sameki I'm currently with Microsoft Core AI and I lead the developer experiences and tools within the Response. Personally and as a team, we're on a mission to make sure that this buzzword of trustworthy AI and responsible AI is something that can be very easily operationalized in the machine learning and generative AI lifecycle for any developer building with AI.
speaker-2: gonna jump in here because I'm getting excited. Hey folks and hi friends. thank you for tuning in today. Mehrnoosh I was smiling as you're talking about trustworthy AI and responsible AI and kind of getting it more than to be a buzzword and that's something that Chris and I talk a lot about of hey this is actually not just something to flag of yeah it's there somewhere in the background and forget about, it's critical. can you give us because I know you've been in this space for before it was cool. can you give us some insights into just a differentiator from that perspective? Because we know we should not have technology without those ingredients. But I'm curious of your take and where you're seeing kind of the real life impact of if you break it it's really gonna be broken.
speaker-0: Absolutely. So when I started in the space of trustworth AI responsible AI, we were dealing more with traditional machine learning, talking about classifiers, regression models, time series, et cetera. Back then the concern was very much around explainability of these machine learning models based on what features, what factors they're making their predictions. Or sometimes it was also about the fairness of the prediction. Let's say if it has classified someone to have a high chance of paying off the loan and hence that person might get the loan because of that prediction. We wanted to know if the machine learning model is systematically giving a a more favorable outcome to people of certain ethnicities, age group, gender groups, et cetera, et cetera. So and at some point we shifted to more of an error analysis, trying to understand how errors are embedded in the machine learning responses, if they're systematically a higher Or groups that are getting systematically higher error rates than other groups, and how can we fix it? When the generative AI boom happened, and now with AI agents, the conversation completely shifted to a new set of concerns that we have. As you know, with AI agents, they're incredibly autonomous. They can take actions on your behalf, they can interact with databases, interact with a bunch of internal and external tools. They sometimes have access to data sets that only us as employees or as certain high privileged users might have. And so it started becoming incredibly important to make sure we understand how these agents are navigating what they have access to and the tools that they have access to and all these different actions that they're making. What are these actions? What is allowed and what is not allowed? how can we give them a little bit of a identity so we can stop them, quarantine them, and and maybe restart them and kind of control them as a virtual entity. And then the other thing that was incredibly important is now with this huge autonomous action, a range of autonomous actions that they can make, can we really understand and measure with regards to certain regulations in mind or requirements of our company in mind? How are they really doing the job? Are they really adhering to the task that they're supposed to do? Are they doing any prohibited actions? That is absolutely no no. So with more power of these tools and agents and generative AI applications comes a lot of new, I think, heightened risks that we need to really make sure we're putting the right evaluation and also protection in place.
speaker-1: That's wild. So I mean, it it is a little scary, right? Like I see, you know, Ioana and I have had some interesting experiences with some deployments. We'll leave it at that. But I guess the fear the fear that I have is that like we as a society haven't quite grasped like what what agents actually can do, you know, from an end to end perspective. Like I think we're still trying to we're still thinking that they're these little automations that kick around in the background and do stuff. But actually, I feel like when I look at what companies are doing and Microsoft partners and the the ecosystem, I'm like, we're still very much like on the like five to ten percent margin of what these things can really truly do. Right. Do you see like in your role, do you see that or do you see like agents being used for a lot more wider tasks and things?
speaker-0: Yeah, so first of all, I see that folks are still very slow to take those agents into production. So everyone is being really careful about the use of agents at the moment. They are very much in the stage of development pre production. And that's where a a lot of like tools and and infrastructure that, for example, my team builds come handy. But you're you're absolutely right. I think People know this is a very incredibly powerful tool. And they have started experimenting with it. They have their doubts, not in terms of the technology capability, but in doubts of what could go wrong. And they're taking their time at this stage in order to more characterize what those potential risks could be before putting it in production. And a lot of folks have started from I'd say internal processes. So a lot of customers that I talk to are building agents for productivity of their own employees. Like if it's a healthcare productivity of their own doctors, nurses, medical staff. If it's a HR department or HR company, the productivity of their employees in order to maybe service the other employees better with questions regarding their benefits or other more personal things that often we reach out to HR for. So it's it's a lot of companies have started experimenting how can we make our own processes more efficient with that. And then I think the risks is a little bit more contained because it's of course it many things could go wrong there too that that's why we're gonna talk about the tools. But then a lot of folks also have started thinking about it just they can deploy externally. To their audience, but they are taking their time of taking it to production. And that's where we engage quite a lot with them. So it's actually quite nice to see people are excited, but more than they were with traditional machine learning, they're cautious because they know that that power could also kind of like kick back with some risks. And that's where I think we engage the most with customers.
speaker-2: That is fascinating. I love your insights in terms of how you're seeing people deploy and the focus, you know, internal versus external. And I would say that caution is positive, right? We should be cautious. Absolutely. And I also validate the fact that unless you're an expert in the space, it can be a new topic for a lot of folks who are like, hey, this sounds really cool, but we don't know what we don't know. and this is where I really value the work that you and your team does because you have shared knowledge and resources and tools that everybody can use and get them to like, hey, this is a new space, but we have the tools in order to be able to understand like what is our risk profile? How can we measure? So I'm gonna steer us a little bit towards areas of in around measurement and also mitigation. I know that you announced and built I say you, your team announced and built ASSERT which is a new framework that Anybody out there can can use. Can you tell us a little bit more about ASSERT and how people should think about what it is?
speaker-0: Absolutely. Absolutely. So the to the point that you were making, fear often comes from unpreparedness. And as I mentioned, my team is always on a mission to make sure we give you the developers, the tools that could be incorporated seamlessly as a part of your generative AI application or a genetic AI systems lifecycle. So you don't feel like it's an afterthought that, gosh, now I have to go and find all these different things. And Go and measure risks, there's something that is very much part of your CICD to measure and mitigate. So the very first tool that we introduced in the space of measurement was ASSERT as you mentioned. It can be accessed via github.com / responsible AI / assert. And the reason behind this tool or the mission behind this tool is to help you assess your agents, your generative A applications with regards to the niche requirements that you have in mind. Originally when I started in the space of measurements, my team put up a lot of different safety measurements like violence defectorate, harm fair unfairness defect rate, or sexual defectorate, self-harm, this type of thing, jailbreak defectorate. So to what extent application is generating this type of content or is falling for jailbreak attempts. And those generic evaluations were Really useful if you want to get a sense of how your application is doing in production. those are often some very popular dimensions that we see customers care about. like in healthcare, they really want to make sure that there is no response that is suggesting any self-harm to the patient or in is working with an education institutes, they deeply cared about violence or sexual content not being present in the responses. So These measurements help with some generic dimensions. But with ASSERT, one thing that we noticed is people come and say, but I have either some regulations that are very specific or I have some requirements because of the my domain, like I do not want my application to generate legal advice or healthcare institutes would say, I do not want my generative AI applications to generate any medical dosage recommendation. So these type of niche things, we wanted to create a tool that takes this as a human language, natural language, simple words, and turns it into both policies of based on this, what is allowed, what is not allowed. So like in case of a medical dosage, discuss this with your medical professional is allowed, or generically talking about the side effects of a drug is allowed, but get this dose of hyperbrufen is not allowed. So Turning it into permissible and impermissible clauses and policies, generating a data set very specific to the context of your application that now stress tests for this permissible and impermissible policies and passes it to your application, get the responses of your application, and then scores to what extent you kind of complied with that permissible clauses and impermissible clauses and give you a scorecard. So that's pretty much how it works. We collaborated with Microsoft Research and our own core AI research team, generated this pipeline that takes the natural language from you, a requirement from you, turns it into behaviors and policies, measure generates the initial test data, and then passes it to your application, get the responses and score the responses. So your hands-on work with it is very little. Like you bring your requirement, you bring your application, and then you Can check the permissible, impermissible policies and modify them if you want to add to them, if you want to edit what we have generated. And then the rest of it is automated to give you the scorecard. And the good thing about these things is once it's in front of you, you can't miss it. Now you have to think about really fixing it. And and that also gives you a lot more hints of how to do more focused mitigation in a way that it's effective rather than generic trying to improve your application.
speaker-1: Love that. I think something that you said there, which like triggered my brain, is you said the context of your app. Right. And I find this a lot. Okay. So I think I think potentially we have to qualify that properly for the public. And the reason I say that is because I don't think a lot of people who build these things actually truly understand the context of what they're making. I think that like the medical one is interesting. So I actually will tell you a story. I worked with an organization in The UK. Let's put it like this it was more of a test proof of concept type thing, right? So, like I've got to qualify this. And we were building an agent or a set of agents that was potentially going to help people with information on how to come off narcotics. Right. So person is a narcotics user, they need to come off. You can't just do it. It doesn't just happen, right? Like you don't just go, I quit. Because if you know anything about the narcotics industry and like medical around that, people can die from just doing that, right? So there's guidance around like how to do that. But as you know, like that is super sensitive information. You can't just be flinging out medical advice. So there's a bunch of context that needs to be put in there about the person, medical records. There's they're called it a a pa a PIM patient information system. Like there's tons of stuff, right? So all that context is like rolled back up into that thing. We'll call it an app. That then helps provide that person or that group information. And I was like, when you were talking, I'm like, you know what? This could be endless. Like the number of things you have to test and ask it are endless. So I I guess I don't know if there's an answer to this. So please, I promise I'm trying to, I'm not like trying to landmine you here. But like, how do you test for that? How do you know? I mean, it's genius what you guys have done, but how do you know the depth of the context that the thing should have? 'Cause in my brain it's Yeah. It's wild.
speaker-0: Absolutely. And and e exactly when I was mentioning the generic evaluators at the very beginning and the simulators that we had created in order to generate some appropriate test data to measure then for violence or other things, that's what that's where they fell short. That let's say I have a healthcare scenario and I keep asking questions, simulated questions of generic, I don't know. like questions about violence or sexual content or try to jailbreak it, it's n really not gonna probably identify the gaps of that application the way that it should in your case with the UK healthcare system that you were working. So with something like ASSERT, one good thing is you can pass a lot of so there's a YAML file where you can configure and say what you want to measure, but in that YAML file, you also pass the configuration of what that agent is about to begin with. What it is supposed to do, what it's not supposed to do, what tools it has access to, what knowledge base it has access to. So you there there's a little framework that we have created where you pass information about that agent to. That helps us on some fronts, many fronts, in fact. One, to generate test sets. That are very specific to your use case. In fact, they're trying to make your agent call those tools or do manipulations on that data in a way that we would like for you to then realistically test it. And then the content of the test cases that are generated are also gonna be relevant to the context of what you have passed to us and what tools it has access to, what knowledge base in that description. And one thing that we're working on and we're gonna put it in in public hands very soon is the notion of sandboxing. So how to make sure then as a part of your evaluation, things don't go wrong. Because of course part of the evaluation, especially sometimes adversarial evaluation, is for you to stress test things in a negative way. But because you're doing it as the creator, it should be a safe space. And last thing that you want is as a part of your testing mass email is sent or your knowledge base is deleted and wiped without no way back. So We're very conscious of not just bringing the access or the architecture and context of your agents into the testing, but also doing it in a sandboxed environment. So the the harm that can go wrong is reversible and is contained only to that testing scenario and it's not gonna create something in the real world. So what you're saying is absolutely real, that's how you do it with ASSERT and that's why we're working on sandboxing to make sure everything is contained.
speaker-2: Love that. And I would add another flavor to it as well, which I think is a secondary benefit, but one that is also very important. What I love about the sandboxing as well, and with adversarial testing, you're creating the environment where somebody who has to do this testing can do it. And also that gives the organization who's deploying it the contextual this person is doing this because it's part of their role to do the adversarial testing. And the reason I bring it up. Chris and I got into some hot waters with some of the testing that we were doing. It wasn't deployed, but it it still got flagged by different IT systems going like, Hey, what are you doing to this system? 'Cause it's being recorded that your adversary already in testing. It's actually breaking the terms of use. Which correct. That is the point of the testing. So having that sandbox, that controlled environment where you're s this is the role that I play, this is what I'm meant to do and can do it safely, I think is also very important.
speaker-0: That's absolutely right. In fact, some of the customers reached out because we also have an AI red teaming agent that is more like a service deployed inside foundry that folks could use. And that tests for a lot of precanned adversarial attacks across a lot of dimensions and we're in the process of bringing more nuanced ASSERTs like custom requirements to it. And so one of the things that early on we had to do is First of all, make sure that the adversarial model that is being used as a part of that service itself is not getting flagged. Cause there there needs to be a model that is generating a lot of these attack like things. and it's very contained behind the service. No one can access, you know, it's very but like for the customer running it. In a lot of cases, that that itself was getting flagged and we fixed that. And then now this exactly to your point of how to make sure that it's it's very obvious that it's a test environment with the right sandboxing still gives you everything that could go wrong, but without exposing you to any dangers or reputational risks or anything that is irreversible and and and terrible to happen. And also creates this culture of this is part of testing and it's content, don't worry about it. so the technology should definitely follow that.
speaker-2: No, that's fantastic. I love it. and I know we're talking a little bit about the the measurement and custom the customs use cases. I think this is where I get really excited because to your point, i it was fantastic when we introduced like, hey, this is the baseline of if you're gonna release something, these are the type of risks you should be thinking of, you know, prompt injections and and so forth. But as agents evolve, as they get more sophisticated, obviously the the amount of measurement has to be tweaked. So this is really cool. So we let's say we're doing things correctly. I'm an organization. I'm deploying my agents. They're customized. I'm using uncertainty in order in order to measure what are my type of harms, what is my defect rate, you know, what do I need to tweak? How do I mitigate? What do I do next once I have the data?
speaker-0: Yeah, th this is another very great top of mind for folks. Sometimes they come and they tell me, listen, we have run all these measurements and these numbers, for the most part, sometimes they don't have a meaning unless, first of all, they're relative to some tests that you've run before and you can compare. But also, people are very confused of how to fix something that the generative AI application is saying. So that's one challenge. And then the second challenge is. Let's say you have deployed certain mitigations for your agents based on the best practices of your organization. It cannot be easily transferred to my agent. So if a couple of things that we're seeing, folks use meta prompt or system prompt improvements in their agents. Sometimes they add filters and classifiers to their input and output to maybe filter for let's say sexual self-harm, hate, violence, jailbreak, these type of things. they have PII filters. In fact, as a part of Azure AI contest safety, we release a lot of those things that people are using. But we also know that folks are having fleet of agents. So many different agents they're developing. And if I have to talk to Chris, to you, to Ana, to everyone in order to compare notes and Pass the mitigations, it's gonna take a long while and it's also inefficient. There's logic everywhere. It's not translatable. So that's where one of the things that we released in the open source, also at Bield, together with ASSERT, was called agent control spec. The reason why this agent control spec was released was with ASSERT, essentially you can understand what the risks are, but then you can also see you want to see, all right. How can I fix it at scale across my fleet, but also what's happening when my agent is running? Do I have some level of protection on its runtime in order to enforce certain best practices more systematically? So we released agent control spec or ACS as a part to have a portable way of describing your runtime policy. So think about it this way. ACS is having an SDK that gets attached to eight different intervention points of your agent. It could be agent startup, agent shutdown, model input, model output, agent input, agent output, free tool call, or the the thing that goes into the tool call and the output of the tool response. And so what it does is it enables you to apply certain policies against those different intervention points with your known Policy languages like CEDAR or or Rego. So you identify that for all of my agent inputs, I would like to apply these classifier endpoints, these filters. For my tool response, I also want to apply PII removal. Or you can also bring regex, just like freeform deterministic rules that I really want if there is an interaction happening or an attempt to interction, financial interaction of more than $1,000 bring human into the loop. Or it could be also LLM judge. You can apply LLM judge to any of these intervention points and task it with any prompt. Now the point of this is this policy gets compiled and then gets reinforced against your agent. And and so the part the huge part of it is that the host's runtime reinforces these policies in a deterministic way rather than leaving it to the chance of these policies to apply or not. And then you can carry this YAML file, this policy against other agents. So suddenly you can have these YAML files applied to all the agents in your ecosystem. You can also have a hierarchy of YAML files. Maybe one agent has different requirements that some others don't have. So we just wanted to give folks a really light open source, maybe portable package that is not another guardrail classifier, but 'Cause there are so many guardrail systems out there that is really great. But more like a control layer that gives those system a common contract, for the lack of better word, that determines where and when those policies should be invoked and what happens when when those actions happen and and fix it or raise it to the human or transform it or warn people or block it. So it has a lot of kind of like veradics of what it follows in the host runtime. So that's what I really like to talk to people about. That I think sit down with regards to the evaluation results, see what is available to you. It could be some meta prompts that you would like to apply to your agent. It could be some classifiers and endpoints and filters you would like to apply to your agent input or agent output or tool input or tool output or model input, model output. It might be some deterministic rules you want to create because you notice that something is systematically going wrong. It could be that you want to help of LLM judge at a certain point. And then create this portable spec and see apply this portable stick to step spec to as many agents that you feel like is relevant. And then from now on you can have a level of peace of mind that at least there is governance happening systematically in your runtime. And you can build on it by hierarchy of other requirements. So that's something that I think they could find helpful if they're about to go to production and they would like to have a level of protection that cannot be jailbroken or cannot be you know, just like removed or anything in any form or shape.
speaker-1: Yeah, this is wild.
speaker-2: You know, we know we have lost Chris to a mental to a mental rabbit hole when she's like, I am so lost in this in the most beautiful way. I can see you thinking.
speaker-1: Yeah, it's because it's because okay, I'll tell you why I'm thinking. It's because I feel like when I look at the way when people start deploying like multi agent systems, all right, they it's normally within a singular ecosystem that they do this. So that and when I say singular ecosystem, I don't mean a tenants. I mean the agent itself is one ecosystem. So what I mean by that is I can go, Okay, I'm gonna put that agent into production. It's one little agent, right? So you can test that. It's easy to test it because you're not dealing with I mean it's not easy, but you're dealing with far less complexities, right? But when you have agent A talking to agent B talking to agent C, so like take that little trifecta, you have to test incoming, outgoing, on tool. Like there's a bunch of things that you have to understand. And it's not always just as simple as like, this agent's gonna access an MCP server. Like what are the outputs when it does that? How do you manage that whole set of guardrails and give the have the control plane to do that? So this is like this is something that blows my mind because I think people think that like red teaming is the whole thing when it comes down to, I'm just gonna test my agents. But actually, you're testing at like an interaction layer, which is way more difficult. And that's why I was thinking I'm like picturing this whole thing in my head of like how I would I need I need the understanding for me. Like I want to see it. And I love I love it. Okay, let me just give you an example. The first time I ever saw agent to agent working, like one agent doing something and physically passing to another agent, I was like, I get this stuff now. Like this makes sense to me. Then all that went through my head, and this was three years ago. I'm like, How are we gonna govern this? And it sounds like you have the tools to do that. Like it sounds like you have the framework, you have the tools.
speaker-0: Yeah, it this is actually a really, really great point because wha when we were talking about ACS, one thing that we were very, very conscious was we don't believe that every single company has to invent their own governance ecosystem and mechanism. This is just not gonna scale. And the same way that A2A and MCP standardize how agents talk to each other or how agents talk and interact with tools. We wanted to have a standard out there that enables you to kind of like express and enforce your controls across the ecosystem. Now, to your point, when like when when you define something with agent control spec, the actual policy interventions that it has, it can apply to when the agent is receiving the input. It can apply before it calls the MCP tool. after it receives the response from the MCP tool, so that the tool response, it can govern the final output of your agent. If there's a model, which always there's a model in between as a part of that agent ecosystem, the input of it, the output of it. So we really wanted to have a story of assert helps me understand and identify risks of before I put it out there, Does this agent behave the way that I expect with my niche requirements? And then agent control spec helps you answer. Now, as my agent ecosystem scales and grows, how do I consistently control what all of these agents are allowed to do? And then have a way, which is we're in the process of bringing it into the foundry so that you can also have an ecosystem of or a dashboard of. Kind of understanding, these are all my agents in my fleet, which we have as a part of foundry control plane. You can visualize that. But then these are all the policies that I have applied to these agents. This is how each agent is doing with regards to this policy production. And this is now let's think about assert post-production. I keep continuously monitoring that agent or based on any evaluation post-production. And I can get alerts of if something is going off the rail, I can jump on it and quickly take an action before it creates a disaster. So I really love to think about this ecosystem and see how can we help people who are building soon gonna build hundred agents, thousands of agents. And all to your point, all these agents in track with each other. How can we help them scale without being also vendor locked, honestly? That's why we build build both assert and Agent control spec as a part of agent governance toolkit in the open for folks to be able to see exactly what logic went into it, create their own PRs and forks of it for their organization and give us feedback.
speaker-2: I love it. I think that's so smart. There's two things there. I I love the and I didn't put together the end-to-end thread that you just explained. And now that you say it, it makes so much sense because it is that connective, like, how do I monitor, how do I measure, how do I mitigate, how do I make sure that I can intervene where I need to intervene? and that continuous monitoring and testing, which honestly, bias view, but everybody should do because nothing is ever static, especially in this world that we live in and with technology evolving. So I love that connected issue of this is how you think of it and this is how you think of sidetrack. Chris and I almost had this conversation of like the new role is managing the agent. If you're like the manager of the agents, this is exactly in your toolkit of being able to do that at scale with irrespective of how many hundreds, thousands of agents your organization has. The other thing that I find super, super interesting is the I don't want to say model because it's not even model, it's the infrastructure agnostic, right? Of you can be running whatever you're running. And more so we're we're seeing it with clients and people have multiple things running in their in their infrastructure organization. It's not like one vendor that they're running across their tech stack or whatever it is. It's increasingly multi vendor. especially when it comes to agents. So how do you how do you able to operate across all of them without having to go like Well for this vendor I have to do this, for this other vendor I have to do this. It's it's just not scalable. So I I love I love
speaker-0: I love that you say that because originally when we were building our evaluations and mitigations a lot in inside the foundry, of course, we always empower our foundry users, those who are building agents, to have access to the best and latest and greatest evaluations and also mitigations. But a lot of these things started getting enabled as a service, starting with our Azure AI content safety and then now AI Red Teaming agent. And then at some point we were like, okay. With ASSERT and ACS, we would love to have as many converters as possible to support different agentic frameworks. So right off the bat, it both of them support pretty much every single agent government that you can imagine, ASSERT, including custom ones. So I'm working with customers all over. They're using Langchain and and of those who are on Foundry and those who are using custom agentic framework infrastructure. So the beauty of it is it can apply to all. And I feel like that's how we make progress going to the earlier part of this conversation that these things are new and with novelty sometimes comes fear, but fear is often lack of preparedness. So so that's exactly how we wanted to make sure that that fear goes away with access, regardless of where you're building and how you're building.
speaker-2: gonna steal that regarding fear and how
speaker-0: Yeah.
speaker-2: we think of deployments and how can we think of safety doing it. and it what you just said Manush it also made me think back to the point of governance and that's how we enable governments, true governance, where you have those tools and you have them clearly communicated and populated through the through the likes of your organization. And I think what I extra love about I'm gonna call it policy. Let's say that we with the framework is like, hey, you know, you have different policies. We also know policy is not static. So being able to kind of trigger those changes in terms of like, well, this is the policy for my agents, but I need to update that policy at scale, this is where I think the the control comes in and it's super helpful because policy evolves. And I use that loosely. I don't just mean from a legislation perspective. I also mean that organization perspective, your rules in engagement change. Absolutely. Before I go to my last my my usual homework question, Chris. Anything you wanted to ask or add?
speaker-1: So the way I see it, it's like trustworthy AI is the intersection of le legal standing, okay, human ethics and technical best practice. And I feel like the things you were talking about involve all of those, right? Like you're talking about how do you ethically help somebody through the red teaming phase. You're talking about like the legalities around like procedure and process. And then you've got the technical best practice around the tools that you all have built that help kind of wrap this together. And I feel like in my brain, I want I want to get my hands on what you guys have built, ASAP, because I believe that as somebody who really enjoys building out these tools and built and working in the AI space, like, I believe it'll help me immediately. So selfishly, I I really want to see what you guys have got.
speaker-0: Would love for you to try it. Yes. And for anyone who is listening to this or watching this, please get your hands on both the ASSERT and Agent Control Spec. reach out to me on LinkedIn if I can be of any help. and and send us PRs, your your issues and anything with it, or feedback. we would love to grow it with you as a part of the community and and we would love to also make sure that this works perfectly for you. So if that's not it, I would love to know. what else we can do and add as the next phase of planning for this product. So thank you for letting me speak about them and I'm really, really grateful I had this very thoughtful conversation with you.
speaker-2: my goodness, the pleasure was ours, Mehrnoosh. we learned so much. I always learn listening to you. So thank you for being so generous with your time and expertise. And folks, if you if you don't follow Mehrnoosh on LinkedIn, please do. because I can promise you post all the time some very interesting links and articles. So thank you for that. And folks, thank you so much for joining us for another episode in Human in the Loop. We'll see you next time.
speaker-0: Appreciate it. Thank you.
