РАСШИФРОВКА 🔗
Basil Chatha: Guys, thanks for coming. So today's topic is going to be on evals. You guys have probably heard of them. I feel like everyone talks about evals, but there's not a lot of content about them online. So I thought, yeah, let's bring in some experts and let's have a chat and hopefully we can learn something. So yeah, do you guys wanna introduce yourselves and we can get started? Sure. I can go first. Hey everyone, I'm Liam. I'm a deployed engineer here at Link Chain. I joined Link Chain semi earlier on, about over a year ago, I was employee number sixty. We're now at four hundred now. ⁓ and what I do is I work with customers every day. ⁓ to help them build evals and build agents. Everyone, thanks for having me. My name is Braden. I'm a VP at Mercore. I help run our enterprise business there. This includes helping companies optimize their internal and external agents. We have a Ford Deployed Engineering team which does end-to-end agent builds, including the evals and verifiers for those companies. ⁓ hi everyone. I'm Samya. So I was I'm leading product at Galileo. We are a series B startup helping ⁓ Fortune five hundred enterprises with with evals, helping them build more reliable AI and agents. And ⁓ we actually just got acquired by Cisco two, three months ago. So now leading a bigger product proc portfolio there. And yeah, before this, I also founded my own AI application startup, Poist, ⁓ which had an acquisition with DeepGram. So yeah, I've been in AI space for some time. Hello everyone. My name is Lotte. I do developer experience at Langfuse. ⁓ we've also been acquired a couple of months ago by Clickhouse. My day to day is a lot of talking to users who are also setting up evals. I run and maintain our Langfuse skills. For help to help with that. And I'm also writing our academy. Good manual. Sounds like we all got acquired recently. So my name is Emmanuel. I'm director of engineering at CoreV. So you may know CoV as a GPU sort of cloud data center company. But about a year and a half ago, they acquired Wis and Biases, which you may know for their experiment tracking platform. With Biases also does agent observability observability, including eval's online monitoring and so on. Before we define evals, Liam, so like Langchain wrote this blog post where you guys talk about how like eighty-nine of teams like have some kind of like agent observability like built out, but only like a third of them actually do anything with the observability that they have set up. Could you maybe talk about that? Like why why are like why is everyone setting it up? But like no one's actually doing anything with traces, with anything that's set up. Could you give some context as to like what type of evals you you're talking about like online evals or offline evals? No no no so what I'm saying is like people are setting up agent observability. Like 90% of companies are setting that up, but they're not running any like 50% of them are running offline evals and
Load full transcript
like 30% of them are running online evals. Yeah. So they traces but they don't have evals. Yeah, yeah. Yeah. The the answer to that is because evals are hard. Right. So there's a lot of things that go into like creating evaluations. Just like what what do you actually want your agent to do? And then how do you go about defining that? And since like LMs are non-deterministic, you know, that's not as regular. It's just like writing a Python script to to confirm that. So there's a lot of like techniques you need to do to monitor your production systems or your offline evals, whether you're you you're making change to a harness, like adding a tool, changing your prompt. Then I have confidence, like once you put that edge into production, you're ready to go. So I would say the number one barrier is just like knowing how to create evals and like to like how do you gather data to like create these good benchmarks and run good evals against your data sets? All right. Let's start with maybe like Talking about traces. So what is a trace? Why do we use traces as kind of like the fundamental unit of evals? Yeah. So I think of a trace as every single decision that the LLM makes along the way or the agent makes along the way from going from into input to output. So that is like all of your LM calls whenever the model makes a decision, whatever tools it calls, whenever it writes to a file, whenever it calls another agent. That's everything that's contained within the trace. And this is just a nice view to help you determine like what's going on in your agent under the hood. You know, once a user puts in that question, what is actually happening to take you from point A to point B? Maybe somebody else can take the next one. So what are offline evals? And then we'll go into online evals next. Yeah. So you think about running an agent, you can inspect it at runtime, which is online evals, which we'll talk about in a minute, but you may also want to have a benchmark suite for an agent that you run offline, sort of a more extensive test suite. We Mercore talk about evals as being sort of the PRDs for agents. Like if you have a good, robust set of offline evals, which you can run against an agent, you can almost hill climb the behavior of that agent, use this to do things like identify the frontier of model efficiency. So at its most basic, a test suite of offline evals are a set of prompt output responses with grading, which you can use every time an agent is invoked. To determine its performance. And then you can swap in models. You can change the orchestration harness and plot out how the performance changes as you tweak those variables. So this is both a quality checking measure and also importantly for a lot of the customers we serve, a model routing technique as well. All right. So next, ⁓ what are online evals? Yeah, I can take that. I mean, ⁓ maybe just taking a step back in terms of what is evals, right? Like I think ⁓ a lot of us might be familiar with non-AI apps, like traditional software apps. We used to do How do you make sure our software is working, right? We used to do testing of it. So you used to define test cases of like, okay, if this happens, this should happen. You would test that software before launching it. Now, similarly, for an AI agent, evals are are those tests. But what becomes complicated is that like you can't AI is non deterministic. So you you can't be of very sure that like if X happens, is AI gonna do the same thing every time? And also you can't be sure that you Of knowing all the different things how how your users might might use your AI. Like your your user might come to your agent and ask n different things. You can't possibly predict all of them in advance. So so that's why evals are exist and how they are sort of different from tests in in traditional software. And but they are similar to tests. They are sort of like if if a user comes in to my AI agent, they they ask XYZ, this should be the output. How should I make sure that output is of good quality and it's not causing any other risk to my business? That that's kind of the goal of the eval. So the off online evals it's is more about like as your agent is running live and users are ⁓ are using it live, how do you make sure it's doing the right thing while it's running live and it's ⁓ it's effective to whatever the goal of the agent is and it's not creating any other risks. So every interaction or every workflow that agent is running through. you you want to monitor that or at least some of those interactions you want to monitor and run those through evals to make sure that the the quality of of that agent is is is good in in terms of those evals. So so yeah, the the the online eval is running right when your agent is running live in production, whereas offline evals are generally happening before your agent launches or before you deploy a new version of your agent to test out that this new version of the agent is working well. Can somebody give me an example of what an eval actually looks like? For some example use case? Yeah, so there are different so we have online and offline here. If we take a very simple chatbot, for example, that's doing customer support. ⁓ an online eval can look like, for example, an adversarial data set of user inputs where a user tries to hack, quote unquote the customer support chatbots to leak in personal information or just ask questions completely out of scope. And all offline evals there would be just a list of these questions and then checks whether the agent actually stayed within scope, just defer the question or actually leaked information or does something that you didn't want to do because you don't want to ship your agent when that's happening. So offline evals are actually there to detect that before you ship your change so that you can still fix that before you actually put it in production. Online evals and are there more an example there can be did the agent
actually complete the request of the user? That's something there is no ground truth, there's no ex completely clear, expected right answer. And you're just trying to evaluate certain qualities or metrics that you want to be met for your product. Do you guys have any like interesting stories where your online eBals found something that was like totally unexpected compared to the offline eBoys you guys had? With some biases we launched recently ⁓ an agent that helps people analyze their experiment results. So you know they are our users' train models, they log their experiment results to to our platform. And the agent is supposed to help them analyze that and figure out the next steps. And a lot of people, not a lot of people, but we found through our online evals that some people were trying to use the ⁓ the agent for as an arbitrary just chatbot, just using free tokens essentially, ⁓ to just ask general questions. I've heard similar stories from some Amazon support chatbot, I think, where people were just using them as a regular chat bot. I think there's even some sort of black market platforms to ⁓ to access free tokens in this way. And so you can see this not by looking at traces one by one, but by by setting up online judges that can that can monitor those things. So that's that's a funny example that happened to Yeah, maybe ⁓ an example of why we believe you generally need to inspect the traces, not just the output to have effective online evals. One of the the agents we've deployed for customers and also for ourselves is basically a Sales data update copilot. Like it it pulls from your CRM, Slack Notion, email, updates contacts and all the live deals, determines where they should be in the pipeline. So the team can run this automatically and get contacts in every deal as it progresses through. And models are ⁓ lazy. Like if they don't have a way to accomplish a task, they'll make up another way to accomplish the task. So one of the issues that we saw very frequently in sort of the V1 of this agent was if the ADIO integration was broken or like the Slack integration was not connected or someone not set up their auth token for Notion. It would just make up that data or just skip going and accessing that source and come with something very believable, but it was missing the call transcript or the Slack discussion. And by inspecting the trajectories and making sure you had live evals or verifiers to make sure it actually accessed every system that it needed to access, you can see that it had all of the data in its context before it made the update. Whereas if you were just writing evals that looked at the output, you could say, look, it has all the fields, it was updated, it like says the right things. But in fact, you would not know that it was missing the context because that's not always visible a priori from the output of the agents. This is why live evals really powerful for the reasoning trace inspection, not just the output. I also think
online evals are particularly helpful because as you have an agent production, you're gonna get tens of thousands of these traces coming in every single day. So you just like as a human cannot comb through these traces to find like what is actually valuable to look at. So when I think about online evals, I'm actually thinking them more as enriching these traces for me. adding some data to look through these traces and let me know what I as a human should look at just because I don't have time to read through, you know, thousand traces, ten thousand traces. Like if it's scoring it on helpfulness like one through ten or looking at tool trajectory, I might then just like go in and look at like the, you know, eval scores that are underneath three. ⁓ and then I as a human can go in and verify what's going wrong here. Then I can run it, add it to like an offline data set to build up benchmarks to run in the future as well. Could you talk a little more about that? Like, so what kind of data are you adding to those evaluates to make it easier for you to look through them? Yeah. So I I feel like an example would be helpful. Yeah. So like for example, let's talk about tool call trajectory, right? Like you might set up an online eval in which I want the tool to be called and orders A, B, and C. Like I want it to query, you know, I want it to read Slack, and then I want to look want it to look at Salesforce information information, then I want it to look at like the gong calls for that particular customer. Like we have a go to market agent which connects to like our Google Calendar, my Gmail, me, my AE's Gmail, any like information we have, a notion about the customer, Salesforce, Gong, like all the transcripts, granola and things like that. And you know, you specifically want them to like go into Salesforce first, like gather all that account information before you like go out and search out all the people on LinkedIn and like you know, go gather like gone call information about them. So there's a specific order in which you want these tools to be called. So you can set up an online eval to make sure, you know, is the agent, you know, calling the the tools in the order that I expect. And then you can have it categorize this or bucket these types of traces and like the types of trajectories that is happening. And then like when I go into my trace view, I can then just simply like filter through my traces as like the buckets of categories that the trajectory went down. And then like, you know, I might. have it scored as well to like find like unique ones or ones that really went wrong. Right. And then I can then as a human just look through like the five to ten unique traces that are powerful or helpful for me. Or let's say like there's a lot of like 10 out of 10 traces that I think are really helpful. I might add this to like a benchmark in the future to like, you know, train on or like this is a really good benchmark example. Like add this to a data set. So
I can like whenever I make a change to the engine in the future, I can make sure that it still passes that test. Interesting. Could you explain what a verifier is? I think a couple of you guys talked about that. Does someone else want to take a verifier question? ⁓ yeah, I'm not sure about the term verifier, but few things related that we see a lot is like how do you even trust your evals, right? Like how do you verify your evals itself are working well? Because ⁓ I mean it's sort of like evals are j themselves generally powered by an AI model at the behind of it. Either they are model based or they are code based. And so how do you trust your evals themselves are accurate? Because you're relying on them to actually find issues and ⁓ trust those. So setting ⁓ Well what we what we see, what we generally recommend our customers, users is ⁓ that itself is also an iterative process. Like you need to continue improving your evals. It's not like a one and done thing. Like you might create an eval, for example, as mentioned, that tools should be called in this order. But But maybe in the future you add add more tools or there are specific cases where you need to think about how the tools should be called that you discover as your agent is online. So you need to continue define improving the definition of your eval itself because they generally tend to go stale. So yeah, ⁓ like for example, in our platform Galileo, many times we have ways where we automatically flag that okay, your eval seems itself seems to be inaccurate and you can provide feedback on that eval itself and then We try to auto improve that eval based on that feedback. But this is a issue we see customers struggling commonly with. Like they're trying to improve their agent, but at the same time they also need to continue improving their set of evals then kinda making sure they are of high quality and verifying those. All right. So presumably you guys have been working on evals for a while. So what would you say is kind of the most surprising thing that you've learned across all the projects that you guys have done? I can I can take this one. So I've seen the the pre AI era and the post AR I've seen some So pretty stark parallels. I remember being a software engineer and try to maximize my test coverage. So getting hundred percent test coverage and then still getting bugs in production. And I basically learned that it's as a better ROI to set up broad online monitoring. So at the time it were things like no data dog or new relic type stuff to catch issues or slow queries, slow ⁓ database queries, for example, than trying to hell climb and get the the full possible test coverage and and so on. So Having broad online monitoring that then you convert into specific ⁓ test cases in your in your test suite at the time was more efficient than trying to imagine all those particular failure modes. And the same thing is happening with agents in the sense that it's impossible to imagine all the possible failure modes of your agent. It could be tool call failures, could be latency timeouts, token overruns, but it could also be misbehaviors of the agent. It could be people asking things of your agent that it's not even supposed to to cover. And so it is you know, a lot of people are trying to build the perfect eval sets to ⁓ you know, add more and more examples and so on. ⁓ I mean it's a worthwhile while thing to do, but the better ROI is to set up those ⁓ sort of online nets, if you will, ⁓ to catch all a broad set of of failures. You won't know exactly like the exact failures, but at least it will flag this particular instance, particular convers conversation that then an expert can review and then put into an offline data set and set up the correct ⁓ sort of evaluator for it. So for a long time we took we people have been told, ⁓ do evals, super important to do eval before you go to production. I would argue maybe the baseline, sure. But it's better to ⁓ invest time into your online sort of catch-ole nets to catch those those things when they occur in production and then convert them into ⁓ offline evals rather than trying to do the other way around. So yeah, so this this interesting parallel. Yeah, I think when you go and you do this with sort of significant scale, particularly in customer environments, there's A bunch of like hard lessons that you learn that you may not have thought of in advance. Some examples. If you have too many online or live evals, you'll actually spend more money running the evals than you will running the agent. So setting like the correct sampling rate, understanding how you have, you know, where needed deterministic verifiers to check elements of quality which are mandatory versus qualitative scoring that you can sample is is important. We deployed an agent where we had a hundred percent sample rate initially and it like four X the budget because we're spending three times as much. Tokens running the evals as we were, ⁓ the live evals as we're actually running the agent. Writing evals correctly is extremely hard. LLMs cannot write good evals. Like they can take a first pass sometimes, and we do a tremendous amount of research internally on automation. But like a human being who knows what they're doing needs to actually look and see that the thing written makes sense. They in particular cannot write evals for themselves. Like if you're using an LLM to write an eval for the same family of LLM, it is probably actively destructive for it to grade its own homework. You need to have an ensemble of models that you use. So when we are doing, you know, serious offline evals, we'll route to multiple families of models that we know from the benchmarks we've run are best for specific families of tasks. Like you can't ask model A to run online or offline evals against model A and expect it to know that itself is wrong. Understanding when you need experts versus internal company context. what the realm is of like
reasonable disagreement on qualitative tasks. It's really easy if you have like overly rigorous offline evals that you then hill climb on to basically force fit the model. It's sort of like overtuning in prompt space rather than overfitting in in parameter space. And you just like over time learn all these things as they go wrong. It's very much like an art as much as a science understanding all these edge cases. Yeah, maybe maybe one thing to add to coming back to the online monit monitoring thing, actually the one number one thing we recommend to users who are starting out is use online e-vales to capture user signals. It's slightly different from evaluating the quality of your agents, but it's just the cheapest, easiest thing to do. ⁓ so for example, if there's any application where you have interaction with a human. Try to always set up some kind of monitoring to catch user dissatisfaction. This can be caps lock, the words try again, or what are you even saying I don't understand, or anything that's sounds frustrated, because this will help you surface yeah, traces you need to look at v really, really fast without needing to think about metrics. Because as mentioned here, writing e-ails and also choosing the right metrics is is extremely, extremely hard. Capturing just user signals is often very cheap and Very or relatively easy to do. So a lot a lot of you guys touched on this, but like, yeah, you guys are getting so much data from users actually using the agents that it's not possible for a human to like look through all of them. So you then have an LLM basically judge, you know, all the data that's coming in. How do you know that the LLM isn't cheating? That it's actually like judging correctly. Yeah, I mean I think I I I talked about the I like the issue around like the evals themselves not being accurate, but What what we generally see recommend our customers is that they can't completely trust evals, right? Like they to some extent they still have to take a sample of what's happening in production and then have a human review it so to to make sure that what what LM judges are saying are accurate. So what we what we generally recommend our users customers is that like ⁓ every every month, every few months, depending on how risk tolerant those customers are. Like a lot of the banks we work with, they they really do this very religiously. Like even if they've launched an agent, every week they'll have somebody who is a subject matter expert actually go in and as a human review some sample of those ⁓ traces and and see whether their judgment actually then aligns with those automatic evals. And they're always tracking whether the human judgment aligns with the automatic evals. And whenever they see a drift between those two, they want to make sure that they're improving those emails based on that feedback. ⁓ so that's that's a process we generally recommend, especially if you are in a market which is much more ⁓ less risk tolerant, right? Like that you need to do through Yeah, I would say when we go and we build a high priority agent for a customer, our Ford deployed engineers and subject matter experts do actually look at every trace during the initial rollout to make sure that the evals we're writing are capturing what needs to be captured. And then it's an iterative process to create the test suite. But you You cannot just like a priori think these are the things that matter and just trust like the LLM as a judge to get it correct. It is a process of iteration with human expertise and substantial research effort on how you set up LLM as a judge, when it works, when it doesn't work, when you need deterministic verification. There's a lot of expertise needed in doing that. And where I see people get this wrong when we go in the customer environments is they've just had Claude write some evals and then they run it. And they have Claude judge the evals Claude's output. And that like will work approximately zero percent of the time. Brayden, so you work on enterprise AI deployments at Mercore. So ⁓ you know, you probably see ⁓ tons of different kinds of agents. Are there certain types of agents where you know creating evals for them is tends to be more difficult than others? Yeah, if you think about the broad families of agents that exist, you have agents where they are completing a task and you can Define in an objective sense that the task was completed correctly, even if the task is difficult. So if this is something like you're you're taking in an invoice and you're reconciling that invoice with another data source, this may be hard from a act systems access perspective, right? It may need to access many different sources, it may need to do it in a specific order, it may need to follow very specific rules, it may need to do natural language processing along the way. But there's a sense in which you can say that the task was performed. correctly or incorrectly and sort of build a system of evals and verifiers, you get confidence that you're hitting, you know, 99% or whatever confidence threshold you need. I would say those are difficult from an engineering perspective. They're difficult from a deployment perspective. You need a really deep understanding of business processes, but once you have defined the requirements, actually setting up the evaluation verification is a relatively straightforward thing to do, as long as you've done all of the other very hard stuff first. There are other use cases that we deploy, right? So we have private equity clients, we've deployed like due diligence automation agents. There's elements of that which are objectively correct or incorrect, right? Like given a set of rules for calculating adjusted EBITDA, you've either done it or you've not done it. But there are other elements which are innately subjective, right? Like, how do you know that strategic conclusions are correct? How do you know that you're applying a framework in the right way? How
do you Stylistic things are extremely difficult, right? Like you can tell a model to be concise, but what does concise mean? How do you stop, you know, AI slop from making its way in a very obvious way into the wording? So I would say that when you are deploying an agent which is qualitative in its output, and a range of experts could have genuine disagreement about what good output looks like, that is extremely difficult to write evals for. And the risk is that if you are overly specific in those evals, you will overfit in prompt space. And end up with output which actually doesn't cover the range of variants that you'd want in a library. What about agents where the input and output is not text or code? So maybe like voice agents or computer use agents. Have you guys worked on any projects like that? And how do evals work for agents like that? Yeah, I mean we've we've done stuff like that. I would say voice you You generally want to decompose these into a format that LLMs can work with. So like you are you're rarely going to, in the case of like a customer success voice agent, run the evals over the voice output. You're gonna decompose it into the text output and then run the evals over it. As an aside, when we have deployed for customers ⁓ high reliability voice agents, we have often opted to do a reasoning model generating the text and then doing the text to speech as a separate step. rather than single shot native voice models, even though those have lower latency, because the ability to incorporate a a reasoning model and inspect its traces is much higher. And that sort of becomes worth it from a quality and verification perspective. I can add to that. Some of our customers are building voice agents as customer support, for example. And the way they set it up is indeed breaking down breaking down the actual text outputs to evaluate the correctness of it. And then ⁓ the audio itself would also be evaluated but more for tone and sort of general attitude more than for the content itself. And obviously regarding depending on what you want to evaluate, you have to use different LM judges. Not all LMs are able to receive audio as input. So that further reduces the possibilities of LM judges. Plus obviously those are more expensive. And it's heavier, obviously you have to upload your online pipeline has to upload the audio to the judge. So like latency is longer. So you need to make sure you don't block execution or anything like that. So probably ⁓ do break it down and ⁓ evaluating the audio itself is probably not necessary. ⁓ in this case it was a high volume customer that really cared about tone of the support agent, so they they did that. But in some cases validating the text and the content ⁓ that was spoken by by the model is probably sufficient. Maybe I missed this, but how do you evaluate the tone? You you feed it to another LLM that can can take audio as inputs and you ask that judge to obviously that takes its its own set of alignments. And obviously it's also expensive, so you have to sample the data quite a lot. ⁓ but that's one way of doing it. Yeah, one of the things to add there is with multimodal like audio or image, if you're doing evals on that, but even on text, like as as ⁓ I think it's also saying at Mercor, like when you do that at scale, evals tend to become very, very expensive, right? Like, but especially with multimodal, like if if you have voice and image, they become very expensive. And and sampling is one way to make sure that you are doing it, you can do it more cost effectively, where you select a sample of logs and run evals on that. But what we see a lot of with our customers is that because they are enterprises, they did they really want to do evals on all the traces because you don't want to miss out even if an issue is happening with even one percent of your traces, there might be a security risk. There might be a a compliance risk that's happening, even if it's affecting a small population of your customers. So what what we provide to our customers is also a small language model to do these evals. So it's not just using LLMs to do the evals, but also We we we help them with a small language model which can run at like a ninety times cheaper cost than using a LLM to do those evals. So small language models that can also take in the voice input or take in that image input and be able to kind of ⁓ interpret the tone of that, which can reduce the cost of ⁓ doing these evals ⁓ to a much lower cost. You guys trained your own model, right? Like Luna? Yeah, we we train our own. small language models. I mean it's built on top of open source models and then we've fine-tuned it to work ⁓ very effectively as an eval model because ⁓ I mean if you use a general LM it's it's it's it's made for all different kinds of purposes, but we've fine tuned small languag smaller models which just give you a binary output like zero or one because that's all you need for evals. And then so then you can fine tune it. or further for different eval type use cases and then you can make it much cheaper and also it also becomes much, much lower latency. So because LMs can take a long time to respond. But if you're going to put this, that allows you to also put this eval in the path of your agent itself. So a lot of our customers then are also able to use it as a runtime guardrail. So it's not just a online eval that's giving you some sort of insight what's failing, but it you can use it as a if else condition that like okay if if I see my eval that the tone of the audio is going to be something that's really negative, I actually don't want to send that response to the user and and ⁓ send them something else. I think you're bringing up a good point about those binary classifiers. When you set up online evals it's it's tempting to ask the LM judge to score on a like scale of like I zero to ten or whatever or percentage. And this is really bad. You should probably not do that at all. ⁓
the better quality output, the higher sort of precision will be For binary classifiers, it's much easier for an LM to say is this offensive content or not, not like on a scale of zero to ten, how offensive is is this? Because then how do you define a six versus a five and so on? And LMs are better at like making a binary judgment call, true or false, is this this or this is this the opposite, rather than grading this on like a like hurt or or ⁓ a percentage scale. So so binary classifiers are much more efficient ⁓ and higher quality than than anything else. Maybe an example and that's Indeed, so completely agree that letting an LM rate on a scale is probably not like a good idea in most cases. An example there is for example if you want to evaluate the politeness and you would say rate it of from one to ten. It's very hard to say what is a five or a six. Even for a human, it's the same for an LM. Also, if you would instead say give me a categorical score for helpfulness, and your options are root, insufficiently polite, sufficiently polite, and extremely polite. In that case, if I would be the developer of that application, I would say, actually, I don't want my agent to be extremely polite. That's actually a bad case. I what I only want sufficiently polite use cases because I don't want overly worried responses or probably the user will be frustrated if something is not working and the LM is like, ⁓ you're absolutely right, because we all hate that response. So that's also a good way to make yourself as the developer of your agents. really think about okay, what is it I actually want, what are the things categories I'm defining and also what will be the action I want to take when a certain value comes out of it. Because you by do yeah, by letting it rate on a scale of one to ten, you're actually not thinking sufficiently on okay, what do I want to do with it then? Braden, so could you walk through like what is the full life cycle of a project whenever you get a new project that comes in? And then I'll tell you why I'm asking. Yeah. I mean I think we ⁓ on the enterprise side we have a range of different things that we do, right? So we have some customers we work with who are extremely far along their journey and they know exactly what they want. So for example, we have public partnerships with Harvey and Ramp to help them build benchmarks. And these look very much like the shape of work that we do with a Frontier Lab. If you think about customers who are maybe full end to end, like they they're starting from nothing, they have an idea of like the agent they need and then they need to go all the way to production. ⁓ we spend a lot of time on context ingestion. Like we have a belief that a lot of what makes a company special is internal to that company. And so we have products like an AI interviewer, which can go and capture workflow capture tools, the ability to track telemetry across different enterprise applications to see how people do their do
their work. So there's an element of discovery to even understand like you have a general idea of the agent you want to build, but then you want to go and actually build it and what does that mean? So there's a tremendous amount of scoping effort and understanding of company contacts to even get to the stage where you're building something. You then go and you do the build. There is often true like software engineering work to make this happen, right? Like you may need to go and set up the integrations, go and actually make sure it can access systems that don't natively expose an MCP or API. And so we do that work. You then get into the build process. The Agent is a specification, a set of subagents. We have an orchestration platform that we can deploy. We can also work with other orchestration platforms for not picky. So there's the creation of the agent, the sub agent, the specification of the applications it needs to access. You then get into the initial set of evals that you write for it, live or online and offline. These are to some degree a guest. Now, in certain cases, there may be hard deterministic rules that need to exist, right? So we've worked in areas like accounting, tax reconciliation, invoice reconciliation. Here you can define in sort of a deterministic way what good looks like. And so those sorts of mandatory gates that it needs to pass are always built. ⁓ but then you get into a testing stage, right? So we often Have users go and use the agent after it's constructed. You gather the user feedback. I think the point made that user feedback is an extremely rich information signal is correct. Like even if the live evals say that the agent is great, if a user abandons it halfway through, like it's probably not great. And so then there's an iteration process, right? And this is technology led, but it's also FDE led to go and get to a place of acceptance. And then you improve the agent to a state that it's ready for sort of enterprise grade acceptance and wide rollout. As part of this process, we will use the offline evals to figure out the ideal model for the use case. There's an element of model routing, which is part of the deployment as well. And users in this case could be like internal business users, correct? Yes. We do ⁓ we do customer facing agents and we also do internal business use cases. So like how much is it is it difficult to get like the experts in the room to like make sure that the evals are correct or not. Because I'm assuming you have to, you know, get the, you know, who whoever you're building the agent for, like, hey, get your accountants in the room and see if this is all correct. Yeah. I mean, it is difficult. We one of the things we believe really deeply as a strategic thesis is you need to reduce the amount of time needed to get expertise from someone. ⁓ and so the higher the efficiency of that information capture, the better off you are in a deployment. So we have things like a AI interviewer with screen sharing where they can just walk through their workflow. And we try to extract this at scale. We also go and ingest context from the company, their SOP documents, examples of past deliverables. So you can extract as much of the what good looks like in an automated way rather than than needing to get them into the room. And in generally, for workflows, we will have our Ford deployed engineers as well as experts sourced from our talent network. go and do as much of the eval checking ⁓ and quality checking as possible to remove that burden from a customer. The goal being that generally when we go into an organization, they want to deploy agents, one of the biggest bottlenecks they have is just time of the team. And so we try to reduce that burden of creation to the greatest degree possible when you do a project. Maybe to add to that one good way, especially if you're in a high risk industry and You don't want to just ship your agents without setting up events because that's actually one thing we I recommend often, just ship and think about the exact offline events later. If you're in a high risk environment, that's hard to do. So for example, if you're automating something that a lawyer normally does, one thing you can do in the beginning is just let an agent propose, let's say it's in an email workflow, let an agent just propose an email and then just capture the user feedback from the lawyer and capture what it's doing with the agents. agent responses and just you try to minimize the number of edits the lawyer makes between what the agent proposed and what they're actually sending out. And so this is also a very good way. And actually you're not asking much of them. It's really reducing friction. Ideally as the agent gets better, it's actually making them faster. And you are continuously actually gathering a lot of very useful info. Yeah one one challenge we had on the tooling side so at at Core Weaver We're building just like ⁓ like chain. So we're we're building tooling for for people who do this. And it's pretty difficult to build a tool that is both suitable for a software engineer to inspect the lower level spans and the latency and token cost and tool call failures, but a tool that is also usable by ⁓ let's say, a compliance officer or or a designer to evaluate the quality of the output of the agent. So that's a pretty challenging task to ⁓ offer a tool that is both like deep but also sufficiently easy to use for For what we call SMEs, like subject matter experts. So that's that's a challenge. ⁓ imagine if you think about the old days, you know, your PMs will never go into your data dog, right? They don't really care about data dog, they care about the how the page looks like and the the workflow, but they don't use the same set of tools as engineers. But now, because agents, the logic is so much encoded in the context itself, which is written by ⁓ PMs and SMEs. then they need to use the same tools for monitoring and evaluation. And that's ⁓
that's a challenge. Yeah. One thing we notice is getting like all this set up is like such a a time sync. ⁓ so like one thing we did at Linkchain is we came out with Lanks with Engine, which is a agent itself that will sit behind all your traces and ingest them as they come in. And then as it reads through your thousands of traces every day, it actually creates clusters for you of common errors that it sees. I mean this is really just nice for like seeing like the common error issues that are happening. So this is often like a nice way to get started like finding like Okay, what errors are actually happening with my agent? And then you can automatically create ⁓ evaluations from there. And as another thing that I think is really hard as well is like, how do you get your annotators to look at these traces? Like a lot of these people are coming in are not typically technical. So how can you build an environment that's nice for them to come in here and mark up these traces? So one thing we've also built at Langsmith as well is this idea of custom apps in which you can essentially like vibe code whatever UI that you want for your, you know, subscript experts to come in here and look at these traces. For example, like sometimes some people just really love Excel. So you can build Excel within these custom apps. Or let's say you only want to look at the inputs and outputs. You can build that in. If you specifically want to look at tool calls, you can build up whatever UI you want within custom apps to just specifically look at those. So like we found that like the mediums that people want to like ingest these traces, especially from the substitute experts, are very niche to like what fields they're in, right? Which is kind of why we've given them like this sandbox to build whatever they want. Actually, I had a question about what you do when you run evals on like long horizon. Agents. So let's say you have these agents that run for a while, they do a ton of tool calls, whatever. Let's say there are certain traces that you see that come up that tend to be wrong. Hey, like my input was whatever my ⁓ it was supposed to make like these three tool calls. It didn't make these three tool calls, and then we get the output and the output is wrong or right, whatever it is. How do you take that data and then improve the agent based on that? In terms of looking at trajectories? Yeah. Like You're looking at these trajectories. Okay. Like some of them are wrong. Now what do you do? I think it depends on your agent. Like, as you were saying, like, is there a correct path to this trajectory? Like, is it verifiable? Like should it do A B and C. If it's subjective and like the the subject expert can't agree like what the correct path is, that's where things get a little bit more difficult. Like, of course, like you can flag these traces with interesting trajectory paths and you can even bucket these trajectory paths to see the common ways that these long horizon agents
are going down. Like, did it spin up a sandbox first? Did it start running in there? Or did it go ahead and and gather information? So like One thing you do is bucket these trajectory paths. Like one thing we're looking into at Langsmith is giving like nice visualizations as to the paths that your agent can take. And then from there you can put it in front of a human. I think LMS that you judge are nice and like to like essentially like categorize your traces, but I think of them just as enrichment. Like you got to put them in front of a human at the end of the day. Like I would love for my agents to improve my agents and I hope we get there, but I just don't think we're there yet. And I think the human is like the ground source of truth. So I primarily just look at it to like bucket the categories. And then take like these super like interesting examples, then put them into the experts so they can decide like what good looks like. Yeah. I mean, we ⁓ we published an article recently about one of the agents my team built for some of our investing clients, right? It's like a six hour average runtime, something like a hundred subagents working across a bunch of different files, basically automation of investment research. I completely agree. If you're running a six hour task, like a human should look at what happens in that six hour task ⁓ essentially a hundred percent of the time. Now, there's technology you can deploy to identify things that go wrong in the traces. You what you really want to do is identify things which are wrong in the output. So you can connect the things that are wrong in the output to what went wrong in the trace. So you can make a more intelligent edit to the specification of the agent than just telling it not to do the thing that went wrong in the final output. But some of the stuff you'd uncover, you'd you'd never uncover without a human looking, like a Some stuff just got evicted from context based on the context management. We had to like rebuild parts of our context management engine over the six-hour task, like various subagents for not sharing across themselves correctly. There's a a ton of richness that like LLM as a judge will not capture. And if you're running like a multi-hour task, like a human should just look at what's going on. On that note, ⁓ one very good thing to look into is the process of error analysis. Is someone called Hamel Hussein who has a very popular cross course on evils who really preaches the method and I think that's a really great way to go from you look at traces, there's something wrong. How do you translate that into evils? And so what what he explains is actually you look at your traces, you write down what went wrong and just do that fifty, a hundred times. I mean you can do that over multiple ⁓ days. It doesn't need to be at one goal. And then you or that you can actually let an agent analyze what comes out of that. And it will cluster it in failure modes. And hopefully you will find three to five failure modes that happen multiple times. And these are probably the best things to start setting up evals with. I mean obviously there's no replacement of a human can look at each and everything the agent is doing. That's definitely useful. But one of the eval types that we found successful there is what we call agent trajectory evals or agent flow evals. So what what we have a lot of our customers do is like when when when an agent we are when you're asking an agent to do a long task, generally we recommend having some sort of runbook or playbook that you had before how humans were doing that task. So sort of like translating that run book or playbook into evals that like, okay, if the task of is of this type, Then you should f the the agent should follow XYZ steps. If the task is of this type, then the agent should follow XYZ steps. And translating that basically into evals has has been useful for many of our customers to catch a lot of issues with these ⁓ longer running agents. ⁓ because you can almost directly translate those run books into evals that like for queries of this type, this this should happen in in terms of conditions. If you're able to describe that in an eval, a lot of issues are are be able to caught in that way. But obviously it it's not foolproof and ⁓ Kind of having humans look at it helps you improve that process continuously because there's no perfect run book that any company has that captures each and every detail of what a human was doing before to do that same thing that an agent is supposed to do. Yeah. One thing we've also started working on at Linkswith and we've implemented is like as you have these longer running traces, like the traditional granular trace view just was too much to look at. Like when you have these traces that are running for six hours, there's just so much information. So we spent a lot of time like doing really condensed messages view to help you just look at like what's actually happening in the six hour run. Cause I don't really care so much about every single guardrail or every single agent decision that was made, like at least from a top level. I would like to see like overall what's happening and then click into like each part or this specific subagent run to then get granular usage into where I think the air is. But like we've had to like zoom out a lot from like our original trace of like showing every single detail to just like the high level details and then allowing you to click in from there just because these traces are getting so long. What are some examples of long running agents that you guys have worked on or seen? I I feel like you have a bunch of long running. I'll let you go. Yeah. ⁓ yeah, I mean, we we mostly build long running agents. I think the two categories of agents you build for customers are either places where you have extremely high reliability requirements or it's a long running task. So some examples automation of investment research for investment
firms. This is a popular agent that we deploy. Basically, sales copilots where you need to integrate research from a bunch of different sources, CRM, email, Slack. pull together all of that context to brief someone. We've done some very specific workflow automation agents. Like there's companies where they have a core internal process that they're running with like a bunch of Excel spreadsheets. We've built the automations of those so that you can allow employees to basically supervise the output rather than do the output directly. We have some SDLC agents that we've deployed. These can be sort of quite significant in the data sources they need to adjust and the runtime that they work over. So I I would say like most of the hard problems that people are deploying FDEs for either are are critical reliability or they are long run. What are some of the hardest eBell problems that you guys have worked on? ⁓ I think for us. I mean, there there's a lot of ⁓ commentary which is, you know, I would largely agree with that doing Likert scale quality grading, you get much worse output. And this is this is completely correct. But often what customers want are long running agents which are taste based, right? Like they want to write in their style, produce an investment memo in their style. They innately have qualitative elements. And so, how you understand where LLM as a judge can score on a scale versus not score on a scale, when you are able to break things down into like a subset of binary classifiers, when you just fundamentally have to understand that there'll be some variance, when you need to use other techniques for style transfer, I think getting customers happy with taste-based qualitative long running outputs is like the one of the hardest research problems we work with. I think one of the challenges we are seeing a lot with evals is like as eval as agents are kind of taking actions across systems in a company, right? Like they're not just producing some qualitative output, but they're actually going and updating a database or kind of actually going and sort of like shopping something for you. You could say, for example, like it's getting harder to do those evals because then you just can't rely on looking at what the agent did, the trace of the agent, what ⁓ because the agent might claim that it did something successfully. And and it might be even calling the right tools to do it successfully. But unless you actually go into that system and look at that database and look at the final screen, like did the action actually happen, did the refund actually happen? Did the purchase actually happen? ⁓ a lot of times that last mile is hard for the eval system to see. And that becomes challenging. So I think like that's to some extent it's still ⁓ I would say like an unsolved problem. But the the best solution there is to like make sure you can completely trust those those end. sort of the end tools that you're giving your agent to access to that you can
trust that those are doing the right thing if the right parameters are passed to them all the time. And checking whether the agent is always calling the tool with the right parameters as as before it hands it off to to those tools to do something. The the agent I was mentioning earlier, the ⁓ research assistant is actually more than a research assistant, it's just auto research. So you can actually tell it to here's my my model artifact that I logged, like a a deep learning model, go and make it better. And the agent is responsible not only for coming up with the ideas, but actually also implementing the changes. So whether it's like a model architecture or training code changes, and actually going out and running those training jobs on GPU infrastructure and then evaluating the outputs, comparing the output with the baseline, and then if it's positive, then merging it into the baseline and then doing this over and over again. So this is essentially PhD level work. So designing a PhD level eval for those tasks is very challenging. So A lot of it is done manually by just trial and er and error. But you can imagine that the infrastructure to even evaluates like that requires access to compute, requires an ability to evaluate the ⁓ the ideas that were generated by by the agent and evaluate the final model as well to see if it's ⁓ better than the than the baseline. So these are very definitely very challenging and require a lot of human work. So what do you think the solve for that is going to be over the next year? Do you have any predictions? solve for auto research. Well, I happen to believe that auto research is Still a bit of a niche use case. I think ⁓ quant fonts are are using it definitely to try to ⁓ automatically improve their trade their trading models. But the vast majority of the ⁓ industry is not quite doing that yet. There's an like extremely long-running tasks that are almost unbounded. So I don't think we at a tooling layer, I don't think we need to solve yet for like the 10% use case. There's already a ton of companies that are even struggling to put a regular agent in production. So already solving for that is ⁓ would be ⁓ a huge advancement. So you know it's like if you were trying to build tooling for NASA, like you know, that doesn't really apply to the rest of of the industry. So I don't know exactly what the the solution is in this case. You probably need specialized tooling to evaluate that. But my point is that it maybe it's okay that we don't solve this yet because there's already a lot to do before we get there. Do you ever get to a point with maybe like more simpler agents where you're like the agent has basically seen every edge case that could be imaginable and As you were saying before, right? Like running ⁓ evals can be very expensive. So do you ever get to a point where you're like, we've this agent has seen anything, as has seen everything that it can? Should we just shut off all of our evals that we're running? ⁓ I
would never shut off everything, but like do you like when you first roll with the agent, is my sampling rate a hundred percent most likely? And then like over time I'm dropping it from there. That's typically how we advise customers to do it, especially as they scale up volume. Because like when you roll out, you're not rolling out to a hundred percent right away, right? You're going like 10% of users, 25, 50, 100. So I'll keep it at 100% sampling rate. You know, maybe we've been stable for like a week or two. Like we're pretty confident in this thing. Then you can just drop the sampling rate from there. At least that's what we've seen. Yeah. I I think anyone who's spent time in QA knows users have a tremendous ability to imagine edge cases. So you should be very cautious about believing you've seen all of them. I completely agree on sampling rate, right? Like we have an agent we run internally that sees ⁓ tens of thousands of invocations a day. Like our sampling rate is now single digit percentage instead of a hundred percent. I will say though, there are important reasons that you want to keep some monitoring rate in place. One is if you're doing like API calls to a model, if you switch the models over time, you may have drift in the performance or the way that it answers. That's important. You may also want that time series so you can make decisions about model routing. You may A, B, test different models to see if you can switch, if you get better performance. You may also, in some cases, have drift in user behavior. Like that, you have no edge cases over a period of time doesn't mean you have no edge cases forever. So I think the correct sampling rate is is rarely literally zero. Yeah. Also on that, I think so we look at it on evals, we look at it at it's two twofolds. One is defining your metric sets that you want to track and then actually setting up evals for tracking these metrics. And your metric set is definitely a living thing. So in the beginning, okay, every time you have a metric you might want to track or a failure mode you see, you need to ask yourself, okay, is this a one time fix? Or is this more of a generalization problem? If it's a one-time fix, just fix it, don't set up an evil. If it's more of a generalization problem that it's harder to fix with one a change to your prompt, then you need to set up an evil. At some point you will see this specific metric hasn't changed for months. Maybe it's fine, maybe you solved that problem, then you can turn it off or put the sampling rate at a very low rate. And maybe there's another problem that comes up and then that needs to be become part of your metric sets. So definitely you need to also keep evaluating your metric sets. Also to keep your cost low and also to make sure you're not looking at noisy data. You need to make sure you're actually optimizing for the next most important thing for your agents. Yeah, I actually think you brought up a really good point. Like we've been talking a lot about like eval drift or agent drift, but user behavior also drifts over time. So like within Langsmith, we have a feature called insights in which you can like cluster upon like what types of questions users are asking and like typically rerun this like week by week on our internal agents to see like How are people like adapting to the agent and using it over time? Because clearly they get more familiar. Maybe new use cases are arising. Maybe the tool call like acts as like a database that has now has new information that's enabling them to do completely new tasks. So like opposed to just like evaluating like how your agent is doing and the trajectory it's taking, you also need to evaluate like how your users are interacting with your product as well. And there's an another aspect is which is attackers. And you could you could run an agent in production for two to three years and everything is fine and suddenly somebody attacks. Trying to access, you know, your context, trying to access to to jailbreaker agents to access data that it doesn't have access to. Could be using your tokens illicitly. And so you can't just, you know, i takes your your hands off the wheel and not look at it anymore. At least you have some baseline ⁓ safety checks and things like that. Yeah, I think just to reiterate that point, like we work with a lot of ⁓ again, like bigger enterprises, banks and and ⁓ telecom companies. And for them, like if they're launching an agent, suppose like a top telecom company we work with, they have an agent on their website that's talking to all of their customers. They just can't afford to at least today ⁓ have evals not turned on for hundred percent of their interaction. So like they need at least some set of evals to have the guarantee that like some security attack is not happening, some some some interaction is not happening that's gonna create a compliance risk because they even have to report that to their compliance teams, ⁓ some of that data. They they can't have legal risk happening. And I think security especially becoming more and more of a challenge because as as as you all know, like we've heard about, like as model scheme keep improving, they're also becoming better at ⁓ like maybe you are using a much more powerful ⁓ model under the hood in your agent, just creating new security risks that you didn't thought about before with your previous evals. And now maybe there's some attacker that's using a much more modern a ⁓ model to attack your agent. So those those things are ⁓ becoming so hard to track as as new models keep ⁓ becoming better at ⁓ doing security attacks that you need to at least have Some level of base evals as a more serious company, as a more serious enterprise always going on. Also to add to that, you can see metrics, you can put them in three categories. So one is operational metrics like latency and costs. You will never turn it off. You will always want to optimize if you can. Then you have
qualitative metrics. These are the things that we mostly talk about when we're talking about evals. These are the ones that will be your most living artifact this set. And then then you have your guardrail metrics. And this is the thing that I think we're talking about the most here that you don't that you never want to switch off. So yeah, evals mostly we talk mostly about qualitative metrics. ⁓ but definitely guardrail metrics are super important and it's a bit of a separate issue, I would say. What are some mistakes that you guys see when companies are building evals themselves that they that you see over and over again before you know you guys come in? Braden, you were saying like one is just they just prompt Claude. Yeah, you can't ⁓ LLMs can't write evals for themselves. In the same way a human couldn't, right? Like if you were like, ⁓ why why don't I just write all the things I could do wrong and check if I did them wrong, right? Like it it just doesn't work. So that's the most common mistake. I think not understanding like when they're appropriate and when they're not appropriate is common. I think bringing in human experts and not knowing how to engage those human experts correctly in writing an eval is a common error. ⁓ Well what would be a way and you engage them incorrectly? There's this delicate balance, right? Where you have a human expert who has all the context knowledge in their head. And you need to get that knowledge out of their head into a format which is useful to grade an agent, right? And so there are multiple failure modes here. One is the human is not ⁓ trained to write evals in most cases, if you have just someone in an organization. So It's actually quite difficult for you to just list out everything that would verify if an output, even in a subject you're an expert in, is good. This is why when we have our AI interviewer, it's designed to like repeatedly prompt them, what about this? What about that? Can you share this workflow? Like most people cannot just single shot the things that matter. And on the other hand, like they may just not engage in a structured way. Like if someone has a bunch stuff going on in their day and it's just like check these evals, see if they're good, they will generally like not even know what to do with that instruction. And so if you think about like the core loops of our business, one of the core loops is how do you efficiently take the knowledge that is an expert's head who are not expert model trainers and convert that into a format where it is useful for agentic work. Yeah, I think related to that, I think the most common mistake we see is like, yeah, people just trying to They struggle to figure out what evals to use for their agent, right? Like in ⁓ and most platforms, including ours, offers some out-of-the-box evals for common use cases like that you can use right away. And a lot of times we see users just gravitate to that and treat them as gospel, which they are not supposed to be. Like they would they'll just kind of because those are just low-hanging fruit, they can start using these evals like we have around hallucination or we have around whether agent is completing their actions or not. So there's a lot of evals we provide and most platforms out there provide out of the box. But it's a common mistake for the customers to just like latch onto them and treat them as like all they need. And those are just what we think of as like starting templates. Like they they they they generally need to be they can find they can help you a little bit, but you need to then create your own custom evals, figure out with subject matter expert what really this agent needs and tweak those out of the box ⁓ evals to to reach a state where where it it works well for your specific agent. I think one area we also see Customer struggling is they're sometimes too academic about setting up these evals and data sets and that like they're like, ⁓ my goodness, we need 500, 1000 examples, be ready. Like typically what we recommend is just like start out with a few examples that you really want to win on, like some easy wins. Then from there, like run that benchmark, get your agent just like good enough and get it into the hands of like your beta users or like your your first initial tests. Then from there, you'll start getting your production traces. Then you'll see how people are actually using your agents. And you're probably still like combing through these by hand, but you know, you just have a few hundred traces. But as you scale up, you can find like common like error patterns. Like you might have set some online evals. And that's where you can find like the unique use cases that maybe your agent went really wrong. And that's where you're adding those unique cases to your data set. So when you run these benchmarks in the future, you then have like these larger data sets that you vouch on like production traces. Like this is how people are actually using it. And this is what my agent should do. So I typically just see like tell people, just get started, like build out your first like benchmark and then just like put it in the hands of your users to see how your agent's reacting and how your users are using it. How does that work for regulated industries? See, that's a lot harder. Yeah. And I I actually think like it's completely different. You have an internal versus external agent. Like, and if you're working in like a regulated industry, like internal agents are so fun to work on. It's like an employee engineer, just because like it doesn't need to be perfect. You can just put it in the hands. It's just like, you know, like if your sales guy says this is wrong, they can just ping you in Slack, right? And then you send it over to the agent team. Hey guys, this is wrong. Like we need to fix this. In the regulated industries, like I I assume you guys should be like testing it internally, like doing like beta users before it, especially if it's
like dealing with transactions. That's where like honestly most of your time is spent getting it from like ninety to ninety nine percent on the eBallot test set. And just like typically those agents are a lot less sophisticated just because like you need to be so certain of like the range of outcomes that it can have as well. Yeah. One example we've seen ⁓ with some of our customers that are using agents as customer support is that the agent would not talk back directly to the customer, but would talk to a customer agent that is supposed to basically get inspiration for their script from the agent. So they can in real time ⁓ sort of change the script if they realize that something's something's wrong. So and no customer support is not arguably the most regulated ⁓ sort of industry, but ⁓ that that's one way like keeping a keeping a human between the agent and the and the and the user not super scalable and it doesn't you know reduce the workforce cost that you're supposed to do with an agent. But that that's one way of doing it. Another I mean we s we we see this this problem all the time that I can't believe I'm telling that to people, but no don't spend too much time on your offline evalu before you go to prod. I can't believe have to do that say that because I would say the opposite to people a few years ago. But I would for a an an agent that is not safety critical or not regulated, it says just put it out and let it fail and learn from that ⁓ and build your evals from there. But don't forget to do that because if you let it fail for too long, then you're just gonna go away. So yeah. Yeah, I we spent a lot of time thinking about regulated industries or a close cousin, sort of high reliability requirement agents, or you need sort of multiple nines of of output reliability. I think there's a a few things, right? There is no substitute for genuine expertise in building those. And we like sources from our talent network. We get it internally from the company, but if you Do not understand what you're doing. You cannot build a high reliability or regulated agent and you need multiple sources of information to get it built. You in those cases, you do run at a hundred percent sample rate and you check the output before it goes to the user, right? So you build gates that it has to pass for the output ever is shown to a user. And then like agents are a hammer. That doesn't mean everything in the world is a nail. But we do a lot of projects are actually, it might be that you have a layer of deterministic code. And then the agent provides instructions to the bespoke deterministic code, which then generates the final output. There's one project we did where the actual like output to the user is an application that we built. And then the agent's role is as like requirements evolve, it updates the instructions given to the deterministic application, which generates the final output. So you do need to think in a really systems-designed, high reliability, expertise-driven way. For any agent where you need to hit multiple nines of reliability. Also, I think what was described before, try to go to production as soon as possible, learn from the data, and then later on you will see okay, this needs to be in a data set, this is a useful eval, and now it's really good. I think you can also apply that to regulated industries, but you just start with an internal phase of your agents where you do that whole cycle and learn from the data. And that's most of the time also your users are then your experts. So you get really reliable data. And then at one point you see, okay, now it's really good. And also by then you hopefully have defined, okay, I want to the agent to score this percentage or this this well on these metrics on this data sets before I want to go to production. And that's something you can also create from this data you learn from that that you get from the internal phase. And then you go to the external phase. And I think that's for a lot of companies, especially if you're automating something that humans are already doing, a very yeah, logical and scalable way to actually get to an agent in production. I'm actually curious, FirmurCore, do you guys ever work on projects where your customer doesn't have the expertise in-house to ⁓ evaluate their agents? And if so, then how do you make sure that the People that you're hiring are correctly evaluating your e vals, I guess. Yeah. So two buckets, you know, of use cases. You have external agents, you have internal agents. For external agents, this is actually relatively common, where a core function we provide through our talent network is sourcing experts who have expertise that the company does not have, right? And a very common example of this might be a software company building agents for an end industry and they need the source experts for that end industry. To help build the evals for the agent. And so this is extraordinarily like routine for us. A large portion of our business is this kind of work and is is closely adjacent to the work we do with the labs as well. For internal agents, I think there are multiple pools of expertise, right? There is the company context and the way that they operate. And that knowledge is in people's heads, it's in written documents, it's in the enterprise applications that we integrate with and build a context graph over. And then there may be expertise specific to the activity of the agent. So if you think about, for example, like a FPNA or accounting agent, there's the way the company operates, which is necessary, but then there are the rules for how accounting invoicing should work, which may involve external expertise. And you need to bring both of those components together. I would say if you think about the the things Mercore has built as like motes in our business. Being really, really good at finding experts, vetting them, and then training them to create output in a format which is usable for an agent
is something we have, you know, years at this point of hard won experience doing. Yeah. And I'm assuming part of that is also like the experts are kind of evaluating each other's work, whether they know it or not. Yeah, there's a whole like ensemble, right? Like you have experts, you have other experts review the experts. There are FDEs and people in the company who review that output, who check that it is. you know, in formats that are correct. We have a ton of research investment and automating elements of quality control. So it's defense in depth to make sure the quality is there. And Some, so this is unrelated, but ⁓ you mentioned this earlier. You said that you have these like built-in evals for hallucinations. How does that work? I've never heard of that actually. Yeah, I mean ⁓ for hallucinations specifically, I think a common, a very, very common way how hallucination happens ⁓ is that the LM responds with something that was not given to it in in its context. So so so that that's where ⁓ because generally in any enterprise agent ⁓ you want you want the agent to only respond based on what what it's given to it in its context, not something that world knowledge it has based on its training before before being tra ⁓ trained as ⁓ this enterprise's agent. So in that case, what what we have d what we have as a hallucination eval is that we've ⁓ we've sort of prompt tuned that eval to look at like make sure that the response that the agent is coming with is is only adhering to all the context that it was given to and there is nothing that it's responding with that's that's not ⁓ in its context. So that's one of the more most common types of hallucination we see and and that we have a eval for. But again, as I said, like it doesn't work for hundred percent of the cases. There are cases where we need to tweak it to to work for a specific agent. But obviously that it's something that a lot of our customers gravitate to and sort of see as gospel right away. Yeah. Also within lengths width, we have about like thirty pre built evaluators that you could use. But I typically advise customers to not use these out of the box. Like these are simply like templates. But I think we mainly have them there just to like help inspire ideas of how you might go about evaluating an agent. Because each business use case and agent is so different. I feel like, you know, how it gathers information and how it goes goes about executing is entirely different. You need to build that business logic into the evaluator itself. Cool. Well Those are all the questions that I had. Is there anything that I missed that maybe I should have asked you guys? Right? I mean, I think just one I mean, I was having this feeling while listening to this panel is like all of this business just feels so complicated. Like everything around evals, there's just like so much expertise somebody needs to have and this is so much new for the industry to figure out. So ⁓ I do feel ⁓ that I like it's it's sort of a temporary phase we are all going through, like where AI came in, like agents coming in and there's still people are figuring out how to build these agents and then building more and more complex agents. Like this is a this is sort of a temporary phase we are going through where we are all figuring out how to build reliable agents. And eventually we should we should as an industry reach a phase where a lot of this gets simplified. At least a lot of this becomes platforms that eighty percent of it is taken care of for you and then build teams that are building agents can focus on that sort of like 10, 20% that they they can uniquely do ⁓ and and add their taste to to their agent ⁓ on top of it. So ⁓ yeah, I mean today I think all of this feels very complex to some extent. Like how do I figure out the right evals? How do I figure out ⁓ how do I automate those evals? The cost is so high. ⁓ but I think ⁓ we we'll eventually reach a state where where a lot of this will be platformized and then ⁓ you you can you can focus on like the ten, twenty percent where where you can really make sure your agent is ⁓ you're working to your taste. Cool. Do you guys have any predictions about where evals are going over the next twelve months? I'll go. ⁓ I think we're gonna see some more fine tuned evaluators, especially if people have like larger production agents out and they're getting, you know, millions of traces, right? Like the online evaluators are very costly, right? So as you find like your most important online evaluators, I think we're gonna take We're gonna see more people take their these open source models with open weights and like fine tuning them specifically to be evaluators. Yeah, I think fine tuned evaluators are something I mean, that's very common already with most a lot of our customers already use them. So that's definitely we see that continue to evolve, like those fine tuned evaluators continuing to kind of serve every possible use case that there is in terms of evaluation. But the other thing ⁓ I see that in the next ⁓ twelve, twenty four months is that like ⁓ today, as I was saying, like There's a lot of effort still involved in like kind of figuring out what evals to use, what how do you actually then utilize these evals to improve your agent? So so what I feel is like the AI agents are also getting powerful enough that where they can do a lot of this work themselves. So it will again not take care of take care of all of this. But what we are seeing more and more is moving towards a world where agents are almost fixing themselves. You could think of it as like auto-fixing agents where agents are able to You can you can use another AI, more powerful A AI model that that can be looking at your traces to figure out like what are some common issues that are happening. And then look at those patterns to also look at your code to then see like how do I make changes to this code to actually fix those issues and then be able to
almost patch those issues. So I don't think it'll take care of hundred percent of issues, but I think that will be a common pattern as models keep becoming more powerful. You'll have these auto self-healing agents or self-improving agents. That that can take care of a lot of these issues automatically. And and I think that it'll almost become necessary given how security threats are evolving, also, right? Like I mean, you won't have time to fix these security issues inside your agents. You'll have to almost have some system that's automatically fixing these security risks inside your agent. Yeah. I think as token spend continues to blow up and it becomes an increasingly acute use case and more enterprises get to the point where it is an acute issue, using evals for model routing will be Extremely prevalent because it's self funding or more than self funding. Like the business case is just extremely easy to write. Yeah, on automation, I also think agents fixing themselves is really coming. One thing I think will stay in human hands for the longest is definitely your metric part of the evals. ⁓ just defining what good looks like, what's important because agents are very good at saying, I need to fix this specific thing, but actually it's not important at all. And so yeah, I think as yeah, we as platforms have a bit of a responsibility to make sure it's very easy for humans to set your metrics and also capture user signals because this is a very, very good signal also for agents to know okay what's actually important. And then for example, an agent's pro proposing, ⁓ I think this cheaper model or faster model can actually also achieve the same scores on your evil set. That's something the agent can definitely do on itself or fixing small bugs because of of a user signal is something that an agent can definitely already do. I guess I I agree with most most of those things. I'm pretty interested in ⁓ automated evaluations. So the platform should tell you where your agent is failing and how to how to fix it. So basically you should wake up in the morning and have a couple of PRs ready to to review it, even the A B testing ⁓ for that particular branch ready to review. So that you just as an engineer you just have to say, okay. Looks good, click and then merge and then it goes to prod. So even designing your judges, designing your data sets and so on should probably be at least the seed should be automated. Of course, there's always more taste and more expertise to be added by a human, but the baseline should should be automated by the platforms. Sorry, one more question on the fine-tuned evaluators. What exactly are you fine tuning them to output? Yeah, I mean it's the same output that any evaluator does, right? Like so so as we said, like generally the best practice for evaluators is to have a binary output. Although I mean in in practice, sometimes you need to have like a like curd scale or a qualitative or a categorical output. But so even in those cases of those fine-tuned evaluators, generally it's giving a binary output, like okay, is this a positive tone or not? Is this or whatever you're trying to kind of really classify as a judgment? Like train them on basically like whatever the humans think is good. We want the LLM to be as close to that as possible. Yeah. I mean, ⁓ we want it to be similar to whatever your ⁓ L L ⁓ evaluator is doing, right? Like the same data set that we are y giving to it, like the traces and the output that comes out of it, you want you train the fine tuned evaluators on those those same things. One other thing we also have is like self improving evaluators. And we have this with lengths with engine. So engine as it reads through traces every day will send you a Slack update or an email of like errors it found overnight. And then as you like say this was a really good error, like you tell Tell like, hey, like this was actually not an issue. It should actually like update itself. So like whenever it runs every single day to notify you, it's like based on your preferences from the past day. Nice. That's cool. All right. Those are all the questions that I had. Let's open it up to audience questions. Hi, ⁓ I had a question about setting up evals for complex and like messy. workflows and specifically for like financial analysis because the answers are oftentimes precise, require accuracy, and oftentimes also data is scattered and not like structures sometimes, like in Excel pivot tables or things like that. And I was wondering if there's a way to set up eval so that a human like doesn't have to like evaluate all the chases and the all the different combinations that the user can ask the agent. I'm only asking this because I was developing like I I was working out as a product manager for building a financial analysis agent for an enterprise and there they had really messy data and I kind of had trouble coming up with an e ball that could like essentially take my place in making the agent more accurate. So I I was wondering if there's a good example there. Yes as you mentioned like one of the traits it seems to be is that like it's generally giving some sort of precise output, like a ⁓ almost like a mathematical kind of output. Like so in I mean, one of the advantages in that case is like it's not subjective, right? Like you can come up with some pretty deterministic rules as to like what the output should be based on the input. So I I think like ⁓ almost defining like code based evaluators might be a good approach in that case. Like if you're able to kind of come up with a logic of like using code itself as to define an evaluator like based on this input. this this is what the calculation should be. Might work well. But I we also mentioned that the data is in pivot tables and stuff like that. So there might need to be ⁓ we also we have also had cases where you have to have sort of like chained evaluators where you sort sort of have an LM first look at the in the
trace and kind of extract the useful part of it and then have a code that processes that input to kind of come up with a judgment of whether this output is right or not. So But using a combination of LLM and code might might be useful in that case, potentially. Yeah. I mean, we ⁓ we build these types of agents for ⁓ for a living. So ⁓ it's hard is is the short answer. I think some tips you might consider. People often put too much work into the agent. Like if the issue is that the data is messy and there are inconsistent sources, a human could also not do that correctly, right? And so you shouldn't If you solve the data cleaning and context problems upstream of the agent, the odds of the agent being successful in the task are dramatically higher. And also like at scale much more efficient from a cost perspective. It's like a really bad use of frontier model tokens, like crawl across a bunch of messy Excel's and like reconcile pivot tables. For the The evals. Some of this is an agent specification, right? Like financial tasks may have a discrete output, but there may be judgment, right? Like a company may think about the way that they recognize revenue differently based on the company context, the way that you like do EBITDA adjustments, funds have different ways that they think about this. So this is like a not an agent eval problem. This is a codifying the SOP into the prompt and the subagents that they actually have a reference document that they know what procedures to follow. Then you can build a set of test cases where you can check that follows those procedures correctly. And that lets you know that you can sort of get to the march of nines of reliability. But where I see these things go wrong is people solve what are either context management or data cleaning problems in eval or prompt space. And that is usually where it goes off the rails. I have a follow-up question about that. So if you build like these types of financial analysis agent repeatedly, I'm assuming you'd have like a standardized way of taking in the codifiable logic so that you don't miss out on anything. So the problem I had was like there are so many company specific knowledge and there's specific ways of calculating things that the logic itself was like there were so many combinations of different logics going on. So I'm wondering if there like there's something better in place for you guys. So I think the analogy to this is if you think about like an ERP, right? Like SAP is nominally a software product, but an SAP deployment is like twelve months. Right. It's like services masquerading as software. The problem of going into a company and understanding how their specific financial reporting works and the associated rules is ⁓ probably harder than HGI. Like I'm I'm only very slightly exaggerating. Like it's it's really hard. ⁓ and so this is why so much of our business is built around understanding company context and codifying those rules. There is not and probably cannot exist a general purpose agent that doesn't need guidance on the company rules that can operate in complex organizations, financial environment correctly. Double clicking on a question that you had before on the frontier and pain points on eBails. So I think I think you said qualitative is a lot harder than quantitative for for the obvious reasons. Also heard that obviously you cannot really see what happens after the tool call, you know, judgment, sub agents and so on unless you have observability on those and so on. You know, you mentioned playbooks. I was wondering whether you know kind of really hard playbooks than one may have. Not thinking necessarily about ten hours of running time and I'm thinking about branching the playbook, coming back. You know, you have some playbooks, even in customer services, they're really, really hard, right? To solve an issue even with a customer and so on. Is the tech now ready for these kind of use cases? There's still something to build, ⁓ are we done there? What what are your thoughts? I mean we when we go and we scope an agent for a customer, we think of there being like basically three buckets of agents you can Category one agents, basically the capability is in the model parameter, and a frontier model could like largely single shot the task. And this is mainly just a data integration and like an eval and follow the rules problem. You have other things which are not yet in the model parameters, and like no amount of prompt engineering and subagent specification context management is going to make it able to solve that problem. Like the highest value business cases are in the middle, right? They're Problems where they are not yet fully in the parameters, like you couldn't single shot them with a frontier model, but with the right subagent design and prompt engineering and like associated data engineering, you can get the models to actually perform the task in an agentic architecture. I think really complex customer service playbooks are are in that middle area. Like you can do it, but it takes real work and codification of procedures and creating evals. And mapping out the sub agents to to get there. So it's those frontier problems where we spend a lot of our time figuring out the deployment. ⁓ yeah. So I was just wondering, it sounds like whenever an agent is not performing well in eval, how you fix it is kind of fixing the harness or context layer. So I was just kind of wondering, has there have you guys ever encountered a situation where no matter how you fix the prompting or sub agents or harness or whatever? It seems like the bottleneck is more in the base model intelligence instead of, you know, whatever harness you do. And I'm just curious like what specific domains you see that kind of problem in. Yeah, I have a baseline guideline when people try to build anything around is is try the the best ever model, like the the state of the art. And if you can't do it, just just give up. Like there's
no there's a small world probably not not be able to do it. So if this is a limitation, then maybe your your what you're trying to do is not yet ⁓ possible, right? So yeah, that's one guideline that gives people. ⁓ I would say this happens a lot. Like I think it is it is relatively common that you get into a particularly an ambitious problem and you determine like single shot agents for that task won't work. Either because it can't produce the output, but more commonly because the reliability's not there or like it the context isn't sufficient. Sometimes you can solve this with sub engineering, right? Like you can decompose the problems into sub agents. You can build agents for portions of the workflow. There may be real engineering work around it. Like if you have context management or data access issues, doing the actual engineering work to manage that for the agent can free up a lot of context space and token budget. But but I completely agree. Like not, you know, a meaningful percentage of the projects we scope out. The answer is, you know, check back in three months and ⁓ we'll have AGI. But right now it's too hard. Hi. I had a question on the things you talked about where you're trying to extract knowledge from the experts and you need it in a format that's useful for the agent. So in your experience, what does good look like for that format? And also do you have any lessons learned in terms of you collected all the context from experts and then you realize that the agent wasn't refined enough to build what they wanted? You had to change the format and sort of Yeah. I mean, one of the questions we ask in an interview, like this is a tip if you interview at Mercor, someone will probably ask you this. ⁓ what does an eval look like? Like if I if I gave you a file, like what what would that file be? Surprisingly high percentage of people like go in the interviews and and don't know the answer to that question. ⁓ but you need to ask experts questions in a way that helps them give you what you need to write the evals, right? So if it's like, Just riff on what good and bad looks like. That's often insufficient. It's like what are the 10 things you would look at first to determine if this output was done incorrectly? Structured extraction accomplishes a lot of that. And then you need a few iterations of this. Like you're rarely gonna single shot from an expert everything you need. And you you need to bounce back and forth a bit, like try outputs and go back to the expert. So there's a there's a real art to this. Like it is not a trivial problem to like get the context out of a company and put it into an agent. Just a follow up. So I guess this is a typical product manager's role in the current world as well, right? When you work with engineers, is is there something different in terms of the details of the format you need for an agent build? It is more about well, there there's two dimensions to this. There's like actually writing the evals and you need to people in Mercore go through training. Like what do good evals look like, right? Like there's real expertise in how you do this correctly. It's not like trivial or obvious. But separately, there's also expertise in how you actually specify agents. Like what should be a subagent? How should the prompt be written? That that itself is like an area of expertise. So I think of both of those as like skills that need to be learned rather than something you can trust a model to do for you. Hi, my name is Kave. So I'm building something of a startup. ⁓ I'm gonna give you a vague question. If you have any thoughts, feel free to share. we're helping people write books with AI. And so you think of two different levels set. One, to make it so what they've written is gonna be rewritten good enough that it seems like a professional author. And then two, to emulate the style of a famous author like Shakespeare, whatever. So I think evaluating that on the fly and adjusting it is gonna be very important. I wanna know if you have any ideas or approaches for how I go about that. The good news is that you have a huge amount of data out there, like all those books that exist out there, or there was Shakespeare or, you know, Steinbeck or whatever author. So you can start by building eval datasets based on that. And then you probably are going to need some amount of some LM judges to compare styles, because obviously you can't really do this with ⁓ heuristics of compare like is does this sound like Shakespeare? But I would start building data sets of the styles that you're trying to mimic based on Publicly available data, obviously, all unless you want to pay for it. So that's one way to start, I would guess. All right. Unless you have a follow up question. I mean, anything's helpful. I appreciate it. Cool. ⁓ yeah, that's all the time we have for today. I'm sure you guys will be here for a few more minutes in case people want to come up and ask you questions. But yeah, let's give a round of applause to our ⁓ speakers.