00:00:07 Rob: Hey everyone, I'm Rob May, I'm the CEO of Neurometric.
00:00:09 Cooper: And I'm Cooper, COO here at Neurometric. We're co-founders and this is Inference Time Tactics, a podcast for builders, researchers, and engineers who want to understand what's happening in this next era of AI.
00:00:22 Rob: We're joined today by our other co-founder and CTO, Byron Galbraith. And we're going to get into a little bit about what we're building here Neurometric and why it matters and how teams can evaluate these systems in the wild, particularly as it comes to business data and sort of inference time evaluation.
00:00:42 Cooper: To that end, let's start with CRMArena-Pro from Salesforce, which put a spotlight on enterprise agent reliability. It showed that top model scored about 58% on single-turn tasks and about 35% on multi-turn. So Byron, first, could you provide some background on CRMArena for context and then expand on what these numbers actually tell us about agent reliability?
00:01:06 Byron: For sure, CRMArena is a benchmarking platform suite of tools that was put out by Salesforce research. And they, like many other people working in the space who are focused on practitioners realize that a lot of these academic benchmarks like sort of, you know, ARC-AGI or some of these math problems these reasoning tasks, that they probed certain concepts of intelligence but they didn't really mean anything when it came to solving actual real world tasks like asking questions that could be answered by sales rep or customer support rep out of a CRM.
00:01:48 Byron: And so they put this suite of tools and tasks together to attempt to probe this idea of if I had an actual LLM powered agent that was trying to help me do sales and support tasks out of say, Salesforce for instance, how good could it actually be. And so they created a whole synthetic data set that represented a fictitious company with a fictitious CRM populated with all kinds of data. And then they created a bunch of subtasks that probed a variety of different things like workflow execution, policy compliance, sort of information retrieval, database querying, doing numerical computation, finding, you know, information like, Hey, you know, which sales rep had the most sales in the past quarter or, you know, what month did we see the highest churn rate for a particular product line?
00:02:41 Byron: And so you now have all of these tasks and the idea is can I give this to an LLM powered agent to then answer successfully? And this, you know, again, very different than just sort of arbitrary, generic abstract concepts. And what they found is even probing many of the leading models today or in the last few months, the success rate on many of these tasks was very… the variance was very high. And so some, it could do pretty well. You could get upwards of like 80% on a specific sub task and others would just really struggle. And no model did better on, you know, was consistently the top performing model on everything and different techniques had different outcomes.
00:03:27 Byron: And so this kind of goes back to the challenge of for practitioners, for businesses looking to implement sort of agentic workflows or to really get real augmentation in their internal processes, there’s a huge open space out there still. There is no just like go grab a model off the shelf, go get the best model from whatever provider you like and hook it up. Like that just will not succeed. And so this is really telling because it helps people understand really the complexity of the space and how to go about doing things here.
00:04:00 Cooper: And so this is something we're really interested in at Neurometric is going beyond academic settings and into real world enterprise instances of agentic AI. And so as we dig into implementation, Rob, do you want to set the stage a bit on what we're releasing and why this matters?
00:04:20 Rob: Yeah, so there's two pieces to the early Neurometric software and one you can think of as… we'll split it like front end and back end. And so on the back end, you can think of it as an engine that allows you to test different models, different inference time compute tactics, algorithms against business types of eval data. So that you can start to figure out if a best-of-end algorithm or a chain of thought algorithm or something like that would run on the models that you intend to use. And hopefully it'll help you to really spin up a bunch of examples of that and try lots of things.
00:04:59 Rob: And that's valuable because the the trend that we're seeing in a lot of these inference architectures is people start and they build something on one giant model and then they slowly start to peel off the pieces of their infrastructure, their agentic workflows, whatever that don't need that models full capabilities, right? And they do this because they can those models can be access limited sometimes and throttled. Sometimes they do it because it's just for expense reasons or something like that. And so you take a company that builds everything on GPT-4, GPT-5, whatever the latest thing is that they're using, and you can turn around and start to say, this piece of the workflow could actually use a small language model, or this could use GPT-3, or this could use something open source. So we can help people with those determinations. So that's the backend piece.
00:05:47 Rob: And the front end piece is a really cool visual drag and drop interface that allows you to test things out very simply. So if you're setting up a system that has an LLM as judge evaluation point, you can decide which LLM to use as the judge and you can try and run it really quickly and it's really easily and doesn't require any coding. So yeah, that piece will come out a little later, sort of like late next week probably. But who knows when this podcast gets posted? Maybe it'll be out. We'll see. But yeah, those are the two pieces. It’s going to be pretty cool and pretty useful to a lot of companies.
00:06:23 Cooper: So as we've gotten into implementation, Byron, do you want to talk about what we learned in practice and what it took?
00:06:29 Byron: One of the big challenges I think a lot of companies face, especially if they're trying to get into using these tools right of the way and to sort of jump in, the first thing you think is, oh, it's just prompting, right? It's just building this stuff up. Get some prompts. Maybe you hook up a database or something. But even just that hooking up a database is challenging. There's a lot that goes into this flow. And so the first thing we sort of wanted to do was building out this generic sort of framework backend for running these sorts of evaluations to help you understand and understand how these techniques work.
00:07:02 Byron: And so one of the things we saw with CRMArena is that it is inherently in sort of the agentic realm in that you cannot do any of these tasks without calling an external database which is a very common application for doing business deployment. And so you have to be able to manage this back and forth interaction between a query that comes in, having the LLM correctly go and get the data it needs, and then to use that data as part of its then further response and then potentially have multiple rounds to go through this.
00:07:41 Byron: And so one of the things we found is then just getting that hooked up and running effectively and performantly does take quite a lot of work. It is not something that, you know, you can just kind of do a drag-and-drop sort of thing on your own to get running. And so getting these things to work in a way that can then eventually be deployable is… got a lot that goes into it.
00:08:07 Byron: The other thing that we discovered is again having to build out this sort of agentic system where you do have these kinds of components, it definitely adds to the complexity because now it's not just say a single prompt and then you get something out at the end. It's potentially multiple prompts that have to be called and generated at different times depending on different outcomes. And so getting that integrated into our system did take quite a bit of effort.
00:08:36 Byron: The other thing that we wanted to do as part of this was to think through, you know, as you're trying to gauge how well this is working, it's inherently important to understand that large language models are machine learning models and they are inherently stochastic. They're random. And that's almost counter to a lot of business application use cases. Like you want reliability, you want things to work the same every time, that every customer is going to get the same result, that every agent is going to get the same answer to the same kind of question.
00:09:09 Byron: But the reality is that there is no like 100% guarantee of that. And so we had to work around and we wanted to make sure that as part of our evaluation, as part of our exploration of these tools that we do test for this that we do look at, you know… it's not just we ran it once and we got some number out. It’s we run it many, many, many times. The exact same thing’s fixed and you still do see a decent amount of variance in the outputs. And so this is something that is something companies definitely will need to work with and that we're trying to help make you understand, like how to work with this.
00:09:45 Byron: And that's actually where I would say some of these test time compute ideas we're talking about, like Rob mentioned, best of N, or we look at some of these other things where you probe the model many times, that's in many parts to help smooth out that variance and to make it better and more reliable.
00:10:01 Cooper: And how stable were these results across runs?
00:10:04 Byron: Yeah, exactly. So even… we just took one of the subtasks in this… activity priority understanding, for instance. So just simple tasks. You had to come up with like one of five different answers given the data that you pulled out of the database. And we ran a subset of all the tasks they had. There was 20 different ones we ran. Ran them five times. Everything was fixed. The model was fixed. The technique was fixed. Everything else was remained the same. And we still saw a significant variance from run to run.
00:10:38 Byron: So we saw things where you could have anywhere from, you know, like maybe 90% of all of those queries were even answerable. Like it just… sometimes the agent won't even answer. It'll just fail. It'll run out of time or some other constraint will cap it off. But you know, we saw some runs could get words of like 90%... would even answer. And of those 90%, most of them would be correct, but not all. And so you would see ranges at the end result of like anywhere from like 60 to 80% correct just on a small subset of the exact same thing.
00:11:19 Byron: And so this really does lead to this notion that even if you did one of these little like, oh, I did a little run, I did a test, I saw some things and it came out well, it could be that the next time you run it, exact same thing, you'll get a different answer. You may also find, and one of the key things that is also really important in the test time compute world, is identifying the complexity of the problem. So even though the task is the same, we noticed that there were certain segments that we could break this into where there were certain queries that it just never got right. And there are ones that seem to always get right.
00:11:55 Byron: And so you can imagine worlds where part of your whole flow is some sort of prediction routing step. It kind of says, oh, hey, wait, this question… we always know that this model or this technique will always solve it or will solve it with very, very, very high accuracy. So we can route it to that and you can be really confident. Anything that might have a lower variance, you know, higher variance, so lower consistent result, you could then route to some other process or even a human or something if you needed a certain operational threshold.
00:12:26 Byron: And so it's just even small tasks like this. really represents the fact that you cannot… you can't just go off of these benchmark numbers and say, well, you know, GPT-5 got a slightly higher number on this than someone else. Therefore, it's going to give me, you know, it's going to solve all my problems. There's many research, our findings and others that have been published that basically say the opposite is true. That these numbers, they're very, very weak proxies for actual operational success.
00:12:57 Cooper: And were there any production constraints that jumped out as particularly frustrating when pushing complexity?
00:13:03 Byron: One of the main things we run into, and I think this is true for any agentic system, this is true for any process that's going to be calling the LLMs many times, is latency. And then also cost, potentially, if you are using an external provider. Every time you call a model, you're going to incur more latency because you have to now wait for it to process the input, wait for it to start producing output and then the time it takes to get all those tokens back to you. Or in the case of an agentic system, those tokens will have to go into say a execution step like, oh generate SQL or SOQL that's going to now go to my Salesforce instance. And you have to wait for that to run, come back. If that fails, I got to try again, you know, maybe with some correction, do it again.
00:13:51 Byron: And so we see that you get high amounts of… potentially high amounts of latency. And so designing systems to be really operationally efficient at throughput is important. And so that's one thing that we initially ran into as well is that if you're… say using a third party, say Amazon Bedrock, and there are rate limits involved, suddenly you get a kind of a hard limit on the number of queries that you can do. So trying to balance like this effectiveness, the ability to call models multiple times, maybe use different models, a whole ensemble, this really gets… the complexity of getting these things to run and also at a high throughput rate really, really stood out to us.
00:14:37 Cooper: Yeah. So a pretty timely topic came out this week around vibe testing. How should teams evaluate these systems?
00:14:47 Rob: Yeah. Vibe coding and vibe testing, I like… I'm not a fan at scale, right? Like they're good for prototypes and it's good for triage and it's good for stuff like that. But I think you still need good engineering practices and I think those are being developed in how you think about actually building and coding and deploying all kinds of systems in this AI world. I mean, one of the things that happens a lot with the systems… I think people sometimes just sort of like accept inference as it is what it is. I'm going to deploy on this platform. I don't think they realize how many knobs you have to twist and turn to get the outcomes that you want for your system in terms of the performance accuracy, the latency, the cost. And there's so much you can do.
00:15:37 Rob: And it's like any other engineering problem, right? How many models do you want to use? How many options do you want to have? How do want to manage all this? What does it take to maintain it long-term? What does take to build it in the first place? Even with all the vibe coding and all the AI, we're not escaping that world.
00:15:55 Rob: And when it comes to testing, think ah a lot of the stuff that we're building and launching, we hope to open source soon and to be able to have a… you can think of it as a harness where you can plug in lot of your own evaluation data and run a lot of these things against it to see what works best for you… and you know, things like that. Because I do think a lot of companies aren't going to want to share that data with third-party providers.
00:16:21 Cooper: And some of those providers are releasing tools like Stacks, right, experimental dev tools, to take some of the headache out of evaluation. But there were some limitations there, Byron, I know you were mentioning.
0:16:33 Byron: Yeah, so Stacks was just announced by Google as a sort of evaluation tool. Again this idea of like, you know vibe testing how do we do our prompt engineering or context engineering as people are calling it now, or all this. And so what Google put out was this new tool called Stacks, which is essentially a way to challenge – you know, to help with this, the major, major challenge of evaluation. As we were talking about that, how do I know that my prompt really works? How do I know how it works across models? Let's say I'm not doing anything with test time compute at all. All I'm doing is just changing the model. Keeping the prompt fixed, I want to know how this works.
00:17:16 Byron: And the challenge with all of this, any of these knobs, all the knobs, all the levers is you have to have good evaluation. And that evaluation starts with a data set of here's what I expect to have in, and here's what I expect to have come out. This is the good input, this is the good output. And building these is really actually resource intensive for your internal application. That's why we all reach for these benchmarks, because someone went through the effort of doing it.
00:17:46 Byron: So what Stacks does is it's… I will say it's yet another evaluation building tool. There's been many, many, many of these that have been built over the years. Basically, ever since someone was trying to deploy machine learning in production environments, you go back pre-LLM era, you know, the last 10 years, there have been many iterations of tools just like this but geared toward different, you know, oh, computer vision type of paradigms or other sorts of machine learning and AI tasks.
00:18:18 Byron: So what Google Stacks does, which is unique, I would say the novel component of what they've put out that I can see so far is that number one, what makes the evaluation process so difficult is building up that good golden label data set. And this is time consuming and expensive because someone's got to read the results. They got to evaluate them. How do you make this go quickly, especially when you're looking at text? You know, it's one thing to look at an image or a classification label and it can go real fast. But if I'm trying to evaluate, say, turn this, you know, turn this long document into a summary or pull this piece of information out of this other piece of information, it's much harder.
00:19:01 Byron: And so one of the things they've added is this – and Rob alluded to this idea early, which is this LLM as a judge. So actually, why don't we have an AI answer if this is a good outcome or not? Now that has a little bit of fuzziness to it because you still kind of want a human, you know, a domain expert to really sign off on it. But for many, many, many of these things, having an LLM that can be sort of prompted or directionally steered or even fine tuned towards answering these things can be quite valuable and gets you a lot farther than just having an individual do it.
00:19:38 Byron: And so what Stacks does is it comes with a bunch of these evaluation… automatic evaluators out of the box, which is really cool because that really helps speed things up. And then they let you kind of design your own additional evaluators which you're going to have a chance to play with that directly yet. But it does look like maybe you can do some of your own prompting to kind of design in direct… a more powerful model to try to do this for you. And then you can build up and have this evaluation set for yourself.
00:20:07 Byron: So I think it's really cool that they're doing this, but I will say again, it's, in my mind, also doesn't really go anywhere near far enough because it really is just around that sort of tweaking a prompt or maybe swapping one model out for another. And as we've just talked about, there's so much more that goes into it. There's so many more knobs and dials that you can adjust than simply what singular model am I using for everything? And so that's where I think, again, what we're doing is going to add significant value to this space. But it's really exciting to see people like Google putting these tools out because it's very clear that this is an absolute need that people are running into.
00:20:45 Cooper: Absolutely. So we've got this release coming up. What's going to be the main takeaway and where are we taking this next?
00:20:52 Byron: We're absolutely aiming to let you, you know, out of the box, the idea is to let you play with the CRMArena benchmark in a way that, you know, solves all the evaluation part for you because Salesforce already did it. But to really let you have a way to tweak and tune and understand and maybe just dig into certain tasks that you care about. And then we're to have this environment that you can play with. We alluded to it again, this sort of front end that really explores and lets you see the different recipes, the different knobs and dials and how you can set them up and the results that go with them so you can kind of maybe better evaluate which model or which suite of models makes sense for you.
00:21:31 Rob: Yeah, and then I think we're… you know, as we expand some of this, we're in talks with some people to find some other business-related data sets and build some of those data sets. I think over time you'll see a public leaderboard that we'll put out that's related to sort of inference time, algorithms and frameworks that you can use, and how they perform on some of this real world data. And so I hope it’ll do similar to what Stacks does for models, which is let people quickly figure out and triage what things you might be able to run on your specific AI workloads that can have an impact. That should be really, really helpful.
00:22:14 Cooper: I'm really excited about some of the innovations on the user interface and experience. That's going to make this a lot easier to understand and interact with. So… stay tuned. That's it for today. If you're building agent infrastructure or exploring inference time strategies, please reach out and we'll share what we're building next. Be sure to like and subscribe to Inference Time Tactics. Thanks for joining us.
00:22:37 Rob: Yeah. And as always, thanks for listening, you guys.
00:00:09 Cooper: And I'm Cooper, COO here at Neurometric. We're co-founders and this is Inference Time Tactics, a podcast for builders, researchers, and engineers who want to understand what's happening in this next era of AI.
00:00:22 Rob: We're joined today by our other co-founder and CTO, Byron Galbraith. And we're going to get into a little bit about what we're building here Neurometric and why it matters and how teams can evaluate these systems in the wild, particularly as it comes to business data and sort of inference time evaluation.
00:00:42 Cooper: To that end, let's start with CRMArena-Pro from Salesforce, which put a spotlight on enterprise agent reliability. It showed that top model scored about 58% on single-turn tasks and about 35% on multi-turn. So Byron, first, could you provide some background on CRMArena for context and then expand on what these numbers actually tell us about agent reliability?
00:01:06 Byron: For sure, CRMArena is a benchmarking platform suite of tools that was put out by Salesforce research. And they, like many other people working in the space who are focused on practitioners realize that a lot of these academic benchmarks like sort of, you know, ARC-AGI or some of these math problems these reasoning tasks, that they probed certain concepts of intelligence but they didn't really mean anything when it came to solving actual real world tasks like asking questions that could be answered by sales rep or customer support rep out of a CRM.
00:01:48 Byron: And so they put this suite of tools and tasks together to attempt to probe this idea of if I had an actual LLM powered agent that was trying to help me do sales and support tasks out of say, Salesforce for instance, how good could it actually be. And so they created a whole synthetic data set that represented a fictitious company with a fictitious CRM populated with all kinds of data. And then they created a bunch of subtasks that probed a variety of different things like workflow execution, policy compliance, sort of information retrieval, database querying, doing numerical computation, finding, you know, information like, Hey, you know, which sales rep had the most sales in the past quarter or, you know, what month did we see the highest churn rate for a particular product line?
00:02:41 Byron: And so you now have all of these tasks and the idea is can I give this to an LLM powered agent to then answer successfully? And this, you know, again, very different than just sort of arbitrary, generic abstract concepts. And what they found is even probing many of the leading models today or in the last few months, the success rate on many of these tasks was very… the variance was very high. And so some, it could do pretty well. You could get upwards of like 80% on a specific sub task and others would just really struggle. And no model did better on, you know, was consistently the top performing model on everything and different techniques had different outcomes.
00:03:27 Byron: And so this kind of goes back to the challenge of for practitioners, for businesses looking to implement sort of agentic workflows or to really get real augmentation in their internal processes, there’s a huge open space out there still. There is no just like go grab a model off the shelf, go get the best model from whatever provider you like and hook it up. Like that just will not succeed. And so this is really telling because it helps people understand really the complexity of the space and how to go about doing things here.
00:04:00 Cooper: And so this is something we're really interested in at Neurometric is going beyond academic settings and into real world enterprise instances of agentic AI. And so as we dig into implementation, Rob, do you want to set the stage a bit on what we're releasing and why this matters?
00:04:20 Rob: Yeah, so there's two pieces to the early Neurometric software and one you can think of as… we'll split it like front end and back end. And so on the back end, you can think of it as an engine that allows you to test different models, different inference time compute tactics, algorithms against business types of eval data. So that you can start to figure out if a best-of-end algorithm or a chain of thought algorithm or something like that would run on the models that you intend to use. And hopefully it'll help you to really spin up a bunch of examples of that and try lots of things.
00:04:59 Rob: And that's valuable because the the trend that we're seeing in a lot of these inference architectures is people start and they build something on one giant model and then they slowly start to peel off the pieces of their infrastructure, their agentic workflows, whatever that don't need that models full capabilities, right? And they do this because they can those models can be access limited sometimes and throttled. Sometimes they do it because it's just for expense reasons or something like that. And so you take a company that builds everything on GPT-4, GPT-5, whatever the latest thing is that they're using, and you can turn around and start to say, this piece of the workflow could actually use a small language model, or this could use GPT-3, or this could use something open source. So we can help people with those determinations. So that's the backend piece.
00:05:47 Rob: And the front end piece is a really cool visual drag and drop interface that allows you to test things out very simply. So if you're setting up a system that has an LLM as judge evaluation point, you can decide which LLM to use as the judge and you can try and run it really quickly and it's really easily and doesn't require any coding. So yeah, that piece will come out a little later, sort of like late next week probably. But who knows when this podcast gets posted? Maybe it'll be out. We'll see. But yeah, those are the two pieces. It’s going to be pretty cool and pretty useful to a lot of companies.
00:06:23 Cooper: So as we've gotten into implementation, Byron, do you want to talk about what we learned in practice and what it took?
00:06:29 Byron: One of the big challenges I think a lot of companies face, especially if they're trying to get into using these tools right of the way and to sort of jump in, the first thing you think is, oh, it's just prompting, right? It's just building this stuff up. Get some prompts. Maybe you hook up a database or something. But even just that hooking up a database is challenging. There's a lot that goes into this flow. And so the first thing we sort of wanted to do was building out this generic sort of framework backend for running these sorts of evaluations to help you understand and understand how these techniques work.
00:07:02 Byron: And so one of the things we saw with CRMArena is that it is inherently in sort of the agentic realm in that you cannot do any of these tasks without calling an external database which is a very common application for doing business deployment. And so you have to be able to manage this back and forth interaction between a query that comes in, having the LLM correctly go and get the data it needs, and then to use that data as part of its then further response and then potentially have multiple rounds to go through this.
00:07:41 Byron: And so one of the things we found is then just getting that hooked up and running effectively and performantly does take quite a lot of work. It is not something that, you know, you can just kind of do a drag-and-drop sort of thing on your own to get running. And so getting these things to work in a way that can then eventually be deployable is… got a lot that goes into it.
00:08:07 Byron: The other thing that we discovered is again having to build out this sort of agentic system where you do have these kinds of components, it definitely adds to the complexity because now it's not just say a single prompt and then you get something out at the end. It's potentially multiple prompts that have to be called and generated at different times depending on different outcomes. And so getting that integrated into our system did take quite a bit of effort.
00:08:36 Byron: The other thing that we wanted to do as part of this was to think through, you know, as you're trying to gauge how well this is working, it's inherently important to understand that large language models are machine learning models and they are inherently stochastic. They're random. And that's almost counter to a lot of business application use cases. Like you want reliability, you want things to work the same every time, that every customer is going to get the same result, that every agent is going to get the same answer to the same kind of question.
00:09:09 Byron: But the reality is that there is no like 100% guarantee of that. And so we had to work around and we wanted to make sure that as part of our evaluation, as part of our exploration of these tools that we do test for this that we do look at, you know… it's not just we ran it once and we got some number out. It’s we run it many, many, many times. The exact same thing’s fixed and you still do see a decent amount of variance in the outputs. And so this is something that is something companies definitely will need to work with and that we're trying to help make you understand, like how to work with this.
00:09:45 Byron: And that's actually where I would say some of these test time compute ideas we're talking about, like Rob mentioned, best of N, or we look at some of these other things where you probe the model many times, that's in many parts to help smooth out that variance and to make it better and more reliable.
00:10:01 Cooper: And how stable were these results across runs?
00:10:04 Byron: Yeah, exactly. So even… we just took one of the subtasks in this… activity priority understanding, for instance. So just simple tasks. You had to come up with like one of five different answers given the data that you pulled out of the database. And we ran a subset of all the tasks they had. There was 20 different ones we ran. Ran them five times. Everything was fixed. The model was fixed. The technique was fixed. Everything else was remained the same. And we still saw a significant variance from run to run.
00:10:38 Byron: So we saw things where you could have anywhere from, you know, like maybe 90% of all of those queries were even answerable. Like it just… sometimes the agent won't even answer. It'll just fail. It'll run out of time or some other constraint will cap it off. But you know, we saw some runs could get words of like 90%... would even answer. And of those 90%, most of them would be correct, but not all. And so you would see ranges at the end result of like anywhere from like 60 to 80% correct just on a small subset of the exact same thing.
00:11:19 Byron: And so this really does lead to this notion that even if you did one of these little like, oh, I did a little run, I did a test, I saw some things and it came out well, it could be that the next time you run it, exact same thing, you'll get a different answer. You may also find, and one of the key things that is also really important in the test time compute world, is identifying the complexity of the problem. So even though the task is the same, we noticed that there were certain segments that we could break this into where there were certain queries that it just never got right. And there are ones that seem to always get right.
00:11:55 Byron: And so you can imagine worlds where part of your whole flow is some sort of prediction routing step. It kind of says, oh, hey, wait, this question… we always know that this model or this technique will always solve it or will solve it with very, very, very high accuracy. So we can route it to that and you can be really confident. Anything that might have a lower variance, you know, higher variance, so lower consistent result, you could then route to some other process or even a human or something if you needed a certain operational threshold.
00:12:26 Byron: And so it's just even small tasks like this. really represents the fact that you cannot… you can't just go off of these benchmark numbers and say, well, you know, GPT-5 got a slightly higher number on this than someone else. Therefore, it's going to give me, you know, it's going to solve all my problems. There's many research, our findings and others that have been published that basically say the opposite is true. That these numbers, they're very, very weak proxies for actual operational success.
00:12:57 Cooper: And were there any production constraints that jumped out as particularly frustrating when pushing complexity?
00:13:03 Byron: One of the main things we run into, and I think this is true for any agentic system, this is true for any process that's going to be calling the LLMs many times, is latency. And then also cost, potentially, if you are using an external provider. Every time you call a model, you're going to incur more latency because you have to now wait for it to process the input, wait for it to start producing output and then the time it takes to get all those tokens back to you. Or in the case of an agentic system, those tokens will have to go into say a execution step like, oh generate SQL or SOQL that's going to now go to my Salesforce instance. And you have to wait for that to run, come back. If that fails, I got to try again, you know, maybe with some correction, do it again.
00:13:51 Byron: And so we see that you get high amounts of… potentially high amounts of latency. And so designing systems to be really operationally efficient at throughput is important. And so that's one thing that we initially ran into as well is that if you're… say using a third party, say Amazon Bedrock, and there are rate limits involved, suddenly you get a kind of a hard limit on the number of queries that you can do. So trying to balance like this effectiveness, the ability to call models multiple times, maybe use different models, a whole ensemble, this really gets… the complexity of getting these things to run and also at a high throughput rate really, really stood out to us.
00:14:37 Cooper: Yeah. So a pretty timely topic came out this week around vibe testing. How should teams evaluate these systems?
00:14:47 Rob: Yeah. Vibe coding and vibe testing, I like… I'm not a fan at scale, right? Like they're good for prototypes and it's good for triage and it's good for stuff like that. But I think you still need good engineering practices and I think those are being developed in how you think about actually building and coding and deploying all kinds of systems in this AI world. I mean, one of the things that happens a lot with the systems… I think people sometimes just sort of like accept inference as it is what it is. I'm going to deploy on this platform. I don't think they realize how many knobs you have to twist and turn to get the outcomes that you want for your system in terms of the performance accuracy, the latency, the cost. And there's so much you can do.
00:15:37 Rob: And it's like any other engineering problem, right? How many models do you want to use? How many options do you want to have? How do want to manage all this? What does it take to maintain it long-term? What does take to build it in the first place? Even with all the vibe coding and all the AI, we're not escaping that world.
00:15:55 Rob: And when it comes to testing, think ah a lot of the stuff that we're building and launching, we hope to open source soon and to be able to have a… you can think of it as a harness where you can plug in lot of your own evaluation data and run a lot of these things against it to see what works best for you… and you know, things like that. Because I do think a lot of companies aren't going to want to share that data with third-party providers.
00:16:21 Cooper: And some of those providers are releasing tools like Stacks, right, experimental dev tools, to take some of the headache out of evaluation. But there were some limitations there, Byron, I know you were mentioning.
0:16:33 Byron: Yeah, so Stacks was just announced by Google as a sort of evaluation tool. Again this idea of like, you know vibe testing how do we do our prompt engineering or context engineering as people are calling it now, or all this. And so what Google put out was this new tool called Stacks, which is essentially a way to challenge – you know, to help with this, the major, major challenge of evaluation. As we were talking about that, how do I know that my prompt really works? How do I know how it works across models? Let's say I'm not doing anything with test time compute at all. All I'm doing is just changing the model. Keeping the prompt fixed, I want to know how this works.
00:17:16 Byron: And the challenge with all of this, any of these knobs, all the knobs, all the levers is you have to have good evaluation. And that evaluation starts with a data set of here's what I expect to have in, and here's what I expect to have come out. This is the good input, this is the good output. And building these is really actually resource intensive for your internal application. That's why we all reach for these benchmarks, because someone went through the effort of doing it.
00:17:46 Byron: So what Stacks does is it's… I will say it's yet another evaluation building tool. There's been many, many, many of these that have been built over the years. Basically, ever since someone was trying to deploy machine learning in production environments, you go back pre-LLM era, you know, the last 10 years, there have been many iterations of tools just like this but geared toward different, you know, oh, computer vision type of paradigms or other sorts of machine learning and AI tasks.
00:18:18 Byron: So what Google Stacks does, which is unique, I would say the novel component of what they've put out that I can see so far is that number one, what makes the evaluation process so difficult is building up that good golden label data set. And this is time consuming and expensive because someone's got to read the results. They got to evaluate them. How do you make this go quickly, especially when you're looking at text? You know, it's one thing to look at an image or a classification label and it can go real fast. But if I'm trying to evaluate, say, turn this, you know, turn this long document into a summary or pull this piece of information out of this other piece of information, it's much harder.
00:19:01 Byron: And so one of the things they've added is this – and Rob alluded to this idea early, which is this LLM as a judge. So actually, why don't we have an AI answer if this is a good outcome or not? Now that has a little bit of fuzziness to it because you still kind of want a human, you know, a domain expert to really sign off on it. But for many, many, many of these things, having an LLM that can be sort of prompted or directionally steered or even fine tuned towards answering these things can be quite valuable and gets you a lot farther than just having an individual do it.
00:19:38 Byron: And so what Stacks does is it comes with a bunch of these evaluation… automatic evaluators out of the box, which is really cool because that really helps speed things up. And then they let you kind of design your own additional evaluators which you're going to have a chance to play with that directly yet. But it does look like maybe you can do some of your own prompting to kind of design in direct… a more powerful model to try to do this for you. And then you can build up and have this evaluation set for yourself.
00:20:07 Byron: So I think it's really cool that they're doing this, but I will say again, it's, in my mind, also doesn't really go anywhere near far enough because it really is just around that sort of tweaking a prompt or maybe swapping one model out for another. And as we've just talked about, there's so much more that goes into it. There's so many more knobs and dials that you can adjust than simply what singular model am I using for everything? And so that's where I think, again, what we're doing is going to add significant value to this space. But it's really exciting to see people like Google putting these tools out because it's very clear that this is an absolute need that people are running into.
00:20:45 Cooper: Absolutely. So we've got this release coming up. What's going to be the main takeaway and where are we taking this next?
00:20:52 Byron: We're absolutely aiming to let you, you know, out of the box, the idea is to let you play with the CRMArena benchmark in a way that, you know, solves all the evaluation part for you because Salesforce already did it. But to really let you have a way to tweak and tune and understand and maybe just dig into certain tasks that you care about. And then we're to have this environment that you can play with. We alluded to it again, this sort of front end that really explores and lets you see the different recipes, the different knobs and dials and how you can set them up and the results that go with them so you can kind of maybe better evaluate which model or which suite of models makes sense for you.
00:21:31 Rob: Yeah, and then I think we're… you know, as we expand some of this, we're in talks with some people to find some other business-related data sets and build some of those data sets. I think over time you'll see a public leaderboard that we'll put out that's related to sort of inference time, algorithms and frameworks that you can use, and how they perform on some of this real world data. And so I hope it’ll do similar to what Stacks does for models, which is let people quickly figure out and triage what things you might be able to run on your specific AI workloads that can have an impact. That should be really, really helpful.
00:22:14 Cooper: I'm really excited about some of the innovations on the user interface and experience. That's going to make this a lot easier to understand and interact with. So… stay tuned. That's it for today. If you're building agent infrastructure or exploring inference time strategies, please reach out and we'll share what we're building next. Be sure to like and subscribe to Inference Time Tactics. Thanks for joining us.
00:22:37 Rob: Yeah. And as always, thanks for listening, you guys.
