Harness Engineering for Reliable, Governed AI Agents: täielik transkriptsioon

tools, the models, the frameworks that I really care about and be able to move as fast as I as I would like. So I was enabled with the right tooling to just execute, 58 00:04:42.000 --> 00:04:46.400 but at the same time, it never left the sight of centralized governance. 59 00:04:46.800 --> 00:04:54.185 So I didn't have to worry about instrumenting my code. It would always log things that I care about. And similarly a central platform team, 60 00:04:54.425 --> 00:04:56.425 if they had to figure out any 61 00:04:56.985 --> 00:04:58.425 erroneous or 62 00:04:58.745 --> 00:04:59.625 malicious 63 00:04:59.625 --> 00:05:00.425 activity, 64 00:05:00.585 --> 00:05:06.200 they always had the control to check what's happening and be able to take care of that. So this entire dichotomy 65 00:05:06.200 --> 00:05:11.240 of whether there's a trade off between federated execution and centralized governance 66 00:05:11.240 --> 00:05:14.680 turned out to be false by looking at Meta's internal ML platform. 67 00:05:14.840 --> 00:05:19.480 And we want to take that learning and bring it to the rest of the organizations out there. 68 00:05:20.175 --> 00:05:25.535 So the whole space of agent gateways and agent harnesses kind of spans 69 00:05:26.414 --> 00:05:44.410 at least two of the major hype trends over the past year or two, where maybe a year or so ago, AI gateways were gaining a lot of attention. There were a new gateway probably every other day. Some of those have been consolidated into other systems. Some of them have ceased operation. 70 00:05:44.650 --> 00:05:46.889 Some of them are continuing to grow. 71 00:05:47.289 --> 00:05:52.945 And then the past six to eight months has seen a lot of focus on the idea of harness engineering 72 00:05:52.945 --> 00:05:55.345 and that being one of the key differentiators 73 00:05:55.345 --> 00:05:57.425 beyond the models themselves. 74 00:05:57.665 --> 00:06:03.425 And I'm wondering if you can just talk through some of the ways that you're navigating some of the 75 00:06:04.060 --> 00:06:19.020 challenges and complexities of being in such a crowded space and what you see as some of the benefits of working across both of those and some of the ways that having a harness and a gateway that work well together can be a force multiplier. 76 00:06:20.044 --> 00:06:21.085 Absolutely. 77 00:06:21.405 --> 00:06:33.405 Happy to zoom in. And by the way, I cannot agree more, like, you know, with the pace that everything else is changing. In in the startup ecosystem now, like, you there's this new saying. Right? Like, you know, that earlier people had to find PMF 78 00:06:33.720 --> 00:06:48.440 or the product market fit once. Now you are fundamentally finding product market fit every six months. Right? The space is changing that fast. But also the interesting thing around this is that this is what is creating the opportunity as well. So this is certainly a challenge for building, 79 00:06:48.520 --> 00:06:51.905 but this is an opportunity for operating, 80 00:06:51.905 --> 00:06:54.225 right, in the startup space. And the reason being, 81 00:06:54.465 --> 00:06:56.065 that think about what's happening 82 00:06:56.544 --> 00:07:03.185 with every consumer of call it like an AI gateway or a control plane layer, right. An organization 83 00:07:03.345 --> 00:07:05.170 builds out a 84 00:07:05.330 --> 00:07:06.290 centralized 85 00:07:06.610 --> 00:07:13.970 platform engineering team that's going to develop this AI gateway because they care about solving those five problems that I talked about earlier, right. 86 00:07:14.530 --> 00:07:16.850 They start with doing this, they 87 00:07:17.170 --> 00:07:19.890 build a thin proxy layer and they're like okay we are sorted. 88 00:07:20.495 --> 00:07:22.175 And then soon enough, 89 00:07:22.815 --> 00:07:27.535 the model proxy layer that they have developed, all the API signatures start changing out there in the market. 90 00:07:27.935 --> 00:07:30.495 A couple of months later, MCP is launched. 91 00:07:30.655 --> 00:07:46.720 And now they're like, oh, I need to bring my MCP's in the same proxy layer. And they have to solve for authentication and authorization and all that, right? And then A2A is launched. So they have to figure out this agent to agent communication protocol through the platform, right? Then multiple agents that their teams 92 00:07:47.040 --> 00:07:51.265 have developed hit production, so they care about ensuring uptime, reliability, 93 00:07:51.585 --> 00:07:53.265 latency optimization. 94 00:07:53.825 --> 00:07:59.105 And then agent identity becomes a real thing, right? So it's an ever changing space from their standpoint. 95 00:07:59.264 --> 00:08:02.225 New models keep getting launched, new protocols keep getting 96 00:08:02.870 --> 00:08:22.025 launched and the complexity and the requirement from this platform keeps on changing every single day, right. So for thousands of team out there whose main job is not building and maintaining a platform like this, it becomes a completely intractable problem to solve. Right? So we have seen that organizations 97 00:08:22.025 --> 00:08:26.185 went from what we call as this build, break, and then buy cycle. 98 00:08:26.425 --> 00:08:41.890 Right? And that has created an opportunity for a startup like us to build a platform like this. Now, of course, at our end, the complexity does not reduce, it only increases because now we have to deal with the stack of all of those thousands of enterprises. Right? While the space is ever changing. 99 00:08:42.545 --> 00:08:59.960 So so we have done a lot of optimizations at our end where we have built out internal agentic solution that keeps pulling what are the new things that are getting launched in the market, what's breaking in our platform, and how do we continuously keep optimizing this besides of course like the fact that there's a team that really cares about solving this problem statement. 100 00:09:00.200 --> 00:09:03.640 Same

thing for agent harness, I feel like this is relatively newer 101 00:09:03.880 --> 00:09:09.960 but we have seen a major demand from organizations that they care about being able to take control 102 00:09:10.605 --> 00:09:12.525 of the agents that are being developed. 103 00:09:13.084 --> 00:09:20.365 And you're just not able to take the control of the agents that that you're developing if you're using a closed proprietary 104 00:09:20.365 --> 00:09:31.810 agent development platform that you just don't have any control over. Right? So so that's where people want more of a vendor neutral harness and it's been fun to fun to work on this. 105 00:09:32.610 --> 00:09:40.834 The reason this becomes extremely interesting is vendor neutral agent harness by design is open to models from multiple providers. 106 00:09:41.154 --> 00:10:00.360 Right? So you are able to access frontier models from the labs, or you're able to work with frontier models that are open source. Right? And all of those models for all of the organizations that you work with are available through the gateway layer. So people can bring their own model extremely straightforward. Right? And this is extremely straightforward to bring their own models, basically. 107 00:10:01.080 --> 00:10:32.810 Similarly, people care about being able to bring their own MCP servers. Now the good news is that a lot of MCP servers that people want to access are these public MCP servers remotely hosted, centrally managed. So they don't have to do a lot of put in a lot of effort to try out different MCP servers. It's all connected through the platform of the gateway. Right? But then these MCP servers can be easily plugged into the the agent harness there and help them invoke these tools. And same thing, you can extend this to agents as well. Agents are all accessible through the through the gateway there. Right? 108 00:10:33.595 --> 00:10:39.115 Now across any of these models, MCP's and agents, if you wanted to apply any kind of guardrails, 109 00:10:39.115 --> 00:10:47.995 your guardrails are also available in your central control plane layer that gets implemented through the gateway, right. So fundamentally what's happening is when you're building your agents, 110 00:10:48.399 --> 00:10:52.320 all the raw building blocks that you need and the corresponding registries 111 00:10:52.399 --> 00:10:54.720 are available as part of your control plane. 112 00:10:55.200 --> 00:10:57.520 The logic to build these agents, 113 00:10:57.520 --> 00:10:58.240 right, the 114 00:10:58.720 --> 00:11:00.320 engine that operates 115 00:11:00.565 --> 00:11:14.725 the model and the tool and the agent loop that sits within the harness and it's able to help you orchestrate your calls better and the two together actually becomes a powerful force because now you're able to build agents that are accurate 116 00:11:14.965 --> 00:11:16.165 but cost efficient 117 00:11:16.460 --> 00:11:28.780 and centrally governed. So that's roughly the

unlock that we're getting by bringing the pieces together, if that makes sense. Yeah. It definitely does. And one of the other interesting pieces of the gateway is that 118 00:11:29.135 --> 00:11:35.455 one of the, I think, motivating factors, at least in the initial push, was to give a unified 119 00:11:35.775 --> 00:11:41.135 interface to all of the different models. But as the different providers have 120 00:11:41.440 --> 00:11:53.840 changed their APIs and as OpenAI itself has evolved their API, that that became a much more difficult moving target, as well as the fact that by forcing everything into that one shape, you potentially lose out on capabilities 121 00:11:53.840 --> 00:11:55.120 of Anthropic 122 00:11:55.120 --> 00:12:06.535 or DeepSeek or whomever else. So I'm wondering if you can just talk to some of the tensions there of having an easier onboarding where you just have one API contract to code against versus 123 00:12:06.695 --> 00:12:10.775 being able to actually take full advantage of the underlying providers. 124 00:12:12.070 --> 00:12:16.630 That's that's a very interesting question. There are two aspects to it that I would like to call out. Number one, 125 00:12:17.030 --> 00:12:22.950 that, of course, like, you know, with the API signatures of the model providers continuously changing, this becomes, 126 00:12:23.110 --> 00:12:29.285 like, you know, relatively difficult to to implement this. But to that end, what we have done is, 127 00:12:29.765 --> 00:12:32.565 for the most part, we do the job of 128 00:12:33.125 --> 00:12:35.125 unifying the API signatures 129 00:12:35.125 --> 00:12:44.990 across, like, you know, this like, the the diverging API signatures on the on the on the right side and bring them all with the unified access for our end developers. So we don't lose the capability 130 00:12:45.150 --> 00:12:55.070 of different model providers, but at the same time, we try to solve for, like, you know, bringing them all into a more unified API signature for our end users. So they don't have to worry about these waiting 131 00:12:55.390 --> 00:12:56.030 signatures. 132 00:12:56.585 --> 00:13:01.065 But at the same time, you're right that this is not possible to execute every single time. 133 00:13:01.465 --> 00:13:08.105 And for that, what we have done is we have also implemented what we call as a custom endpoint that you can invoke through the gateway. 134 00:13:08.505 --> 00:13:14.070 And in that case, like you do not have to be compliant with the gateway specific signature, 135 00:13:14.150 --> 00:13:20.630 and you're able to run them through through the gateway, you're able to get a lot of benefits of the gateway to the end customers. 136 00:13:20.950 --> 00:13:26.855 You obviously get full benefits of the model in that case, but you do lose some out of the box capability 137 00:13:26.855 --> 00:13:35.255 that the gateway is able to do so long as it understands the underlying model signatures. So I guess we have divided the platform into those two parts. 138 00:13:35.575 --> 00:13:36.775 One where you get complete 139 00:13:37.149 --> 00:13:40.190 benefit of the gateway and unified API signature, 140 00:13:40.270 --> 00:13:45.230 and the other case, you bring a custom signature and you don't get complete benefit of the gateway. 141 00:13:45.550 --> 00:13:48.510 Digging more into the harness itself, 142 00:13:49.470 --> 00:14:03.605 as we've noted, that's I think the most recent hype cycle that we're in where everybody's building a harness. It's become a very complex market. There are a few that have gained a decent amount of traction. I think pie is probably the most notable one 143 00:14:03.925 --> 00:14:06.644 largely due to the success of OpenClaw. 144 00:14:06.930 --> 00:14:14.130 I'm wondering if you can just talk through some of the key challenges and components of building a harness that actually works well. 145 00:14:14.850 --> 00:14:15.730 Absolutely. 146 00:14:15.970 --> 00:14:16.530 Absolutely. 147 00:14:16.850 --> 00:14:23.885 Our team has been having a fun time building out this harness. And the first thing is getting 148 00:14:25.404 --> 00:14:31.725 the accuracy right. Right? Getting the agent's accuracy right, getting ensuring that you are able to 149 00:14:32.765 --> 00:14:38.710 get the quality of responses that you get from some of these like you know out of the box harnesses, 150 00:14:38.790 --> 00:14:40.070 managed harnesses 151 00:14:40.150 --> 00:15:00.415 is actually a tough problem to solve. Right? You don't quickly implement a harness and get there. Right? But after that, once once you have gotten there, the the challenging aspect of harness is a lot of the companies that additional companies that you have to deal with to ensure that all of this is happening. Right? So for example, how do you how do you make sure that you give access 152 00:15:01.170 --> 00:15:06.370 to a sandbox to an environment and by the way like of course along with all the recent incidents, 153 00:15:06.529 --> 00:15:09.970 how do you check that the agent is truly sandboxed, 154 00:15:09.970 --> 00:15:11.010 right? It's not 155 00:15:11.570 --> 00:15:15.965 it's it's not able to get out of it. How do you check that the agent 156 00:15:16.045 --> 00:15:21.005 when it's when it's executing the code, it's actually executing it efficiently 157 00:15:21.085 --> 00:15:25.725 and in the right environment. Right? How do you make sure that which of the tasks should actually 158 00:15:26.230 --> 00:15:41.655 be done through coding and like when you do the memory management around that, done through coding, but it's it's done through the tokens directly and you have to build the memory management around that. So there's some of those complexities are actually extremely difficult. Then anything around durable execution, 159 00:15:41.655 --> 00:15:50.695 human in the loop that you have to ensure that certain tools you get feedback from the human beings and then only

you let the agent proceed forward 160 00:15:50.775 --> 00:15:51.815 is complex. 161 00:15:52.135 --> 00:15:52.775 And lastly, 162 00:15:53.550 --> 00:16:11.795 the the one thing that is extremely challenging with the harness layer is how do you manage permissions and access control through agent chaining. Right? Because what what's happening now is agents are often acting on behalf of users. Right? And what I mean by that is as a user, I can send a Slack message to Tobias 163 00:16:11.875 --> 00:16:21.074 or an agent acting on my behalf can send a Slack message to Tobias. Right? How does Tobias know? Whether this is directly coming from Nikunj or an agent acting on Nikunj's 164 00:16:21.074 --> 00:16:21.634 behalf. Right? 165 00:16:22.310 --> 00:16:23.990 And what happens 166 00:16:24.390 --> 00:16:34.070 if this agent invokes a sub agent invokes another sub agent in the chain basically. Right? And each of these different agents had their own identities and had their own access control. 167 00:16:34.565 --> 00:16:42.085 How do you ensure that the least privilege is managed throughout that entire chain? Right? So all of these, like, know, 168 00:16:42.885 --> 00:16:44.885 are actually fairly difficult to handle. 169 00:16:45.125 --> 00:16:45.685 And 170 00:16:46.404 --> 00:16:53.690 what lives in the harness layer of what has to live in a more centralized governance layer, the right thought process around that is also 171 00:16:54.009 --> 00:16:57.769 also something that that made things very complex to implement. 172 00:16:58.250 --> 00:17:01.449 The other interesting aspect of harnesses is that 173 00:17:01.845 --> 00:17:19.360 that term has become a little bit conflated because most people are thinking about it as far as their Claude code or codex or PI agent harness or any of the numerous other harnesses that are out there specifically for coding agents. But then there are also harnesses that are focused more on deployed 174 00:17:19.440 --> 00:17:32.895 production agents that are interacting with various other systems, and you want to make sure that those are more constrained than your local terminal agent might be and have different capabilities. And so I'm wondering if you can talk to some of the 175 00:17:33.535 --> 00:17:34.975 aspects of 176 00:17:35.295 --> 00:17:42.175 how to build that harness in a way that is layered and composable so that you can have that core functionality 177 00:17:42.175 --> 00:17:45.089 that's shared across all of the different use cases, 178 00:17:45.250 --> 00:17:46.690 but be able to actually 179 00:17:46.929 --> 00:17:53.809 exclude functionality or exclude permissions from the harness for cases where you want it to be operating in a more constrained environment. 180 00:17:54.929 --> 00:17:55.729 That makes 181 00:17:56.850 --> 00:17:57.490 a lot of sense. 182 00:17:58.985 --> 00:18:17.460 And and you're exactly right. Like, you know, first of all, you know, there's actually a lot of noise in the market. Like, people actually are not able to understand the the difference between all of these different products and the corresponding intended use cases with which they were they were developed. Right? From from our standpoint of the the stars that we have taken here is 183 00:18:17.940 --> 00:18:19.940 that we are focused on building 184 00:18:20.260 --> 00:18:40.394 more general purpose agent harness. Right? So the general productivity agent harness is what we are like, know, what we have taken a stance on building. But also like you know as you mentioned that we are developing this with an intent that we start with like this entire personal productivity type of an agent harness. Right? And eventually 185 00:18:40.555 --> 00:18:52.000 if we want to focus a little bit more on the on becoming a coding harness, we are able to steer the harness in that direction, essentially. The fundamental building blocks don't necessarily change between these different harnesses, 186 00:18:52.160 --> 00:19:11.125 but the way these components come together and, like, you know, how in one case you're always dealing with a constant context layer, let's say, from a coding side versus in the other case, like, you know, you get a lot of the context from your MCPs and stuff that you're connecting on the harness. Right? So how do you how how does the harness handle those nuances? 187 00:19:11.530 --> 00:19:24.490 Is something that, like, you know, we have basically kept it configurable from our standpoint. One thing, though, that I would mention that that has been that has been a big win, by the way, in exactly differentiating these between different types of harnesses 188 00:19:24.570 --> 00:19:29.274 is by not trying to be all in one, you actually gain a lot of efficiency. 189 00:19:30.075 --> 00:19:39.595 And what I mean by that is if you try to solve all the problems in one harness from the get go, then essentially, like, you think about all of your system prompt has 190 00:19:39.595 --> 00:19:41.274 to become massive. Right? 191 00:19:41.920 --> 00:20:09.749 And because a harness at any point in time might have to be dealing with any of the underlying, call it, tasks that you care about it's solving. And because of that and that system prompt, by the way, goes in every turn of the agent conversation. Like, you cannot avoid that. And when that starts to happen, like, start paying taxes for that. Right? So you are basically consuming more tokens in every task that you are that you are doing, and you are spending more dollars on every task that's that you're doing. 192 00:20:09.990 --> 00:20:20.630 And that has been actually one of the design decisions that we have made that the way we have designed the harness that we we are able to specialize this for different tasks and, like, you you don't have to do all in one at the same time. 193 00:20:21.855 --> 00:20:37.350 That said, I also wanted to share that because of this being one of the reasons, and actually this is one

of the reasons, not everything, we have seen that when we ran the benchmark of our harness against other tools out there like cloud managed agents and all, we were able to 194 00:20:37.750 --> 00:20:38.470 get 195 00:20:38.630 --> 00:20:42.470 over 30% cost reduction for a maintained accuracy 196 00:20:42.470 --> 00:20:44.630 while using this exact same model 197 00:20:44.870 --> 00:20:47.830 and as much as 70% 198 00:20:47.830 --> 00:21:03.075 cost reduction when we let the harness choose the models, like, you know, choose its own model essentially. Right? So it it also had that flexibility of being able to choose the right model from different providers and stuff. So that's how much range we have been able to achieve by, like, you know, 199 00:21:03.715 --> 00:21:05.875 on the task that the harness is accomplishing as well. 200 00:21:06.580 --> 00:21:09.300 On that concept of model routing, 201 00:21:09.300 --> 00:21:15.940 there's also been a lot of focus there. The project that I'm most familiar with in that space is the VLLM 202 00:21:15.940 --> 00:21:28.164 router that lets you dynamically choose which model to route to for a particular request or set of requests. And I'm curious whether that's something that you see is belonging in the harness itself, 203 00:21:28.245 --> 00:21:29.205 in the gateway, 204 00:21:29.525 --> 00:21:43.909 something in between, and just some of the aspects of how to make sure that you are effectively choosing which model is going to have the right result without burning a bunch of tokens on a failed result and then having to retry it with a different model. 205 00:21:45.110 --> 00:21:49.429 The way I see it, you have to have this kind of routing capability 206 00:21:50.134 --> 00:21:58.454 at this point in both places at least the way the industry is going right now. You have to have this in both places and to serve a different purpose in the two places. 207 00:21:59.095 --> 00:22:00.614 Right? A harness 208 00:22:01.095 --> 00:22:04.534 has to be able to figure out smartness 209 00:22:03.950 --> 00:22:10.350 when it's running the loop, right? That which agent I'm calling, which model am I calling, which tool am I calling, right? 210 00:22:10.910 --> 00:22:23.114 Around that time it also has to have certain level of smartness that I know that certain models are really good at coding tasks. When I'm getting a coding task, I route my query to those models. In this task, I have such 211 00:22:23.755 --> 00:22:25.595 a large token context 212 00:22:25.595 --> 00:22:28.634 window that I need to use a model of 213 00:22:28.875 --> 00:22:38.419 model accordingly. Right? So it has to have a certain level of that intelligence built into it. And that's the routing logic that I believe the harness will continue to implement and own. 214 00:22:39.140 --> 00:22:40.100 That said, 215 00:22:40.740 --> 00:22:43.220 you can also imagine that when you're using 216 00:22:43.860 --> 00:22:53.085 when you're making calls to the models, many of these calls are coming from the harness, but many of these calls are directly coming from end users where there isn't an intelligence layer 217 00:22:53.645 --> 00:23:15.905 built into the caller to begin with. Right? So so from a and all of those calls are fielded by a gateway layer. Right? So a gateway has to be also smart by looking at the prompt, right, and figure out which model should it route it to. And of course, gateway solves a problem from a different lens as well, right? Because the gateway has the awareness, the real time awareness 218 00:23:16.065 --> 00:23:36.110 of what's the status of different model providers. Like is one of them serving high latency and I may want to route to a similar model but to a different region or the same model from a different provider, right? And it also of course has a lot of context around how much overall budget and rate limits have been consumed 219 00:23:36.110 --> 00:23:45.985 so far. So it has a lot of real time awareness and it will do the routing based on complexity but also all of this awareness as well and that layer sits rightly within the gateway. 220 00:23:46.305 --> 00:23:53.424 And the task contextual awareness routing sits more within the harness, and both of them need to coexist. 221 00:23:53.825 --> 00:23:57.985 The other interesting aspect of harnesses is because you're working 222 00:23:58.679 --> 00:24:05.799 with these models so closely and because testing anything that touches a model is so challenging due to their probabilistic 223 00:24:05.799 --> 00:24:06.519 nature, 224 00:24:07.000 --> 00:24:08.599 what are some of the 225 00:24:08.840 --> 00:24:14.655 validation methods that you use when iterating on a harness to make sure that you're not regressing 226 00:24:14.655 --> 00:24:20.894 and that it is able to actually work effectively across such a wide swath of different models with different capabilities? 227 00:24:21.455 --> 00:24:31.920 I think that is certainly a challenge for for for the exact reasons that you mentioned. Right? And I I wouldn't say that, like, we have we have fully solved the problem, like, where we have absolutely contained 228 00:24:32.480 --> 00:24:49.725 the the the test cases and regressions and ensure that, like, this is always fully solved. But, of course, we try as much as possible and the community understands this problem statement. Right? So now there are a bunch of benchmark datasets against which, like, you know, you can always keep running 229 00:24:50.285 --> 00:24:54.750 your harness to ensure that you are not regressing on your quality, 230 00:24:55.070 --> 00:24:56.510 and cost benchmarks. 231 00:24:56.510 --> 00:25:02.190 Alright? So we keep running that. And then internally for ourselves, you know, think 232 00:25:02.430 --> 00:25:08.510 I I think of these benchmark datasets as more integration tests from the from the old software world. Right? 233 00:25:09.175 --> 00:25:20.935 And then you still need your functional

test and unit test as well. So what we have done is for different modules that we are building within the harness, we have actually designed some of these test cases as well. Right? So the independent modules don't progress, 234 00:25:21.480 --> 00:25:27.559 but meaningfully with like, you know, so many variables and the stochastic nature of the models and the harness, 235 00:25:27.800 --> 00:25:33.480 the only true way of testing this out is against the benchmark data sets from our standpoint. 236 00:25:33.720 --> 00:25:36.865 But even then so, it so happens that 237 00:25:37.265 --> 00:25:39.985 the end users of Harness should also 238 00:25:40.065 --> 00:25:47.025 actually, like depending on the use cases, they should have at least some coverage of these test cases at their end as well. That's 239 00:25:47.025 --> 00:25:51.799 a final litmus test to ensure that this doesn't regress. 240 00:25:51.960 --> 00:25:53.559 As people are 241 00:25:54.440 --> 00:25:56.200 designing and implementing 242 00:25:56.279 --> 00:25:58.279 agents of various shapes 243 00:25:58.600 --> 00:26:00.039 and given the 244 00:26:00.360 --> 00:26:14.195 wide variety of harnesses as we've discussed, what are some of the ways that people should be figuring out what are the differentiating factors across these different harnesses and how to identify which one is going to suit their particular use case? 245 00:26:14.435 --> 00:26:16.595 It's very interesting question, 246 00:26:16.835 --> 00:26:32.260 Tobias, because the way I'm thinking about this, and I would love to brainstorm with you on this one, like hear your thoughts as well. Right? But of course, while we see that there's a bunch of harnesses out there in the market today that you're able to leverage to build some of your agents, the divide 247 00:26:32.404 --> 00:26:33.524 in the industry 248 00:26:33.924 --> 00:26:45.205 is at a very macro level. Right? And actually, we are seeing our customers choose their harnesses based on those macro divides at the moment rather than trying to get super 249 00:26:45.205 --> 00:26:47.764 micro with it with a bunch of 250 00:26:48.380 --> 00:26:52.700 low level capabilities of different harnesses or at least that's what we're seeing at the moment. 251 00:26:53.100 --> 00:26:54.619 What what do I mean by that? 252 00:26:54.940 --> 00:26:58.859 At the first level you can imagine that you have harnesses that are 253 00:26:59.340 --> 00:27:04.404 SaaS like managed harnesses that you get to use out of the box. 254 00:27:04.645 --> 00:27:15.125 There are some of the harnesses that are more vendor neutral, more open harnesses. Right? And people are able to take a call based on their own needs that am I happy to be locked into this ecosystem 255 00:27:15.929 --> 00:27:20.330 and get the advantage of an integrated ecosystem, right, which could come at a price? 256 00:27:20.570 --> 00:27:27.769 Or do I want to do a little bit more work and get into this open source neutral type of a harness, right? So that's one type of a thing. 257 00:27:28.735 --> 00:27:30.975 The the second here is 258 00:27:31.294 --> 00:27:33.774 that the way some of these harnesses are implemented, 259 00:27:33.855 --> 00:27:40.815 they're intended for certain audience. Like, some some harnesses assume that you're a developer, you understand, like, you know, you're able to set up a bunch of configurations. 260 00:27:41.720 --> 00:28:00.375 You're even, if need be, you're able to write some code, right, or micromanage how the harness is working. Right? So it's intended for a more technical audience, where some harnesses actually are just, like, you know, built for a knowledge worker who does not necessarily come from a coding background and does not want to manage complex configurations 261 00:28:00.375 --> 00:28:01.095 overall. 262 00:28:01.495 --> 00:28:02.135 So 263 00:28:02.375 --> 00:28:12.429 today, I'm seeing that, like, you know, there's a few handful of tools that are out there that, you know, people for the most part are leveraging can be divided based on based on this itself. 264 00:28:12.590 --> 00:28:22.910 Right? We people don't have to go too low level that oh like in this case they harnessed us very well on memory management and in this case they harnessed us very well on a sandbox execution 265 00:28:22.910 --> 00:28:30.304 so and so forth. And we are not seeing people take calls based on that. Are you seeing a different trend by any chance? So, yeah, I think that 266 00:28:30.705 --> 00:28:32.225 in the harness space, 267 00:28:32.945 --> 00:28:40.200 one of the interesting pieces, just as an anecdote, is that a lot of them seem to be written in TypeScript and running on the node ecosystem, 268 00:28:40.280 --> 00:28:40.760 which 269 00:28:41.160 --> 00:28:43.080 as a Python developer, 270 00:28:43.160 --> 00:28:47.799 it is curious to me, but I understand that JavaScript is taking over the world. 271 00:28:48.520 --> 00:28:49.000 But, 272 00:28:49.320 --> 00:28:53.320 yeah, the the key factors that I see when I'm looking at different harnesses are, 273 00:28:54.145 --> 00:28:57.825 yeah, what is the model support, which is generally fairly broad, 274 00:28:58.465 --> 00:28:59.264 but also 275 00:28:59.425 --> 00:29:18.109 how does it integrate with some of the different providers. So I think Claude and Codex are probably the more interesting ones, Claude specifically because they've been having a challenging time of figuring out how their pricing that when you're going through their Claude desktop and Claude code work versus if you're using a third party harness where 276 00:29:18.270 --> 00:29:25.644 you don't get the benefits of their flat rate pricing. So that's one of the considerations when I'm looking at some of the harnesses as a software engineer. 277 00:29:26.684 --> 00:29:38.609 But the other pieces are the MCP and a two a support. So how quickly they're adopting some of these different protocols, particularly as the

protocols themselves change. So MCP just released their latest revision 278 00:29:38.850 --> 00:29:40.049 maybe a month ago. 279 00:29:40.610 --> 00:29:42.289 And then the 280 00:29:42.529 --> 00:29:49.085 what is the footprint of the harness? How much resource does it take? How does that factor into deployability, 281 00:29:49.164 --> 00:29:55.885 particularly for if I'm using this as a deployed agent that I want to run? How does it fit into things like instrumentation 282 00:29:55.885 --> 00:29:57.005 and observability? 283 00:29:57.005 --> 00:30:08.379 Because I wanna make sure that I have a full trace of what the agent is doing so that I can figure out when things go wrong. Why did they go wrong? How did they go wrong? Because they're moving at machine speed, 284 00:30:08.620 --> 00:30:23.914 you have to have other agents that are monitoring those outputs to make sure that you can catch those failure modes versus just if you just have a harness that says, yes. I can run the model. I can plug into all these protocols, but it's a black box and opaque, you have no introspection into the system. 285 00:30:24.235 --> 00:30:31.989 That's definitely a nonstarter for me as an operations engineer. So those are the some of the things that I look at when I'm thinking about different harnesses. 286 00:30:32.790 --> 00:30:38.230 That that that makes a lot of sense. That makes a lot of sense, Tobias. For sure. Like, you know, we we definitely 287 00:30:38.230 --> 00:30:42.230 see our users considering these, but, like, you know, a lot of the times that, 288 00:30:42.710 --> 00:30:47.434 at least from our lens, what we are seeing is we work with organizations. 289 00:30:47.595 --> 00:31:04.370 And on an organization level, I think the the decision matrix is broader than this because people really care about being able to suit some harness to their own broad scope needs, and they choose based on that. But we'll get exposed to more and more of these nuances, 290 00:31:04.370 --> 00:31:07.250 like as our open source agent harness 291 00:31:07.250 --> 00:31:09.809 gets a broader adoption as well. So I think 292 00:31:10.435 --> 00:31:12.674 that would definitely be an interesting learning for us. 293 00:31:13.235 --> 00:31:14.195 So 294 00:31:14.355 --> 00:31:27.929 digging now further into the TrueForge harness itself, could you talk through some of the key design and architectural elements of it and some of the ways that the shape and scope have evolved from when you first started working on it? 295 00:31:28.810 --> 00:31:32.490 So so in terms of, like, I just generally, the way we have designed 296 00:31:32.810 --> 00:31:50.004 first of all, like, I just let me give you little bit of a background on how this entire concept of Harness started from from our standpoint, right, and then I will describe how some of the architectural choices have evolved for us as a team as well. So there were two main pulling factors around this. 297 00:31:50.325 --> 00:31:58.909

We wanted to build some capabilities on our own product like like TrueFoundry AI gateway product where people are able to consume 298 00:31:59.309 --> 00:32:01.710 a lot of the information through the product 299 00:32:02.110 --> 00:32:08.244 but in a more agentic way. I'll give an example. TrueFoundry AI gateway sees a bunch of traces, 300 00:32:08.325 --> 00:32:09.684 agentic traces, 301 00:32:09.924 --> 00:32:18.645 and we want to be able to reason on the traces, alright, that which of my agents are doing well, which ones are prone to any guardrail problem injection attacks, 302 00:32:19.050 --> 00:32:19.930 or 303 00:32:19.930 --> 00:32:25.450 where is PII leaking that I don't want want it to, how can optimize for cost and stuff like that. Right? 304 00:32:26.170 --> 00:32:38.144 So we wanted our users to have that capability where they can more naturally improve the agents that they are, like, you know, invoking through our gateway. Right? So fundamentally, this means that we wanted to 305 00:32:38.705 --> 00:32:45.504 bake in a first class agent as part of our own product so that people are able to leverage this. 306 00:32:46.145 --> 00:32:47.505 And to that end, 307 00:32:48.980 --> 00:32:49.940 fundamentally, 308 00:32:49.940 --> 00:32:52.740 everything that we needed this agent to access, 309 00:32:53.140 --> 00:32:56.980 all the context, all the traces, all the models and 310 00:32:57.460 --> 00:33:05.365 MCPs that the user had permission to was available within the product. And we had all of these things available as context and as MCPs. 311 00:33:05.525 --> 00:33:06.965 So from our standpoint, 312 00:33:07.925 --> 00:33:14.085 this felt like it should have been as simple as that we're able to connect a bunch of these, like, in a context and 313 00:33:14.885 --> 00:33:23.500 use a harness to quickly spin up this agent that we are able to ship with the product. Right? But when we try to use the tools out there today, 314 00:33:23.660 --> 00:33:29.019 like back then, right, we realized that you are either locked into a bunch of these closed harnesses, 315 00:33:29.100 --> 00:33:35.205 which does not allow you this capability of shipping agents as part of the product itself. Right? Or you would have to fundamentally 316 00:33:35.205 --> 00:33:36.005 leverage 317 00:33:36.725 --> 00:33:38.485 more pro code frameworks, 318 00:33:38.485 --> 00:33:44.005 which, like, require you to micromanage the performance of your agents and then quickly things diverge and you keep, 319 00:33:44.485 --> 00:33:50.860 like, upgrading that essentially. Right? So so we really wanted to strike that balance where we are able to optimize for speed, 320 00:33:51.420 --> 00:33:55.419 ship things fast, and they cannot get locked into any ecosystem. 321 00:33:55.740 --> 00:34:11.795 So that's that was one of the motivation that, like, you know, led us to kick things off. And at the same time, we started seeing that a lot of our customers started caring about, like, you know, not being locked into a certain vendor that way. Right? So when we started building this out, 322 00:34:12.115 --> 00:34:13.235 the neutrality, 323 00:34:13.315 --> 00:34:14.355 the openness, 324 00:34:14.355 --> 00:34:16.115 the speed at which you are able to ship, 325 00:34:16.619 --> 00:34:21.099 these were some of the design principles with which we had started building out the harness. 326 00:34:21.740 --> 00:34:23.020 Over a period of time, 327 00:34:23.420 --> 00:34:29.420 when we started actually implementing this, and this actually changed a lot in how we thought about the overall architecture, 328 00:34:29.660 --> 00:34:30.779 we realized that 329 00:34:33.155 --> 00:34:38.595 while we are able to make building of the agent relatively straightforward, 330 00:34:38.835 --> 00:34:43.235 being able to run this, right, and being able to run this 331 00:34:43.950 --> 00:34:51.950 at times and in environments that we care about is another challenge. Like, there's a lot of things that you want this agent to do that you want to be able to schedule, 332 00:34:52.270 --> 00:34:58.335 that you want to run every day or every few hours. Right? Or you want to run this on a trigger. 333 00:34:58.495 --> 00:35:17.450 How do you expose or sometimes you want the agents that you're developing, you want this agent to run as part of another ecosystem. Right? So how do you build out that entire runtime of the agent is another major problem that we realized that we needed to solve. Right? So, like, we continue to evolve the harness from, like, you know, primarily being neutral, 334 00:35:17.530 --> 00:35:22.490 like, you know, to this entire runtime. Now the two different types of runtimes, basically. Right? Similarly, 335 00:35:23.055 --> 00:35:25.535 some of the design choices that we needed to change was 336 00:35:25.855 --> 00:35:28.415 how do you think about sandboxing 337 00:35:28.415 --> 00:35:30.895 overall? And what we thought was sandbox 338 00:35:31.214 --> 00:35:36.415 should be exposed to the agents of the tool. That's the other design choice that we had to make. Most of the harnesses that we know 339 00:35:37.290 --> 00:35:43.210 run the entire agent itself in a sandbox. We actually ended up flipping that design principle over a period of time. 340 00:35:43.610 --> 00:35:47.530 So now what we do is we run the agent loop in the server itself, 341 00:35:47.690 --> 00:35:55.985 and we provision a sandbox only when the agent actually needs to execute that code. Right? So the advantage that we got by doing this was 342 00:35:56.145 --> 00:35:59.905 one server is now able to run many of our agents concurrently. 343 00:36:00.224 --> 00:36:00.865 And 344 00:36:01.345 --> 00:36:15.000 it turns out that that when you are not using writing code, basically, it makes those turns go much cheaper and much faster. And you're ensuring that things like your entire secrets and all are never leaving 345 00:36:16.840 --> 00:36:19.155 the layer that the

hardest can touch at all. Right? 346 00:36:19.635 --> 00:36:22.515 So a lot of these things that we started 347 00:36:22.755 --> 00:36:25.075 editing in our architecture 348 00:36:25.555 --> 00:36:34.980 came as evolution of what we saw as this accuracy cost and efficiency trade off that we really, really wanted to nail down. And 349 00:36:35.060 --> 00:36:36.980 over a period of time, these sort of things evolve. 350 00:36:37.460 --> 00:36:39.460 And as you have been 351 00:36:39.940 --> 00:36:44.820 working on the harness and using that to validate the gateway, 352 00:36:45.380 --> 00:36:47.060 what are some of the 353 00:36:47.940 --> 00:36:48.580 most 354 00:36:48.755 --> 00:36:51.475 complex or challenging aspects 355 00:36:51.475 --> 00:36:52.115 of 356 00:36:52.275 --> 00:36:52.995 building 357 00:36:53.235 --> 00:37:02.890 these two different systems and using them to play off against each other and just some of the discoveries that you've run into that were unanticipated 358 00:37:02.890 --> 00:37:06.330 roadblocks on your path to actually building these products? 359 00:37:07.210 --> 00:37:07.770 Yeah. 360 00:37:08.090 --> 00:37:08.650 So 361 00:37:08.890 --> 00:37:14.490 in all honesty, I think one of the problems that you always have to solve for in such settings is 362 00:37:14.650 --> 00:37:15.530 organizational, 363 00:37:15.530 --> 00:37:15.770 right? 364 00:37:16.855 --> 00:37:18.055 Because the way 365 00:37:18.295 --> 00:37:32.280 you organize your teams building out the by the way, these are problems that everyone needs to solve for. Right? Like, like, people have to figure out the solution. So I would like to share this with with folks as well listening listening to the podcast. When you think about 366 00:37:32.520 --> 00:37:36.360 building out the the gateway itself, you are building for 367 00:37:36.680 --> 00:37:42.520 one intended audience, which is getting used in a certain way. Right? Organizations buy a gateway 368 00:37:42.840 --> 00:37:44.280 for observability, 369 00:37:44.280 --> 00:37:46.680 for control, for being able to 370 00:37:47.295 --> 00:37:48.095 access 371 00:37:48.095 --> 00:37:48.735 the 372 00:37:49.055 --> 00:37:57.535 rich ecosystem relatively fast. Right? When you're building out an open source harness instead, right, you are building for the capabilities 373 00:37:57.694 --> 00:38:07.140 to to empower the developers with a bunch of these capabilities. Right? So that they are able to developers or even knowledge workers to take up a harness and they're gonna build out these agents. 374 00:38:07.460 --> 00:38:08.339 Now because 375 00:38:09.539 --> 00:38:11.140 the the user 376 00:38:11.460 --> 00:38:18.180 is slightly different in the two cases. Right? The team that's building it has to always continuously 377 00:38:17.595 --> 00:38:18.555 empathize 378 00:38:18.635 --> 00:38:20.795 with the end user that they're building towards. 379 00:38:21.115 --> 00:38:24.155 And the pace at which the team is

developing 380 00:38:24.155 --> 00:38:29.435 different components also needs to change because one is serving more production systems. 381 00:38:29.755 --> 00:38:32.075 The other one is serving more 382 00:38:32.075 --> 00:38:41.970 building phase of the ecosystem as well. Right? So so from our standpoint, how do we allocate the bandwidth? How do we divide the team? How do we ensure 383 00:38:42.050 --> 00:38:46.930 that we are able to meaningfully make progress on the on both the two fronts was actually 384 00:38:47.090 --> 00:38:58.005 a relatively big problem for us to solve for. Right? The way we we came up with like, the the approach that we came up is to solve this problem is we actually combined 385 00:38:58.085 --> 00:38:58.725 the 386 00:38:59.605 --> 00:39:01.925 the part of our gateway 387 00:39:01.660 --> 00:39:03.420 that was leveraged 388 00:39:03.500 --> 00:39:10.540 to build a bunch of these agentic solutions directly. Like, we had a component in our gateway where people are able to register registries, 389 00:39:10.540 --> 00:39:11.580 like agent 390 00:39:11.580 --> 00:39:12.620 to a registry. 391 00:39:12.940 --> 00:39:13.660 And from there, 392 00:39:14.285 --> 00:39:16.285 they would they would always 393 00:39:16.365 --> 00:39:18.765 run into some roadblocks 394 00:39:18.765 --> 00:39:30.320 into how they have built out an agent using another closed source product or some other framework. They want to register to the registry and then access it centrally through the gateway. They would keep running into those roadblocks. 395 00:39:30.560 --> 00:39:34.160 And there were a lot of learnings that were coming out about how 396 00:39:34.400 --> 00:39:38.640 the agent development platform or the agent development harness itself should solve the problem. 397 00:39:39.315 --> 00:39:40.595 So we combined 398 00:39:40.595 --> 00:39:48.275 the the team that was always getting that learning from this part of the product to also work on the agent harness layer. 399 00:39:48.355 --> 00:39:55.714 Right? So the top of the funnel, like, the core pain points that people want to leverage an outsourced 400 00:39:54.720 --> 00:39:56.000 open source harness 401 00:39:56.320 --> 00:39:58.480 was always coming to this team. So 402 00:39:59.360 --> 00:40:00.880 the empathy for 403 00:40:01.040 --> 00:40:02.000 the user, 404 00:40:02.320 --> 00:40:06.320 both from a gateway standpoint and from a harness standpoint, is how 405 00:40:07.040 --> 00:40:19.325 we solve for. And a lot of the capabilities that we ended up building within the harness by design, the roadmap of the harness actually came from our early users of the gateway directly. So that way, a lot of the downstream 406 00:40:19.725 --> 00:40:25.710 challenges that we would have needed to solve for actually got front loaded by 407 00:40:26.030 --> 00:40:28.430 working with the right set of users 408 00:40:28.510 --> 00:40:34.430 using the product as a design partner, I guess. You touched a little bit on some of the 409 00:40:34.590 --> 00:40:40.665 cost aspect of the harness being a contributing factor where that's largely the system prompt and 410 00:40:40.985 --> 00:40:54.220 trimming that down can have a substantial savings. But as you have been building the system, what are some of the other key elements of the harness itself that contribute the most to the cost efficiency, 411 00:40:54.380 --> 00:40:57.500 the accuracy of the outputs, and the reliability 412 00:40:57.500 --> 00:40:58.220 of 413 00:40:58.300 --> 00:40:59.340 the agent 414 00:40:59.500 --> 00:41:00.860 operating within that harness? 415 00:41:01.755 --> 00:41:05.994 So there three major factors to this entire cost optimization 416 00:41:05.994 --> 00:41:13.355 and the accuracy and reliability trade off. Like one we covered, which is your system prompt, right? The second one is 417 00:41:13.910 --> 00:41:16.630 generally your harness efficiency. 418 00:41:16.630 --> 00:41:28.790 And what I mean by that is the number of tool calls that you are making, the order in which you are making the tool calls, the order in which you are applying any kind of policies that you may want to do, 419 00:41:29.565 --> 00:41:31.885 which actually has a significant impact 420 00:41:32.285 --> 00:41:33.085 on how 421 00:41:33.325 --> 00:41:37.405 overall harness is working. Right? And to this end, there are certain 422 00:41:37.805 --> 00:41:41.165 more technical concepts that you also bake into the harness. 423 00:41:41.484 --> 00:41:42.285 For example, 424 00:41:43.005 --> 00:41:44.204 do deferred 425 00:41:44.610 --> 00:41:54.050 loading of tools, for example. When you do not need access to certain tools at a certain point, you defer that, right? So how do you bake in that intelligence 426 00:41:54.050 --> 00:41:57.490 that for any given task that the harness is accomplishing, 427 00:41:58.205 --> 00:42:00.525 you always pick the right combination 428 00:42:00.685 --> 00:42:01.325 of 429 00:42:01.645 --> 00:42:05.965 whether you are loading a certain tool, whether you are using the 430 00:42:06.205 --> 00:42:07.725 actual persistence 431 00:42:07.965 --> 00:42:10.445 to be able to store some context and only 432 00:42:11.245 --> 00:42:13.245 get a part of it when you need 433 00:42:13.900 --> 00:42:16.060 to answer a certain question. 434 00:42:16.460 --> 00:42:36.005 What do you solve for using tokens of the model directly versus where you think that being able to write code and execute this would be the right way to solve this problem? So this is what I call as the hardest efficiency. And the right choices that you make in this one is extremely critical to solve for the cost and accuracy trade off. And the last one, 435 00:42:36.405 --> 00:42:38.005 which is the overall 436 00:42:38.085 --> 00:42:42.850 model benchmarks overall. Right? Like, that's the that's the other one that's that's extremely critical. 437 00:42:43.250 --> 00:42:47.570 Many times, like, for the same task,

like, models can have 438 00:42:47.810 --> 00:42:51.330 an order or even two orders of magnitude cost difference. 439 00:42:52.224 --> 00:43:01.985 But their performance could be fairly comparable at that point. Right? So how do you choose the appropriate model for different types of tasks that you're doing? And how do you train the harness 440 00:43:02.224 --> 00:43:03.025 so that 441 00:43:03.905 --> 00:43:15.380 it's able to make those design choices? I think these are the three key factors that we have worked to optimize on. Obviously, a lot of effort that has gone into is problem number two, where different 442 00:43:16.180 --> 00:43:18.660 components of the harness, how do you optimize against 443 00:43:18.980 --> 00:43:19.299 those? 444 00:43:20.115 --> 00:43:25.635 Like, that's where a lot of efficiencies and and citing some numbers here, there have been tasks where we have seen 445 00:43:25.875 --> 00:43:27.555 some other harnesses 446 00:43:27.875 --> 00:43:29.955 which are taking, like, more than 75,000 447 00:43:29.955 --> 00:43:35.590 tokens to implement, like, know, to accomplish the task. Our harness was able to do within 23,000 448 00:43:35.590 --> 00:43:39.510 tokens basically. So we have seen anywhere from 50 449 00:43:40.390 --> 00:43:42.470 even 7075% 450 00:43:42.470 --> 00:43:46.310 of token reduction in similar kind of tasks by getting this part right. 451 00:43:46.974 --> 00:43:53.055 And then going back to what we touched on briefly of different models behaving differently 452 00:43:53.295 --> 00:43:55.535 given even the same prompts, 453 00:43:55.695 --> 00:44:02.840 the general wisdom that has built up over the past couple of years, and, of course, that general wisdom is frequently invalidated 454 00:44:02.840 --> 00:44:04.680 because everything moves so quickly, 455 00:44:05.160 --> 00:44:12.200 is that just changing a model given all of the same inputs can have wildly different outcomes. 456 00:44:12.680 --> 00:44:13.320 And 457 00:44:13.560 --> 00:44:14.840 given the current 458 00:44:15.345 --> 00:44:18.305 generation of models that we are operating, 459 00:44:18.865 --> 00:44:19.665 is that 460 00:44:20.065 --> 00:44:24.145 assumption still accurate? And what are some of the ways that people should be 461 00:44:24.545 --> 00:44:26.785 testing and thinking about their 462 00:44:26.865 --> 00:44:27.665 preconceived 463 00:44:27.665 --> 00:44:45.905 notions given what they've learned from the past two years of evolution where I think the hallucination rate was one of the first ones to gain a lot of ground, and that has is it's still a factor, but the the rate has dropped a lot. I'm just wondering as the models evolve and as they all become more capable, 464 00:44:46.465 --> 00:44:48.065 has that assumption of 465 00:44:48.785 --> 00:44:51.825 switching models will dramatically change the outcome? 466 00:44:52.145 --> 00:44:53.345 Does that still hold? 467 00:44:54.385 --> 00:44:55.265 Actually, 468 00:44:55.910 --> 00:44:58.550 we have seen that assumption break. 469 00:44:59.030 --> 00:45:03.270 It's been now a few months that we have seen that assumption break, in all honesty. 470 00:45:03.750 --> 00:45:06.950 And what that means is by the way, hallucination 471 00:45:06.950 --> 00:45:22.835 is a term. Like, I I think if there was a way to build a heat map of how frequently that term is being used, I think now over the last few months, it has become quite cold, that heat map. Right? We don't we don't hear people talk about hallucinations as much when they are talking about 472 00:45:23.600 --> 00:45:28.400 the model performance. Same thing. Earlier, we really needed to craft a 473 00:45:28.560 --> 00:45:46.515 prompt for each of the models, and that was a big thing that people needed to solve for. Now we are seeing that models are getting better, that that you don't need to micromanage your prompts as much to work with these different models. Right? Now, of course, like when you go from a frontier model to a very specialized model, the specialized model expects 474 00:45:46.595 --> 00:45:51.795 sometimes even if you have fine tuned some of these models and we we work with still with some of the fine tuned models, 475 00:45:52.650 --> 00:45:56.810 That expects prompt to come in a certain way, and that really helps to 476 00:45:57.609 --> 00:46:09.415 to be able to change the prompts. But, like, so long as you're working in a, like, a beyond a certain level of, call it, the number of parameters in the model. Right? Like, once you're working with a threshold, 477 00:46:09.895 --> 00:46:13.975 model is larger than a certain threshold, you actually don't need to worry about switching prompts 478 00:46:14.055 --> 00:46:16.615 anymore for to to get similar kind of outputs now. 479 00:46:17.190 --> 00:46:19.830 And that, of course, has downstream 480 00:46:19.830 --> 00:46:23.430 implications now in terms of implementation of the harness that 481 00:46:24.230 --> 00:46:25.590 you you just 482 00:46:25.990 --> 00:46:29.109 if you you're able to make choices more 483 00:46:30.445 --> 00:46:31.965 independently now. Right? 484 00:46:32.525 --> 00:46:34.445 But at the same time, we 485 00:46:34.685 --> 00:46:49.440 we in our gateway product as well, we keep this flexibility that if people are switching model, they have a scope to be able to change the the prompt as well. And it's especially useful when you're using fine tuning and stuff. Right? So we actually leverage that capability also 486 00:46:50.079 --> 00:46:51.680 in our harness as well. 487 00:46:52.400 --> 00:46:53.920 One of the other 488 00:46:54.160 --> 00:46:56.480 factors of model selection 489 00:46:57.035 --> 00:47:05.515 that I've experienced is that you might need at least a certain threshold of model to be able to effectively drive the harness where, 490 00:47:05.755 --> 00:47:37.395 a while ago, they I've had a conversation with somebody who had the analogy of the LLM is the engine and everything

else that you put around it is the car. So you can have all the horsepower you want, but if there's no car, you're not going anywhere. And so bringing that a little further, it sort of feels like you you have sort of as you're going on the amusement park rides, you must have this many parameters to drive this harness. And I'm wondering what you're seeing as some of the cutoff threshold where if you get below a certain size model, it's not going to be able to effectively use the harness because there is too much in the context window. 491 00:47:37.875 --> 00:47:38.515 So 492 00:47:38.750 --> 00:47:43.230 and and, like, the most the most interesting thing that we are seeing with this is 493 00:47:44.670 --> 00:47:45.310 people, 494 00:47:45.549 --> 00:47:52.510 the the users of the harness, including including ourselves. Right? We have built out a lot of systems that, like, leverage our own harness. And 495 00:47:52.865 --> 00:48:00.385 people actually don't necessarily need to go below a certain threshold anymore to make things work for them. Effectively, 496 00:48:00.464 --> 00:48:05.505 it's not like people are thinking about, should I use a 3,000,000,000 parameter open source 497 00:48:06.224 --> 00:48:08.065 model to run my entire harness 498 00:48:08.550 --> 00:48:15.510 versus should I use Fable five, for example, right? Like, it's not like people are typically operating in those extremes. 499 00:48:15.510 --> 00:48:16.870 People are thinking like, 500 00:48:17.190 --> 00:48:19.830 like, am I doing a Fable or a GLM? 501 00:48:19.830 --> 00:48:21.110 Am I thinking 502 00:48:21.350 --> 00:48:22.150 Haiku, 503 00:48:22.150 --> 00:48:30.705 right? So people are still operating in that domain. And what what's what's nice about this is the overall token cost has reduced, 504 00:48:31.105 --> 00:48:33.105 like, you know, by such a large 505 00:48:33.744 --> 00:48:46.840 margin over the over the period that intelligence per token, as I think about this. Right? Intelligence per dollar that you're spending has has gone up so much that that's the ballpark in which we are we are talking about. So now 506 00:48:47.320 --> 00:48:49.080 you are still able to 507 00:48:49.835 --> 00:48:52.234 like, even if you choose a base model, 508 00:48:52.555 --> 00:49:07.860 something like a HYCU and above, basically, right, the the hardest works. And then for extremely complex task, you go to, like, your GLMs and the Opus and the Fable equivalent, basically. And otherwise, you're able to a bunch of tasks using high equivalent type of products. 509 00:49:08.580 --> 00:49:11.060 And as you have been 510 00:49:11.380 --> 00:49:20.635 working on TrueForge and TrueFoundry and working with your customers, what are some of the most interesting or innovative or unexpected ways that you've seen your harness used? 511 00:49:21.195 --> 00:49:21.915 So 512 00:49:22.155 --> 00:49:25.195 first of all, we noticed that 513 00:49:25.595 --> 00:49:26.555 people are 514 00:49:27.355 --> 00:49:28.555 We saw 515 00:49:29.115 --> 00:49:30.714 through our product itself directly 516 00:49:30.990 --> 00:49:33.390 that we see queries in languages 517 00:49:33.470 --> 00:49:36.110 that we absolutely don't understand. We 518 00:49:36.270 --> 00:49:38.030 were under the assumption that 519 00:49:38.430 --> 00:49:45.815 most people are using the harness in English with the most common user group that we work with. They're saying queries coming in completely 520 00:49:45.815 --> 00:49:47.974 different languages that don't 521 00:49:47.974 --> 00:49:48.615 understand. 522 00:49:48.775 --> 00:49:51.175 That was just a fun thing that we noticed 523 00:49:51.494 --> 00:49:53.415 in terms of the usage of the platform. 524 00:49:53.655 --> 00:49:57.015 Outside of this, what we have seen with our harness usage, 525 00:49:58.119 --> 00:49:58.920 people 526 00:49:59.240 --> 00:49:59.880 are 527 00:50:00.440 --> 00:50:03.720 doing a couple of things that we just did not imagine 528 00:50:05.240 --> 00:50:11.560 will be a use case because it's supposed to be such a deterministic use case. People have actually built out their own pricing systems, 529 00:50:12.225 --> 00:50:15.665 like how they price their product overall 530 00:50:15.825 --> 00:50:19.745 based on some data that they capture using our gateway 531 00:50:19.985 --> 00:50:25.665 and how they build out the end pricing dashboard for their end users using our harness. 532 00:50:26.070 --> 00:50:37.590 And this entire loop came together for our end users and they were able to build out this entire thing within a few hours that their understanding was it would have taken them weeks or months to build this entire system. 533 00:50:37.910 --> 00:50:41.575 That kind of a usage, we had not expected 534 00:50:41.575 --> 00:50:44.455 people to leverage the harness for. So basically, 535 00:50:44.775 --> 00:50:46.695 the input comes from the gateway, 536 00:50:46.855 --> 00:50:51.335 the implementation happens through the harness, and that combination became very powerful. 537 00:50:51.815 --> 00:50:52.375 So 538 00:50:52.960 --> 00:50:57.760 what this has opened up for our users and also just in our 539 00:50:58.080 --> 00:50:59.600 mind in terms of what 540 00:51:00.000 --> 00:51:07.155 we were expecting versus what we are seeing is people leverage gateway to build out pretty much all of their agentic capabilities. 541 00:51:07.315 --> 00:51:10.994 All of the agent capabilities by default get routed through the gateway. 542 00:51:11.315 --> 00:51:15.234 Now, any derivative component that they need to build from here, 543 00:51:15.555 --> 00:51:23.730 that's where now the harness is starting to get used very strongly. So I think that's that's been an insight that we have learned through the usage of the product. 544 00:51:24.210 --> 00:51:32.690 And in your experience of working in this space and building these systems, what are the most interesting or unexpected or challenging lessons that

you've learned personally? 545 00:51:33.744 --> 00:51:37.745 So one of the very interesting things that I've learned building 546 00:51:37.984 --> 00:51:41.825 these products over the last two, three, four years is 547 00:51:42.145 --> 00:51:43.665 when you work 548 00:51:43.744 --> 00:51:45.185 with different 549 00:51:45.345 --> 00:51:47.665 personas with their own priorities, 550 00:51:47.665 --> 00:51:47.905 right, 551 00:51:48.520 --> 00:51:53.160 they are looking at the same solution from completely 552 00:51:53.160 --> 00:51:54.280 different lens. 553 00:51:55.480 --> 00:51:57.800 And it so turns out that if you're building 554 00:51:58.040 --> 00:52:02.120 a system that caters to people with different 555 00:52:02.120 --> 00:52:03.880 goals that they're optimizing for, 556 00:52:04.575 --> 00:52:11.215 and this is the only system through which all of those goals get a common ground, right, 557 00:52:11.615 --> 00:52:14.255 to be solved for, right, then 558 00:52:14.335 --> 00:52:23.240 the only way that you can think about building a product like that is by having a good mix of opinion from all of these different varying 559 00:52:23.320 --> 00:52:26.760 voices in the room. And I'll give an example of why 560 00:52:26.920 --> 00:52:28.120 that has been such 561 00:52:28.360 --> 00:52:30.200 an important learning experience for us. 562 00:52:30.840 --> 00:52:32.680 In the very beginning, I talked about 563 00:52:33.320 --> 00:52:37.000 different pillars that this 564 00:52:35.675 --> 00:52:40.795 gateway and harness product is solving for. It's solving for reliability 565 00:52:41.035 --> 00:52:48.155 that your platform teams, your development teams care about. The fact that you build an application, it should be always up, always running, operate at low latency. 566 00:52:49.130 --> 00:52:53.450 Then there's a security aspect to it that your CISO org cares a lot about. 567 00:52:55.690 --> 00:52:58.810 The four walls within my organization should always be secure. 568 00:52:59.530 --> 00:53:04.410 Then there's a CFO aspect to it that they really care about being able to do this cost optimization. 569 00:53:05.424 --> 00:53:06.145 Overall, 570 00:53:06.305 --> 00:53:19.470 they should have an ROI, and that entire story should be clear. Which agents are we building? How do we do the reporting and stuff like that? And then there's this compliance aspect to it that if tomorrow I have to be able to answer why my agent built a certain decision, 571 00:53:20.110 --> 00:53:21.070 how do I do that? 572 00:53:21.550 --> 00:53:23.070 And turns out that the 573 00:53:23.390 --> 00:53:31.150 interest of the CISO, the CFO, the compliance officer, the CTO, all of these things are now being answered by this common layer. 574 00:53:32.265 --> 00:53:33.545 And the 575 00:53:33.944 --> 00:53:35.465 most interesting way 576 00:53:35.625 --> 00:53:38.184 that you can solve for this 577 00:53:38.184 --> 00:53:42.265 entire problem statement is by hearing the voice of all of these personas. 578 00:53:42.904 --> 00:53:51.680 And where we have seen some of the solutions within the industry where they have tried to create a completely specialized view for the CFO 579 00:53:51.680 --> 00:53:53.360 in terms of the FinOps layer. 580 00:53:54.560 --> 00:53:56.400 And then they have basically 581 00:53:57.040 --> 00:53:58.895 done so much instrumentation 582 00:53:58.975 --> 00:54:03.295 that it doesn't get adopted by the platform engineering or the actual development layer. So 583 00:54:03.855 --> 00:54:09.615 yes, you have built this amazing, beautiful FinOps layer. But because there's no usage coming from 584 00:54:10.335 --> 00:54:13.100 the left side, there's nothing to see on the right side, 585 00:54:13.340 --> 00:54:13.900 and 586 00:54:14.380 --> 00:54:33.065 so on and so forth. We have seen multiple combinations of these break in the industry. We have seen multiple personas pull our product in different directions. And the way we are able to maintain this balance is by listening to these point of views. And I guess that's been a personal lesson that you really want to understand that, yes, who is your main buyer persona? But 587 00:54:33.065 --> 00:54:42.220 who are the other extremely important stakeholders in the room? And you want to make sure that you hear that holistic voice. That's been a major learning for me. 588 00:54:42.859 --> 00:54:45.420 And as people are evaluating 589 00:54:45.420 --> 00:54:47.180 different agent harnesses, 590 00:54:47.180 --> 00:54:50.619 what are the cases where you would say that TrueForge is the wrong choice? 591 00:54:52.105 --> 00:54:53.945 Overall, number one, 592 00:54:54.665 --> 00:55:02.265 at the moment, like, no, this is all a point in time answer. Right? So as of whatever, 09/04/2026, 593 00:55:02.265 --> 00:55:02.825 right, 594 00:55:04.105 --> 00:55:19.890 this answer likely is valid today. But but today, we are not specializing ourselves for a coding harness. So if somebody wanted to build out the users of the coding harness, probably I find out other open source or closed source harnesses that serve that use case better. 595 00:55:20.845 --> 00:55:24.365 In some other cases, if you really care about being able to 596 00:55:25.565 --> 00:55:27.725 basically, if you have a lot of visibility 597 00:55:27.965 --> 00:55:36.770 that roughly this is how my system should behave and these are the three lanes in which my agent should go on and there should not be any other fourth lane that they need to explore because 598 00:55:37.330 --> 00:55:46.530 I care about whatever ten millisecond latency implications, etcetera, etcetera, right? If such is your use case, then you are actually better off being able to 599 00:55:47.650 --> 00:55:50.475 more be in control of the code that you're writing and 600 00:55:50.955 --> 00:55:53.995 not let it run through a harness layer like TrueForge, 601 00:55:54.075 --> 00:55:54.795 I choose 602 00:55:55.355 --> 00:56:01.515 a

more deterministic framework for building out your agents. I think those are a couple of situations where I would not pick TrueForge 603 00:56:01.515 --> 00:56:05.200 at the moment. If you're building anything that is more general purpose, 604 00:56:05.599 --> 00:56:06.080 that 605 00:56:06.720 --> 00:56:07.920 you care about 606 00:56:08.079 --> 00:56:09.280 the evolution 607 00:56:09.520 --> 00:56:11.520 of the agents over a period of time, 608 00:56:11.839 --> 00:56:15.279 you care about being able to build it quickly, you care about being able to build it efficiently, 609 00:56:15.725 --> 00:56:23.645 and you care about being in control of the type of models that you're able to use, you care about being able to control the cost, we feel like TrueForge is a great choice. 610 00:56:24.045 --> 00:56:32.779 And as you continue to build and iterate on these systems and continue to explore the current state of the ecosystem, 611 00:56:32.779 --> 00:56:38.859 what are some of the things you have planned for the near to medium term or any particular projects or problem areas that you're excited to explore? 612 00:56:41.035 --> 00:56:54.955 Absolutely. Absolutely. This is the thing that we have been talking a lot about. And a lot of our own roadmap is shaped directly by our customers as well, the current users of the platform as well. So 613 00:56:54.350 --> 00:57:01.470 one of the things that we have been very excited about is going much deeper into this entire agent identity area. 614 00:57:02.270 --> 00:57:06.270 And how do you manage cross app access? How do you manage 615 00:57:06.670 --> 00:57:09.965 this agent chaining, permission controls? And 616 00:57:10.605 --> 00:57:17.565 I've given an interesting problem statement here. I had mentioned briefly about how do you do least privilege across agent chaining. But 617 00:57:18.765 --> 00:57:24.490 when you start doing least privilege across agent chaining, one of the problems that happen is now your configuration 618 00:57:24.490 --> 00:57:31.290 management of which agents have access to which MCP's and which tools and which users have access to which MCP's and which agents, 619 00:57:31.849 --> 00:57:33.369 that becomes a 620 00:57:33.690 --> 00:57:35.130 very complex problem to solve. 621 00:57:35.744 --> 00:57:41.665 So then how do you take this into a direction where you're not compromising for the UX, where 622 00:57:42.385 --> 00:57:46.145 you understand the intent as well? It's not just about theoretically 623 00:57:46.305 --> 00:57:55.410 what is possible, you understand the intent when you implement that without compromising the user experience while ensuring the safety and security of the system. That's 624 00:57:55.410 --> 00:57:56.450 the direction that 625 00:57:57.010 --> 00:58:00.930 we are taking the product in. And then, of course, there's 626 00:58:01.250 --> 00:58:04.210 constant work happening around cost optimization, 627 00:58:04.610 --> 00:58:05.570 sandbox security. 628 00:58:06.275 --> 00:58:09.795 So some of those areas we are continuing to work on and that's 629 00:58:09.795 --> 00:58:11.555 part of our immediate roadmap as well. 630 00:58:11.955 --> 00:58:12.515 Are 631 00:58:12.755 --> 00:58:16.675 there any other aspects of the work that you're doing at TruFoundry 632 00:58:16.675 --> 00:58:23.250 or on TruForge or the overall space of Vagintic harnesses that we didn't discuss yet that you would like to cover before we close out the show? 633 00:58:23.970 --> 00:58:27.170 There's one thing that maybe I'll I'll call out here is, 634 00:58:27.570 --> 00:58:28.210 overall, 635 00:58:28.610 --> 00:58:29.170 the 636 00:58:29.570 --> 00:58:33.010 like, one of one of the learnings that we have had in this space is 637 00:58:33.485 --> 00:58:35.645 people have been trying to address 638 00:58:36.125 --> 00:58:36.605 problems 639 00:58:36.925 --> 00:58:41.165 around the control plane for enterprise AI or how 640 00:58:41.165 --> 00:58:43.165 you bring together this harness 641 00:58:43.325 --> 00:58:44.525 using smaller 642 00:58:45.725 --> 00:58:47.405 targeted point solutions. 643 00:58:47.810 --> 00:58:48.610 The 644 00:58:49.650 --> 00:58:51.170 goal has been for 645 00:58:51.330 --> 00:59:00.130 a lot of folks has been that, can I do my model routing there? Can I do my MCP registry there? Can I build out my A2A 646 00:59:00.525 --> 00:59:10.685 optimal agent to agent communication layer? People have been thinking a lot about that. One of the learnings that we have had working with the organizations that we work with is 647 00:59:12.444 --> 00:59:15.805 the only meaningful way 648 00:59:16.760 --> 00:59:20.280 that you build this overall control plane is if 649 00:59:22.359 --> 00:59:29.640 you bring these systems together. Like if you bring all of your tokens together in one place is how you only meaningfully solve this problem. 650 00:59:30.435 --> 00:59:36.915 So it turns out that a model proxy or a separate MCB proxy or an agent registry in isolation 651 00:59:36.915 --> 00:59:37.955 or a 652 00:59:38.195 --> 00:59:39.555 guardrail solution 653 00:59:40.355 --> 00:59:41.235 overall 654 00:59:41.235 --> 00:59:44.915 is just not able to solve the problem that organizations are facing because 655 00:59:45.910 --> 00:59:49.750 the true problems don't happen within these point 656 00:59:49.750 --> 00:59:50.550 solutions. 657 00:59:50.870 --> 00:59:54.070 The true problems happen at the interface of these point solutions. 658 00:59:54.550 --> 00:59:59.510 And to meaningfully solve for this entire thing, you need to bring these components together 659 00:59:59.605 --> 01:00:02.325 and do it in a very AI native way. 660 01:00:02.565 --> 01:00:04.085 That's been one of the learnings 661 01:00:04.485 --> 01:00:09.285 that I wanted to share with the folks as well. So whoever is thinking about the architecture 662 01:00:09.525 --> 01:00:10.085 of 663 01:00:10.885 --> 01:00:13.445 how to build and scale and manage all

of their agents, 664 01:00:14.000 --> 01:00:16.480 maybe there's a considerable consideration to be made here. 665 01:00:17.040 --> 01:00:35.765 Alright. Well, for anybody who wants to get in touch with you and follow along with the work that you and your team are doing, I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get your perspective on what you see as being the biggest gaps in the tooling technology or human training that's available for AI systems today. This is such a deep topic. 666 01:00:35.925 --> 01:00:42.565 Right? It's it's it's very difficult to give a quick answer to this. But the way I'm seeing this is 667 01:00:43.340 --> 01:00:44.140 we need 668 01:00:45.020 --> 01:00:47.020 a holistic shift to 669 01:00:47.660 --> 01:00:50.780 solve for this entire problem statement. I literally talk to 670 01:00:52.060 --> 01:01:01.355 teenagers, college kids, and folks are asking what should we be thinking about? What's the next thing that we should be doing? What's the discipline that I should go study in? 671 01:01:01.915 --> 01:01:07.275 Should I still learn coding? Should I not learn coding? How do I upskill myself in this entire 672 01:01:07.435 --> 01:01:08.235 world of AI? 673 01:01:08.960 --> 01:01:17.200 And it's such a deep question that's there in everyone's mind that I feel like a tactical solution around how you learn AI 674 01:01:17.359 --> 01:01:24.495 or how you learn this latest new tool or any framework out there is just not an answer to the question. I think 675 01:01:25.375 --> 01:01:29.935 the only way to meaningfully solve this is what are some of the more fundamental 676 01:01:29.935 --> 01:01:33.215 layers where we expect human beings to 677 01:01:33.455 --> 01:01:39.710 contribute and add value. And what are the fundamental concepts, aspects in which we expect AI 678 01:01:39.870 --> 01:01:40.430 to 679 01:01:40.670 --> 01:01:46.990 assist us? We need to think about that. And that needs to get inculcated into our education system more fundamentally 680 01:01:47.315 --> 01:01:49.875 as opposed to solving this problem superficially. 681 01:01:49.875 --> 01:01:53.155 So I don't have a great answer to this question, but it's a problem that 682 01:01:53.795 --> 01:01:55.555 I resonate with very, 683 01:01:55.715 --> 01:01:58.515 very deeply through many of the latest conversations. 684 01:01:59.235 --> 01:02:00.195 Absolutely. 685 01:02:00.195 --> 01:02:01.395 Yeah. It's definitely 686 01:02:01.670 --> 01:02:02.550 a big 687 01:02:02.790 --> 01:02:12.870 a big thorny issue that the entire world is having to tackle right now. Even just in my own small ecosystem of working in an engineering team, 688 01:02:13.190 --> 01:02:23.715 it requires a lot of reimagining how you work every day because you can't human review everything the agent ships because otherwise, it'll actually never get to production. So it requires 689 01:02:23.795 --> 01:02:25.875 rethinking and reimagining 690 01:02:25.875 --> 01:02:28.435 entire processes and entire workflows, 691 01:02:28.435 --> 01:02:31.315 and it'll be an interesting few years to come. 692 01:02:32.170 --> 01:02:33.290 Absolutely. 693 01:02:33.290 --> 01:02:34.010 Absolutely. 694 01:02:34.250 --> 01:02:39.050 Well, Tobias, I really enjoyed the conversation. Thank you so much for having me on the show here. 695 01:02:39.370 --> 01:02:59.870 Yeah. Thank you very much for taking the time today to join me and share the work that you and your team are doing on TruFoundry and TruForge and sharing some of your lessons learned on harness engineering and AI gateways. It's definitely a very interesting space and very relevant topic, so I appreciate all of the time and effort you're putting into that. I hope you enjoy the rest of your day. Thank you so much. Take care. 696 01:03:01.470 --> 01:03:13.865 Thank you for listening. Don't forget to check out our other shows. The Data Engineering Podcast covers the latest on modern data management, and podcast. In it covers the Python language, its community, and the innovative ways it is being used. 697 01:03:14.105 --> 01:03:24.731 Visit the site to subscribe to the show, sign up for the mailing list, read the show notes. And if you've learned something or tried out a project from the show, then tell us about it. Email hosts@aiengineeringpodcast.com 698 01:03:24.731 --> 01:03:25.451 with your story.