Transcrição completa de The Hidden Gaps in Agentic AI: Trust, Evaluation, and Security

00:05:41.675 --> 00:05:47.348 So So yeah, I mean we've we've definitely come a long way over the last couple years. 00:05:47.348 --> 00:06:05.721 I think um you know where we're at now is super interesting because we've had enough time to play around with LLMs you know, for for a while everybody was talking about fine tuning and, you know, taking these base models and tuning them for our own use cases. 00:06:05.721 --> 00:06:09.632 I don't really even see that conversation happening all that much anymore. 00:06:09.632 --> 00:06:12.036 I think that it's no rag either, right? 00:06:12.036 --> 00:06:14.848 Like it like a lot of that's all it was massive. 00:06:14.848 --> 00:06:17.971 Yeah, that's not being talked about that much anymore. 00:06:17.971 --> 00:06:22.574 now it's you know, the main conversation is all around harnesses, I feel like. 00:06:22.574 --> 00:06:45.233 It's all around how how to leverage harnesses, what are harnesses, what advantages does a clawed code harness have over a codex harness, over um, you know, a Hermes or a uh open claw harness, and how each of these these ultimately lead to this like recursive self-improvement idea. 00:06:45.233 --> 00:06:48.557 I think that's really where everything's sort of heaven at this point. 00:06:48.557 --> 00:06:53.528 Well, it it strikes me that it's yet another new tool. 00:06:53.528 --> 00:07:04.540 And every time that we've had new tools, we go through this cycle where it's like You think you can do everything, but what you're actually learning is how to l use the tool. 00:07:04.540 --> 00:07:13.024 You're building new muscle memory and new skills to basically do it all rather than um have it do it all. 00:07:13.024 --> 00:07:26.850 Like uh like the notions around cloud native, I brought this up before in the past in public, but you know, the the notions around cloud native existed before there was a term for cloud native, before Kubernetes existed. 00:07:26.850 --> 00:07:41.529 Yeah, people like Netflix going on to AWS and they're like we've gotta build a completely different architecture in order to use this kind of, you know, system, this this cloud system that we don't really understand, but it looks nothing like what we've got in our data center right now. 00:07:41.529 --> 00:07:44.271 And so it was, you know, building all these new patterns. 00:07:44.271 --> 00:07:52.055 And I feel like we're in that cycle right now, figuring out what the patterns are and learning those and it's evolving so quickly. 00:07:52.055 --> 00:07:53.836 And then the models are getting better. 00:07:53.836 --> 00:07:57.533 The harnesses that make the models better are getting better. 00:07:57.533 --> 00:07:59.834 And you're getting this multiplicative effect. 00:07:59.834 --> 00:08:14.298 And you know, I don't see AGI or anything like that coming anytime soon, but what I do see is this still almost exponentially increasing curve in terms of what we're um learning how to do using these new tools. 00:08:14.298 --> 00:08:21.401 And uh I'd like to say that uh I still haven't found anything that I can't get Claude or Chat GPT to do. 00:08:21.401 --> 00:08:24.721 Yeah, yeah, I would agree with that. 00:08:24.721 --> 00:08:49.268 I feel like, um, I mean, it's a rare day now that I won't at least try to think about how I can apply Claude or Codecs or something to the problem that I'm I'm facing in front of me, whether it's, you know, internal communications, whether it's a coding problem, whether it's um you know, my life management. 00:08:49.268 --> 00:08:56.541 Yeah, like so I started working on this multi agent harness for operations, an operations harness, the different from a coding harness. 00:08:56.541 --> 00:09:29.754 And I was as typical, you know, starting to fall asleep last night and I'm like, you know, the problem here is that I just haven't figured out evaluations yet and I need to like figure out an evaluations framework, not just testing, but like full evaluations so that I can understand how to improve these different uh agents who are in different roles or in this distributed system and improve their skills over time, uh, evaluate how they're doing the work that I need them to do. 00:09:29.754 --> 00:09:30.986 And I don't have it. 00:09:30.986 --> 00:09:33.399 I'm building without that evaluation harness. 00:09:33.399 --> 00:09:35.451 So I've got no way to really understand what I'm doing. 00:09:35.451 --> 00:09:37.102 They're I'm black boxing them. 00:09:37.102 --> 00:09:38.693 So I've got to fix that. 00:09:38.693 --> 00:09:39.452 Yep. 00:09:39.452 --> 00:09:43.975 a bunch of time today just even trying to figure out, okay, how do I build evaluations? 00:09:43.975 --> 00:10:00.490 And I started like, okay, ChatGPT, go out and find me, you know, kind of the best practices around building evaluations and like did all the research there and then switched over to Claude once I'd kind of educated myself and formed at least enough of of an opinion. 00:10:00.490 --> 00:10:06.354 And then I'm like, okay, I'm gonna try running Fable as my orchestrator. 00:10:06.354 --> 00:10:08.774 'Cause this is a big chunk of work to do. 00:10:08.774 --> 00:10:10.336 Let's see how this goes. 00:10:10.336 --> 00:10:18.597 And I'll tell Fable to run these different substreams that I'll manage using my little CMUX code skill that you're aware of. 00:10:18.597 --> 00:10:27.789 Um, to basically run different versions of either Sonnet five or Opus to do the implementation work to build the e file framework. 00:10:27.789 --> 00:10:37.706 And, you know, I'm still in the process of that, but you know, it's it's again, it's the thing I'm using the tools to help me Use the tools. 00:10:37.706 --> 00:10:59.024 I mean th th this is what was um like most impressive like watching you come up to speed with all this stuff is that it's it's really systems thinking at its at its core and it's really like uh leveraging existing software development, you know, best

practices. 00:10:59.024 --> 00:11:07.827 And you know, everything that you're describing comes back to tools and techniques that people in software space have been using for for years and years. 00:11:07.827 --> 00:11:10.217 It's It's you know test-driven development, essentially. 00:11:10.217 --> 00:11:12.131 For sure, yeah, in a way. 00:11:12.131 --> 00:11:30.860 And so, you know, if you can figure out a way to evaluate what these agents are outputting, and you can build your primitives and build up from your primitives and have your agent essentially evaluate how your primitives move through your system, you base you're you're building a system at this point. 00:11:30.860 --> 00:11:42.581 And it's the only difference being is that now we're leveraging these non-deterministic tools to to do certain steps along the path, helping us with planning, helping us with execution. 00:11:42.581 --> 00:11:47.726 And yeah, I think emails are a whole that I mean that's a topic we could get into for a while. 00:11:47.726 --> 00:12:01.931 I feel like that's one of those big unlocks once you start figuring out how to use LLM as a judge or starting to to leverage um, you know, even deterministic Evals in with your non deterministic systems. 00:12:01.931 --> 00:12:03.621 That's a huge unlock as well. 00:12:03.621 --> 00:12:18.427 And so And so we're trying to do that on my team quite a bit with um with one of our agentic testing harness tools that we're building, which is pointing an LLM or an agent harness at a given system. 00:12:18.427 --> 00:12:27.264 with a persona and a goal, and having it try to solve that goal via an agentic process. 00:12:27.264 --> 00:12:41.447 And the thing that you have to do if you're going to build an agentic test harness is you have to eval not only each step of each loop of the agentic um flow. 00:12:41.447 --> 00:12:44.852 But you also have to eval if it was able to achieve the goal at all. 00:12:44.852 --> 00:12:49.421 So evals become incredibly important in these agentic test harnesses for sure. 00:12:49.421 --> 00:12:50.370 for sure. 00:12:50.370 --> 00:13:06.378 Yeah, you know, it's funny that um you mentioned the LLM judge thing 'cause when I was building the failure analysis tool, I needed to pull from all these different sources to basically figure out how the system worked because it's a lot of different software. 00:13:06.378 --> 00:13:11.620 Um, you know, kind of our our marquee product, Cordon AI. 00:13:11.620 --> 00:13:21.004 And, you know, I needed to get information from Jira tickets, I need to get it from Git repos and Helm charts and you know, conversation. 00:13:21.004 --> 00:13:23.846 And Google Docs and all these different things. 00:13:23.846 --> 00:13:29.109 And I was pretty stunned at how good uh Opus 4.8 was. 00:13:29.109 --> 00:13:48.769 Um, and then I used um an another LLM judge system to kind of cross match with Codex and Gemini in basically, you know, bringing all that information into a sort of ETL pipeline and having it make calls about, you know, is this thing that is, you know, referred to as X. 00:13:48.769 --> 00:13:52.591 The same as this other thing that's referred to as why. 00:13:52.591 --> 00:14:03.504 And, you know, I look through most of its calls and and the way it's designed is it actually asks for a human to come and and be the final arbiter on on what the decision is. 00:14:03.504 --> 00:14:11.796 But when I look at its recommendations, like it's right, like eighty or ninety percent of the time it's like I saw these two things and I think they're the same thing. 00:14:11.796 --> 00:14:13.905 And I'm like, that's impressive, right? 00:14:13.905 --> 00:14:37.736 'Cause it's it's the fuzzy matching and the intelligence that it's applying is really good, in many cases better than a human, really, because a human can't process all of that information and make the connections between a a Jira ticket and something in a Git repo with high levels of accuracy without like really digging into all the details. 00:14:37.736 --> 00:14:45.899 And it's just able to go into all the details and use a larger amount of data than a human could in a far shorter amount of time. 00:14:45.899 --> 00:14:46.899 All right. 00:14:46.899 --> 00:14:48.471 So I got a question for you though, Randy. 00:14:48.471 --> 00:14:52.111 Like who do you think is is most successful? 00:14:52.111 --> 00:15:00.923 Like like what characteristics does somebody have that that are most successful in this day and age, like with LLMs? 00:15:00.923 --> 00:15:06.946 Like how like I have an I have an idea in my head, but I'm I'm curious to to hear what you think. 00:15:06.946 --> 00:15:12.587 Like what does it take for somebody to like really be successful with these tools? 00:15:12.587 --> 00:15:13.548 Like what do they need? 00:15:13.548 --> 00:15:16.919 Like what like what number one, sense of curiosity. 00:15:16.919 --> 00:15:22.323 Uh number two, uh willingness to fail, uh but I think that's cross cutting. 00:15:22.323 --> 00:15:30.471 Uh number three, that systems thinking that you were talking about, really kinda understanding uh holistic thinking. 00:15:30.471 --> 00:15:43.423 Uh number four, good critical analysis skills 'cause the the you know, the LMs are gonna get it wrong and you've got to be able to see when they get it wrong and and and tag it and and correct them. 00:15:43.423 --> 00:15:49.359 You're still the the person who's guiding and and I think I think that's really the key. 00:15:49.359 --> 00:15:56.323 Um, the people who seem to not do well try to get the LMs to do everything for them. 00:15:56.323 --> 00:16:03.197 Like I have tried time and time again to get an LM to write the way I write and it it cannot. 00:16:03.197 --> 00:16:12.572 No matter how much I you know, effort I put into a skill or whatever, like it just it your voice gets lost. 00:16:12.572 --> 00:16:17.563 And your voice is more than than the words that you use. 00:16:17.563 -->

00:16:19.134 It's how you say, it's how you think. 00:16:19.134 --> 00:16:29.208 And the LLM sort of this you know, kind of aggregation of all human beings ways of thinking and talking and and so it gets lost. 00:16:29.208 --> 00:16:36.171 But but you know, having it um be your brainstorming or your sounding board is great. 00:16:36.171 --> 00:16:40.735 Like I the one of the examples I like to give is I was writing a blog posting. 00:16:40.735 --> 00:16:43.557 I got to the point I wasn't sure how to do the conclusion. 00:16:43.557 --> 00:16:45.548 I asked it to write the conclusion for me. 00:16:45.548 --> 00:16:48.201 I read the conclusion, I was like, that is absolute garbage. 00:16:48.201 --> 00:16:50.173 That's not the point I want to make. 00:16:50.173 --> 00:16:54.056 But when I but when after I read it, I was like, this is the point I want to make. 00:16:54.056 --> 00:16:57.779 I suddenly had clarity and I wrote my conclusion myself. 00:16:57.779 --> 00:17:03.267 But it but having it kind of Be the mirror was helpful. 00:17:03.267 --> 00:17:09.598 And it also helped me figure out what was missing in some of the body that would give me that punchline and the conclusion that I needed. 00:17:09.598 --> 00:17:11.200 Yeah, for sure. 00:17:11.200 --> 00:17:15.500 I feel like it's like a plastic representation of almost. 00:17:15.500 --> 00:17:16.580 Yeah. 00:17:16.580 --> 00:17:38.885 yeah, I mean sometimes I've I've I've often thought that you know I used the I used to uh teach engineers like the whole rubber ducky method early on where you talk about your pro if if you can just talk about the thing you're trying to achieve like out loud to even if a fictional rubber duck or whatever. 00:17:38.885 --> 00:17:49.060 It can help and LLMs are like a much it's like a rubber ducky on steroids a little bit where it's like it actually gives you some ideas back and you can iterate on it. 00:17:49.060 --> 00:17:54.679 And sometimes having the having it spit out the complete wrong idea is actually pretty helpful as well. 00:17:54.679 --> 00:17:56.508 Um know what I missed, Justin? 00:17:56.508 --> 00:18:07.553 I don't know I don't know about you, but like one of the things I loved doing with engineers in the past was to be able to get up on a whiteboard and like talk through stuff together and have each other mark up the whiteboard. 00:18:07.553 --> 00:18:12.176 Like I found those like really Really useful brainstorming sessions. 00:18:12.176 --> 00:18:13.897 You can't you can't do that right now. 00:18:13.897 --> 00:18:15.568 I mean, do you ever miss stuff like that? 00:18:15.568 --> 00:18:19.109 Kind of like the human interaction piece. 00:18:19.109 --> 00:18:26.252 Like there are a sounding board, but they're like you said, it's sort of that plastic representation of a human. 00:18:26.252 --> 00:18:32.208 I I also think we're in a place right now where we're still we're probably over. 00:18:32.208 --> 00:18:34.779 on trying to do too much with these tools. 00:18:34.779 --> 00:18:38.401 And I think there probably will be a balance in in, you know, coming back. 00:18:38.401 --> 00:18:40.912 Like whiteboarding sessions are still pretty valuable. 00:18:40.912 --> 00:18:43.102 I think we probably should be doing more of those. 00:18:43.102 --> 00:18:54.255 And I think right now we're all trying to see how far we can push our productivity and the solutions we come up with, you know, leveraging LLMs and LLM tools. 00:18:54.255 --> 00:19:00.277 And, you know, maybe it's that's an interesting topic in and of itself. 00:19:00.277 --> 00:19:05.730 Like what things are we using LLMs for that we probably shouldn't be like at this point? 00:19:05.730 --> 00:19:12.219 You know where where you know when should we we we be using our own brains versus you know offloading it. 00:19:12.219 --> 00:19:19.601 I I think Andre Karpathy posted something on Twitter where it's like you can you can offload your thinking but you can't offload your understanding. 00:19:19.601 --> 00:19:23.500 And I I kinda I think about that a lot actually. 00:19:23.500 --> 00:19:24.711 And it's it's totally true. 00:19:24.711 --> 00:19:35.946 It's like I can I can I can try to have an LLM come up with you know, it can go off, do a bunch of web research, write me a blog post, whatever, but do I understand anything that it just wrote? 00:19:35.946 --> 00:19:41.690 Like I I I gotta spend the time to actually get to you know, to to grok the information. 00:19:41.690 --> 00:19:54.291 That brings up a good question 'cause you you were doing a lot of this R and D stuff and then you kinda recently got hijacked and you were asked to work on some production code that's really important. 00:19:54.291 --> 00:20:15.342 How did things change when you were doing from doing more speculative stuff when you had kind of complete freedom and you could kind of to a certain degree maybe you're reviewing some of the code but to a certain degree you're trusting the LM to what you're doing now where you need a higher level of rigor. 00:20:15.342 --> 00:20:16.981 Are you still trusting the LM? 00:20:16.981 --> 00:20:18.410 Are you reviewing the code more? 00:20:18.410 --> 00:20:20.603 Are you just using it more strategically? 00:20:20.603 --> 00:20:23.503 Like what changed in your process if anything? 00:20:23.503 --> 00:20:40.394 Yeah, so I mean in the in my R and D role I f I and I I feel like I it was sort of evolving with the models at the time as i as well and and these models are phenomenal at building proof of concepts, like really really alpha versions of an idea. 00:20:40.394 --> 00:20:46.457 And and that's still I think one of the huge sweet spots of, you know, Claude or Cloud Code or Codex or anything. 00:20:46.457 --> 00:21:19.352 It's you know, you have a you wanna bang out a a a a a proof of concept web app that, you know know pieces it together some data that and and visualize it in a nice you know react web front end something like that i mean

that's that's really it's really really good at doing stuff like that and so I was doing a lot of i ideation and when I started to get into this more production role you know, you you do have to shift your focus. 00:21:19.352 --> 00:21:24.576 I mean, there there is a a priority of quality, obviously. 00:21:24.576 --> 00:21:29.398 You can't, you know, proof of concept isn't isn't gonna cut it. 00:21:29.398 --> 00:21:37.794 And you have to you have to put systems in place so that, you know, code is created, let's say, with a consistency. 00:21:37.794 --> 00:21:59.586 You have to put um you have to make sure that you have your CI C D pipeline in place so that and it's I guess what you're what you're doing is you're you're putting guardrails around the whole agency process so that it can't um Because you don't want your agents, you don't want your LLMs to just go off in crazy forks or paths. 00:21:59.586 --> 00:22:02.132 Like you're trying to, you're trying to get work done. 00:22:02.132 --> 00:22:16.117 So you're trying to, you know, the LLM is is trained on all the information in the world, and you're trying to scope the information and the context that it works with to your specific problem set. 00:22:16.117 --> 00:22:21.959 And this is bas like you can kind of say that that's what everybody does anytime they use any LLM. 00:22:21.959 --> 00:22:28.743 LLM is you're you're trying to scope all the world's possibilities into what it is you're trying to solve at that given moment. 00:22:28.743 --> 00:22:35.990 So your context engineering becomes incredibly You do the same thing when you're trying to write production code. 00:22:35.990 --> 00:22:49.969 You're just trying to scope it like don't don't um, you know, let's say, you know, don't write code with with nested if statements, like, you know, use switch statements, something as simple as as as that. 00:22:49.969 --> 00:22:59.323 Like if if that's a coding style that your team has, um you just need to really make sure that you your your Clot Md file has that defined. 00:22:59.323 --> 00:23:08.626 Make sure that you've got your markdown files for you know, the rules and and for formatting rules and all the other things. 00:23:08.626 --> 00:23:22.127 And so I think it just it it means that you have to spend more time planning more time creating those guardrails and those boundaries so that your your agents and LLMs don't go off the rails. 00:23:22.127 --> 00:23:23.640 I mean that's probably the biggest shift. 00:23:23.640 --> 00:23:27.803 The other big shift I would say is using tools like LLM Wiki. 00:23:27.803 --> 00:23:34.907 Now that I've got to bring in context from a million different places, you know, I've got to track all this stuff. 00:23:34.907 --> 00:23:44.076 I'm using LLMs to help me with that, bringing in um, you know, Google Docs, PDFs, Slack threads. 00:23:44.076 --> 00:23:51.374 emails, you know, meeting transcripts, all the things so that you know I have a tool, I have my second brain, basically. 00:23:51.374 --> 00:23:53.912 That's that's been a huge win for me. 00:23:53.912 --> 00:24:13.752 I mean, one of the things that you're saying that resonates with me that I still don't have an answer for, but I think it's 'cause we're early on is How do I get that environment set up where I feel like I can trust the outcomes that I'm gonna get? 00:24:13.752 --> 00:24:27.396 Because right now I'm kind of doing it all ad hoc every time and learning every time, but the it there's a lot of stuff out there I can use, but there's almost too much stuff and curating it is sort of a pain. 00:24:27.396 --> 00:24:38.182 There's no like, okay, you're gonna build uh Python based system that's going to be a web based API that's gonna talk to this and this. 00:24:38.182 --> 00:24:57.516 So you know you can pull off the shelf, you know, this set of skills for writing Python, this set of skills for doing test driven development with Python, this set of skills for, you know, whatever it is, um, so that you can pull that box off the shelf and just use that box and modify it. 00:24:57.516 --> 00:25:00.935 Instead, I feel like I'm building the box every time. 00:25:00.935 --> 00:25:09.799 for each project and I'm getting better and better at building the box that's the framework, but I'm doing it all by hand. 00:25:09.799 --> 00:25:12.811 It feels like everybody else I see is kind of doing it by hand. 00:25:12.811 --> 00:25:19.204 There's no yet emergent best practices that everybody's using. 00:25:19.204 --> 00:25:22.797 There's lots of there's lots of patterns. 00:25:22.797 --> 00:25:30.064 Like we talked about sort of those cloud native patterns kind of uh turning into something like Kubernetes and we haven't really seen that yet. 00:25:30.064 --> 00:25:34.055 I I I feel like everybody's kinda doing it a slightly different way. 00:25:34.055 --> 00:25:35.929 I don't know if that's good or bad. 00:25:35.929 --> 00:25:57.806 Yeah, I mean it's a tough one because you know, every team that I've ever worked on has their own coding style and rules in place and and and I I've I mean it's gets philosophical pretty quickly like how you know, two spaces are four, right? 00:25:57.806 --> 00:26:02.374 Like like what what what's what's more appropriate, you know. 00:26:02.374 --> 00:26:03.273 you know. 00:26:03.273 --> 00:26:07.345 So I I feel like it's fine for every team to be doing it differently. 00:26:07.345 --> 00:26:08.796 I think you're right though. 00:26:08.796 --> 00:26:20.779 There's a huge opportunity for some normalization, for some standardization, and and I think that you know, this idea of rebuilding the box every single time is is pretty inefficient. 00:26:20.779 --> 00:26:30.516 You get better at it each time, but it would be nice if It'd be nice if we could all agree on like this is the set of skills that would make the most sense. 00:26:30.516 --> 00:26:45.776 I I have to say though, I have to laugh about skills a little bit because um for

the longest time it was just like a collection of prompts that I had like saved and and and uh and and these these prompts, you know, I I still use them. 00:26:45.776 --> 00:26:49.107 I've just converted them now to to the skill md or whatever. 00:26:49.107 --> 00:26:55.349 And it's funny to me how the labeling of a of of a thing, a prompt. 00:26:55.349 --> 00:27:02.678 a tech like changed the whole perception and you utility and usage of it. 00:27:02.678 --> 00:27:16.960 You know, i it's just all of a sudden i it it made it much easier for people to grasp this idea that and and like it gave people the ability to share skills and and use them in tooling a little bit more than just a prompt, I guess. 00:27:16.960 --> 00:27:22.746 But yeah, I just I d real really one of the patterns that is solidified cleanly. 00:27:22.746 --> 00:27:28.441 Like MCP still kind of hasn't found its spot, but agent skills definitely have. 00:27:28.441 --> 00:27:44.374 Like I'm at the point now where I'm doing something and I there's a skill that's being in use and it you know, the cloud or whatever does something funky and I'm like, Okay, go update the skill so you don't do that funky thing again. 00:27:44.374 --> 00:28:01.554 And so so it's it's it's kinda like you said, it's like a codification of the prompts or the chain of thought prompts that we had before, um, but in a way that's reproducible, standardized and and a pattern that's that's clearly winning. 00:28:01.554 --> 00:28:06.910 I'm just not sure where the rest of that is yet. 00:28:06.910 --> 00:28:10.422 Yeah, yeah. 00:28:10.422 --> 00:28:24.189 Um What do you feel like is the most useful like new AI tool that you've come across recently? 00:28:24.189 --> 00:28:30.134 Do you ha is there anything that you feel like you've you that everybody should know about? 00:28:30.134 --> 00:28:31.086 about? 00:28:31.086 --> 00:28:34.368 Just any AI tool? 00:28:34.368 --> 00:28:35.845 Uh I don't know. 00:28:35.845 --> 00:28:38.814 I'm fishing for for little bear. 00:28:38.814 --> 00:28:53.051 I uh so uh the one that I started using recently that I've um been having really good luck with is Alibaba's open code review. 00:28:53.051 --> 00:28:54.553 Have you seen this thing? 00:28:54.553 --> 00:28:55.875 Yeah, yeah. 00:28:55.875 --> 00:29:24.126 So they combine skills um for doing uh adversarial code reviews Um with 'cause I what I was doing now is I was manually running Claude plus Codex plus Gemini and I was like, let me go take this open code review thing for a test drive because Alibaba claims they bake the hell out of this thing 'cause it uses a set of deterministic code to do, you know, basically code analysis along with the uh, you know, L L judge capability. 00:29:24.126 --> 00:29:33.071 And I r I ran them head to head and uh open code review stuff came up with a whole bunch of things that Codex, Gemini and Claude combined didn't find. 00:29:33.071 --> 00:29:34.343 So I turned that into a skill. 00:29:34.343 --> 00:29:40.406 Um and then I added it to kind of my general dev skill that I have now. 00:29:40.406 --> 00:29:48.790 Um so there's an always a gate so that whenever a stream is gonna an agent's gonna merge something back, it runs it. 00:29:48.790 --> 00:29:53.413 And um, you know, it's it's been pretty good like so far. 00:29:53.413 --> 00:29:58.589 Like it seems to find a lot of stuff that's um That the agents didn't themselves. 00:29:58.589 --> 00:30:01.551 So I I've had a lot of good luck with that so far. 00:30:01.551 --> 00:30:02.916 Nice. 00:30:02.916 --> 00:30:03.861 Nice. 00:30:03.861 --> 00:30:05.945 I have to I have to spend more time with that for sure. 00:30:05.945 --> 00:30:14.666 Um You know, I c I can't live without my Claude code at this point for sure. 00:30:14.666 --> 00:30:23.731 I although I have to say like the the latest uh OpenAI model five six or is is very, very good. 00:30:23.731 --> 00:30:29.634 Although I just burn burned through a tremendous amount of tokens in a very short amount of time. 00:30:29.634 --> 00:30:37.679 Peter Steinberger just released a skill recently for uh codecs for coding. 00:30:37.679 --> 00:30:43.294 uh an anthropic or or quad for for planning. 00:30:43.294 --> 00:30:45.637 Um need to spend more time on that. 00:30:45.637 --> 00:30:54.307 That seems very interesting because I I I do think each model has its own personality. 00:30:54.307 --> 00:30:59.611 And in some models are are good for some things and not good at other things. 00:30:59.611 --> 00:31:01.859 it is a bit subjective. 00:31:01.859 --> 00:31:08.816 I I think one of the areas I'm most interested to hear your thoughts on, Randy, is is where. 00:31:08.816 --> 00:31:24.154 local models fit in going forward because I'm starting to see a lot more chatter about model routing and the future of model routing in in these adentic workflows going forward. 00:31:24.154 --> 00:31:41.462 I mean so much of what we do a lot of like the plumbing work, like wire this to this, like throwing um fable at at a problem like that is is a it's just a giant waste and you know you don't need Fable for a lot of things. 00:31:41.462 --> 00:31:41.982 Yeah. 00:31:41.982 --> 00:31:53.442 And you know, I think going forward model routing seems like the the the logical next step for for where a lot of this stuff is headed. 00:31:53.442 --> 00:31:53.942 is headed. 00:31:53.942 --> 00:32:12.480 So Sam Wittenve Whit Whitaven, I'm probably butchering his name, sorry Sam, if you wind up listening to this, he had this great review of Needle by the Cactus folks, which is a little small model that's I mean it's tiny that can run on like a hand. 00:32:12.480 --> 00:32:15.642 Like a one billion parameter even I think it's true. 00:32:15.642 --> 00:32:16.945 I think it's a couple of things. 00:32:16.945 --> 00:32:18.317 I think it's 25 million. 00:32:18.317 --> 00:32:21.500 It's really teeny and and all it does is tool calling. 00:32:21.500 --> 00:32:33.329 And he was talking in

generality about a trend towards um strip-down models that um knew less about the universe but could do basic reasoning and tool calling. 00:32:33.329 --> 00:32:41.798 And this was really interesting to me because I'm working on this operations harness that runs these workflows that have been pre-vetted by operators. 00:32:41.798 --> 00:32:45.271 And I'm thinking to myself, you know, that's perfect, right? 00:32:45.271 --> 00:32:57.557 Rather than me having you know, a smart model uh write some kind of meta workflow around a bunch of sub workflows. 00:32:57.557 --> 00:33:37.537 Instead, if I could have a smaller model, maybe not quite that twenty-five million, you know, a little bit smarter than that, but you know, just with core reasoning and really good tool calling ability to run a set of workflows you know, tasked by a smarter uh model, then what could happen is in when perturbations come up or some kind of anomalies, you know, I can trust this thing, you know, to very efficiently, probably using the local model even, uh, run the set of workflows and kind of be a smart workflow orchestrator so that if it came across problems, it could try to resolve them if they were very simple. 00:33:37.537 --> 00:33:40.741 But if not, punt back up to the smarter model. 00:33:40.741 --> 00:33:45.355 with kind of that original analysis and and so I I think we're gonna see a lot more of that. 00:33:45.355 --> 00:33:58.832 Um and I think that's definitely where things are gonna head where we're gonna have multiple tiers of these models that are sort of good at different things, uh, with the routing in between them, some of it kinda a little bit more hard coded and some of it smarter. 00:33:58.832 --> 00:34:04.365 Yeah, I mean that that reminds me of the whole um the auto researcher project. 00:34:04.365 --> 00:34:18.391 And I think there's there's a ton of opportunity here for uh small models, microdata sets, leveraging things for anomaly like doing anomaly detection in in edge systems. 00:34:18.391 --> 00:34:23.054 I think there this is a a a huge opportunity space still. 00:34:23.054 --> 00:34:43.289 and I th I I like your direction of you know know these models can exist on their own almost these small micro models and then orchestrating them with some of these more you know, uh smarter uh reasoning models and whatnot. 00:34:43.289 --> 00:34:44.590 I think that makes a ton of sense. 00:34:44.590 --> 00:34:47.262 And that is that is a direction I see as well. 00:34:47.262 --> 00:34:57.706 But talking about this like a long time ago, which is mixing determinism, deterministic, you know, code or or or workflows or whatever with non deterministic. 00:34:57.706 --> 00:35:11.621 And I think uh where people are evolving towards is that the Closer you are to understanding the domain and and having things be very well structured, you get to determinism. 00:35:11.621 --> 00:35:17.135 Um, and then the further away you are from it, the the more you need that non deterministic. 00:35:17.135 --> 00:35:23.483 But there's a place for intelligence at each of the layers in between. 00:35:23.483 --> 00:35:26.682 And, you know, it's not always a hammer. 00:35:26.682 --> 00:35:30.137 You want to use like, you know, a slightly different tool in each case. 00:35:30.137 --> 00:35:51.748 And um And I I I think we're gonna see that evolve and it kind of gets to what we touched on, but it's you know, it's all the rage now talk about harnesses, but I think that people are starting to see that the harness is a um a force multiplier for the model. 00:35:51.748 --> 00:35:55.175 And then, you know, the question is what kind of force multiplier do you need? 00:35:55.175 --> 00:36:18.541 Do you need do you need a, you know, like something to force multiply a fable, which is very different than something to force multiply a little small model that like I was just talking about and you know, how do you pick the right ones and that's where sort of like smart harnesses and and smart routing comes in and being able to do the systems thinking and chop things up and and use the right tool for the job, right? 00:36:18.541 --> 00:36:25.364 If you're gonna work on a car you've got like, you know, a hundred different tools you can use depending on what you're trying to do. 00:36:25.364 --> 00:36:26.724 Yeah. 00:36:26.724 --> 00:36:27.605 Yep. 00:36:27.605 --> 00:36:36.429 Yeah, I mean this goes to you know, the idea of like how do you approach a problem in an agentic way? 00:36:36.429 --> 00:36:42.673 And you know, the way that I would I would recommend is you know, you know, start with your primitives again, kids. 00:36:42.673 --> 00:36:50.019 Like figure out like what are the smallest components that you're working with and then you know, build up from there. 00:36:50.019 --> 00:37:00.447 I think that you know, most people get lost in this space when they start they start y I think you mentioned it earlier like we we have so many options. 00:37:00.447 --> 00:37:20.449 Like you harness is a big idea, it's big concept and when you're looking at how to automate something or you know you're looking at how you can leverage the the best thing that you can do is is is just try to like narrow it the scope down to its smallest component pieces and then try to automate those individual pieces and build up from there. 00:37:20.449 --> 00:37:44.804 I think that um I think that's another space that, you know know we probably all need to do better within Marantis and and I think as a community in general which is you know we need to like we need to raise the tide across all, you know, everyone and and teach people essentially how to use these tools better. 00:37:44.804 --> 00:37:48.331 Um definitely a space that I'm interested in exploring and helping out. 00:37:48.331 --> 00:37:56.882 And it's it's an area that I I try to focus on within, you know, Morantis already with with uh our internal our internal Slack channels and things like that.

00:37:56.882 --> 00:38:16.730 But I mean, do you have any ideas on on on best approaches again for for people, you know, um getting started or, you know, build like how how would you recommend somebody automate one of their, you know, uh business processes or how to automate like an existing system? 00:38:16.730 --> 00:38:20.800 Do you have a process that you go through when you try to build some of the tools that you build? 00:38:20.800 --> 00:38:21.672 you build? 00:38:21.672 --> 00:38:29.668 N no, I mean I guess it kinda reminds me of when I wanna learn like a new programming language. 00:38:29.668 --> 00:38:36.934 I go find some project that I is important to me that I care about to to to you know, basically work on. 00:38:36.934 --> 00:38:44.992 my br my brother's been in security operations for a long time and you know, he hadn't used any of the tools. 00:38:44.992 --> 00:38:55.775 So a couple of months ago I was like, dude, you gotta go play with this stuff and understand it and and he he's been so heads down and I'm like, what are you what are you doing right now? 00:38:55.775 --> 00:38:57.697 What's top of your plate? 00:38:57.697 --> 00:39:06.563 And he needed to put together some documentation about how to move from you know, one VPN provider to another VPN provider and get it into service now. 00:39:06.563 --> 00:39:13.478 And I was like, Well, cloud can do that for you if you've got all the source material and it can drive the browser. 00:39:13.478 --> 00:39:18.771 And we walked him through I walked him through that in real time in about an hour. 00:39:18.771 --> 00:39:27.465 And now he's got like boxes he's bringing online to, you know, basically run different agents and, you know, he he's drunk the Kool Aid. 00:39:27.465 --> 00:39:29.155 So I I think I think that's it. 00:39:29.155 --> 00:39:33.813 I'd say like that curiosity thing we talked about is It's just the number one thing. 00:39:33.813 --> 00:39:43.409 Like you've got a problem right now and whatever your area, you know, of expertise is, you know what the problem is, you're the domain expert. 00:39:43.409 --> 00:39:53.038 You can literally just bring up Cloud or Chat GPT and say, This is the problem I'm trying to solve and and just outline it and say, Give me options. 00:39:53.038 --> 00:39:56.721 Tell me what are the different ways that I could try to solve this. 00:39:56.721 --> 00:39:58.672 And and then just follow your nose. 00:39:58.672 --> 00:40:02.715 And I I literally think that's the best way to like get started. 00:40:02.715 --> 00:40:16.070 Um and then um I think the second part of that is to look for something you think is too hard or that you can't do or that you're not sure it can do, and then go see if you can. 00:40:16.070 --> 00:40:22.385 Because like I said, I I'm having a hard time finding stuff that they can or can't do. 00:40:22.385 --> 00:40:24.717 I'll give you an example just super quick. 00:40:24.717 --> 00:40:30.751 I need to come up with um top ten uh, you know, failure analysis. 00:40:30.751 --> 00:40:33.355 to you know give to some folks internally. 00:40:33.355 --> 00:40:37.188 I'm using somebody else's source material. 00:40:37.188 --> 00:40:44.916 I went in with one of my colleagues a couple hours ago and we just marked up the Word document. 00:40:44.916 --> 00:40:55.264 And you know, I highlighted the stuff that was green that definitely should be addressed, yellow that you know maybe should be at least partially addressed, red that, you know, should be ignored. 00:40:55.264 --> 00:40:58.797 And then I went into Chat GPT and I said, can you read this doc? 00:40:58.797 --> 00:40:59.867 It's like yes. 00:40:59.867 --> 00:41:02.518 I said, okay, I need to extract this information. 00:41:02.518 --> 00:41:03.760 If it's green, it's this. 00:41:03.760 --> 00:41:04.590 If it's yellow, it's this. 00:41:04.590 --> 00:41:06.090 If it's red, ignore it. 00:41:06.090 --> 00:41:14.293 You know, provide me, you know, a markdown I can feed into Cloud Design to give me a presentation that basically summarizes all of this. 00:41:14.293 --> 00:41:20.777 And bam, fifteen you know, five minutes later I've got the markdown, I shove it into Cloud Design. 00:41:20.777 --> 00:41:24.869 You know, fifteen minutes later I've got a high quality presentation. 00:41:24.869 --> 00:41:26.480 That's awesome. 00:41:26.480 --> 00:41:27.782 And that's awesome. 00:41:27.782 --> 00:41:30.445 You know, like it how much time was I just saved? 00:41:30.445 --> 00:41:30.985 Hours? 00:41:30.985 --> 00:41:33.757 Yep. 00:41:33.757 --> 00:41:35.699 Yeah, for sure. 00:41:35.699 --> 00:41:59.085 I know a topic I wanted to to bug you about is um security, because it's kind of near and dear to our heart heart right now and and how companies think about AI and policy and how how what's the you know, one of the the big topics, and I don't know if we have uh time to dig into this right now, but um, you know. 00:41:59.085 --> 00:42:54.405 know every company on the planet right now is trying to figure out how to wrangle AI within their company given the fact that it opens them up to potentially more attack vectors you know you have the whole concept of data sovereignty and you know, all these these frontier companies sucking up all of our information and you know like w what is the what's the way that companies should be thinking about all the AI tooling that's available, all the software tools that integrate AI, like what what you know I I feel like we're at this intersection of um, you know, you don't want IT and security to to to slow down progress. 00:42:54.405 --> 00:43:04.771 but at the same time we all understand that we can't open the company up or any you know, every company can't open themselves up to to more attack vectors. 00:43:04.771 --> 00:43:07.594 So how what's what's the best way for companies to think about this? 00:43:07.594 --> 00:43:14.914 Uh I I I don't know if I don't know if people are gonna like my answer

here. 00:43:14.914 --> 00:43:20.297 Um, so I did security for a large part of my career, uh, mostly on the Y Hat side. 00:43:20.297 --> 00:43:28.581 And one of the things I notice is that whenever there's sort of a new tool or something like that, um security is is always lagging behind. 00:43:28.581 --> 00:43:42.599 Not not in terms of the teams and what they're doing, but more like the concerns of the business because the business value of the new capability is so high that people are the the business executives are willing to take on risks. 00:43:42.599 --> 00:43:46.541 Cloud you know, case on point, you know, the adoption of cloud computing. 00:43:46.541 --> 00:43:50.454 Um, there was all this fear, uncertainty, a doubt about cloud computing. 00:43:50.454 --> 00:43:59.159 Um, you know, there people are thrashing about it, but, you know, shadow IT happened, people were routing around, they were just using it. 00:43:59.159 --> 00:44:01.369 They just went because there was too much value. 00:44:01.369 --> 00:44:17.806 And I think we're in a similar situation where there's these attempts to try to lock down and and and you know um make it harder for people to use the tools to uh and you know create controls and all that. 00:44:17.806 --> 00:44:28.230 And I what's funny is I I I think that there's still not this understanding that the more you lock it down, the less utility there is. 00:44:28.230 --> 00:44:44.443 But then I think what's even funnier than that is probably the solution is more AI, which I hate to say, but You know, right now we've got this kind of proxy gateway inside of Mirantis that, you know, proxies our usage to the different endpoints to do some enforcement. 00:44:44.443 --> 00:44:49.076 But I'm kinda stunned at how little intelligence is actually in the gateway. 00:44:49.076 --> 00:45:00.563 Um, 'cause if you look at what I'm doing with Claude on a day to day basis, it'll be like, you know, you just pasted uh, you know, secret and you know, you should rotate that immediately. 00:45:00.563 --> 00:45:02.764 And I'm like, Dear Claude. 00:45:02.764 --> 00:45:07.577 That's a temporary secret for a dev environment that's gonna be thrown away in fifteen minutes. 00:45:07.577 --> 00:45:09.416 You don't need to worry about it. 00:45:09.416 --> 00:45:31.028 And I I just think that, you know, the white hats haven't caught up to the fact that they should be putting intelligence into those gateways that can actually look at what's going on in real time and and make some decisions about like what's happening, you know, is this really something that that needs to be dealt with, you know, or can can it can you have a pass? 00:45:31.028 --> 00:45:37.807 So like the the leveling up of the security teams hasn't happened in a lot of places. 00:45:37.807 --> 00:45:59.367 Although I didn't send this to you, there's a whole new um sorry I'm I'm I'm ranting and bogarding the time, but there's a whole new exploitation framework that just came out um that is gonna uh come out uh and at uh Black Hat and DEF CON coming up that's AI powered, you know, hacking harness for basically attacking. 00:45:59.367 --> 00:46:02.335 And and it's called like Kitty Sploit or something like that. 00:46:02.335 --> 00:46:03.596 A buddy of mine sent it to me. 00:46:03.596 --> 00:46:23.690 Um and and so the the research guys, the the the gray hat black hat types, they're already figuring out how to use the tools, but a lot of the white hat guys, they wanna apply the old ways of doing security and compliance and all the things that are too slow, too static, too rigid, you know, to sort of deal with it. 00:46:23.690 --> 00:46:43.219 And I think that, you know, they all gotta level up and they gotta start thinking about, you know, how to build uh you know, the smart, you know, AI powered tools to basically allow people to get more done, but to be able to flag or and or intercede when something, you know, real is happening Yeah. 00:46:43.219 --> 00:46:43.590 Yeah. 00:46:43.590 --> 00:46:50.045 I think it's it's just a matter of um everybody's kind of playing catch up. 00:46:50.045 --> 00:47:00.614 And there's there's the people who've been hyper focused in this space who are starting to leverage the tools and understand the tools in ways that other people are not. 00:47:00.614 --> 00:47:06.197 And then there's the rest of the people who are just trying to catch up and figure out what's going on. 00:47:06.197 --> 00:47:25.635 And unfortunately we're we are stuck in some, you know, legacy outdated approaches to to to trying to r wrangle this stuff, but it's it's a space that I feel like that this absolutely is not solved yet, you know, and I think a lot of companies are wrestling with the same same thing. 00:47:25.635 --> 00:47:27.556 So cool. 00:47:27.556 --> 00:47:30.617 Guys, I kinda have to call it here. 00:47:30.617 --> 00:47:32.577 yeah. 00:47:32.577 --> 00:47:34.798 Shall we do a shall we do a wrap up? 00:47:34.798 --> 00:47:35.659 Yes. 00:47:35.659 --> 00:47:36.257 Yeah. 00:47:36.257 --> 00:47:36.996 gonna say no problem. 00:47:36.996 --> 00:47:42.048 I didn't want to cut you guys off mid uh mid thought or anything, but yeah, we are at time. 00:47:42.048 --> 00:48:08.121 Um I mean if you if someone has a succinct thing in their brain to to wrap everything up, we can, but um if not, maybe not the last like chunk we just talked about, but the one before kind of Randy's like anecdote to finish like like kind of in my head like was a decent wrap up so don't need it if we don't want it but do a little like thirty second outro if you want. 00:48:08.121 --> 00:48:17.628 number one number one things we've learned over this journey, maybe. 00:48:17.628 --> 00:48:18.907 Justin? 00:48:18.907 --> 00:48:19.989 Yeah, sure. 00:48:19.989 --> 00:48:30.704 So Justin, um as we're wrapping up this episode, maybe you could tell me kind of your number one learning over your AI and agentic journey. 00:48:30.704 --> 00:48:42.833 You know, I you touch on this a lot, but it's it it's main maintain that curiosity, I think, is kind of the number one learning is is these tools are incredibly powerful. 00:48:42.833 --> 00:48:48.748 Um I think we're doing a pretty good job at Marantis in in leveraging them. 00:48:48.748 --> 00:48:52.300 You know, we are an an AI forward company for sure. 00:48:52.300 --> 00:49:05.027 And I think that just encouraging and spreading that curiosity and trying to trying to get people to like to really use these tools uh I think there's a huge opportunity for everybody to level up. 00:49:05.027 --> 00:49:15.706 And you know my my I'd say another big learning is um you know, just pay attention to what the smartest people in the space are are are putting out there. 00:49:15.706 --> 00:49:20.789 Um you know, read a lot read a lot. 00:49:20.789 --> 00:49:22.690 Read a lot. 00:49:22.690 --> 00:49:24.414 That's kind of it for me. 00:49:24.414 --> 00:49:25.364 for me. 00:49:25.364 --> 00:49:33.918 Yeah, so I have a reputation for being a contrarian, but I think I'm gonna echo your number one, which is uh the curiosity piece. 00:49:33.918 --> 00:49:42.000 Like I just you know, like I said, I was a skeptic walking in but you know, my curiosity got the better of me. 00:49:42.000 --> 00:49:46.202 I started playing with the tools and it's just continued to escalate. 00:49:46.202 --> 00:49:51.813 So I think it's just a sea change and it's the early days. 00:49:51.813 --> 00:49:58.235 I don't think people have really sort of completely internalized that it's like the early, early days. 00:49:58.235 --> 00:50:02.800 I'd say it's twenty eleven, twenty twelve of cloud computing. 00:50:02.800 --> 00:50:06.454 There's Two or three more years of craziness. 00:50:06.454 --> 00:50:13.862 Like, I mean, just where we are between now and middle of last year, one year ago, like it's night and day. 00:50:13.862 --> 00:50:25.139 I I don't know that the pace will continue quite at this rate, but even if it was fifty percent over the next couple of years, it's gonna be really interesting where we land. 00:50:25.139 --> 00:50:28.492 Totally agree. 00:50:28.492 --> 00:50:30.648 All right, thanks. 00:50:30.648 --> 00:50:31.719 Appreciate you, Justin. 00:50:31.719 --> 00:50:33.291 That was that was a lot of fun. 00:50:33.291 --> 00:50:34.382 Thanks, Randy. 00:50:34.382 --> 00:50:35.623 Yeah, good stuff. 00:50:35.623 --> 00:50:37.934 All right, see you all y'all next time. 00:50:37.934 --> 00:50:38.898 See you. 00:50:38.898 --> 00:50:39.617 Cool. 00:50:39.617 --> 00:50:40.577 Thank you guys. 00:50:40.577 --> 00:50:41.440 Thank you guys. 00:50:41.440 --> 00:50:42.963 Yeah, thank you. 00:50:42.963 --> 00:50:44.177 See ya.