Your Coding Agent Should Never Hold a Credential의 전체 녹취록

Docker. 00:02:25.360 --> 00:02:33.280 I help uh uh Docker to work with some of our most strategic uh customers and help kind of direct some of the initiatives we do. 00:02:33.599 --> 00:02:34.240 Amazing. 00:02:34.400 --> 00:02:40.000 Uh next uh Ellie is uh In Livingston, uh who actually was on the podcast not so long ago with Guypo. 00:02:40.159 --> 00:02:41.199 Uh you work for KeyCard. 00:02:41.280 --> 00:02:41.840 Tell us a little bit about yourself. 00:02:42.719 --> 00:02:43.280 CEO of KeyCard. 00:02:43.360 --> 00:02:44.800 I've been working on agent identity and access. 00:02:44.879 --> 00:02:46.560 So, how do I adopt agents? 00:02:46.639 --> 00:02:47.360 How do I build agents? 00:02:47.439 --> 00:02:50.560 How do I control what those agents can do across the board? 00:02:50.719 --> 00:02:55.360 How do I make my developers and my business users successful as possible with agents today and into tomorrow? 00:02:55.680 --> 00:02:56.159 Awesome. 00:02:56.319 --> 00:03:00.800 And finally, but by no means least, uh, we have Marcelo Sosa from Sneak. 00:03:00.879 --> 00:03:02.319 Tell us a little bit about yourself, Marcelo. 00:03:02.560 --> 00:03:14.400 So I basically help customers uh safely adopt AI, build agents, uh, you know, bring uh you know defensive capabilities into the hands of the defenders and uh just try to protect the world, you know, bit by bit. 00:03:14.639 --> 00:03:15.039 Awesome. 00:03:15.199 --> 00:03:17.039 Now we're gonna be talking about agent baseline. 00:03:17.120 --> 00:03:18.400 Ian, why don't you kick us off? 00:03:18.479 --> 00:03:20.719 Um in a sentence, what is agent baseline? 00:03:21.039 --> 00:03:22.479 It is six controls. 00:03:22.560 --> 00:03:32.719 Uh, and those six controls are ways for you to think about what security controls are need in place in order for me to take advantage and use autonomous agents in my organization or agents at all of my organization. 00:03:33.280 --> 00:03:34.560 Do we really need secure agents? 00:03:35.039 --> 00:03:36.000 No, you didn't need to answer that. 00:03:36.080 --> 00:03:38.319 Uh Martello, who is this who is this for? 00:03:38.479 --> 00:03:42.639 Is this for a CISO or is this for engineering teams that are actually building with agents today? 00:03:43.039 --> 00:03:53.120 I mean, the way I see it is that you know it's really for people that are trying to like understand like what are the security boundaries that they want to have when they're basically adopting agents and getting agents to do you know autonomous tasks. 00:03:53.280 --> 00:03:56.000 Uh obviously we are focusing a lot on the software factory. 00:03:56.319 --> 00:04:10.639 So like, you know, people that are actually using coding agents, and so like you know, why there's a security persona, obviously it also matches up on the platform teams, you know, like how what how they use like dev containers, for example, and kind of how they get the sort of boundaries of where agents uh uh are operating. 00:04:10.800

--> 00:04:16.800 So we kind of see these multiple personas, uh both on the security and the platform teams, like playing a role here. 00:04:17.120 --> 00:04:18.800 And you mentioned software factories there. 00:04:18.879 --> 00:04:25.759 Um Ellie, what what would you say is the definition of a software factory in given the context of this uh this paper, this this meeting? 00:04:26.160 --> 00:04:30.319 I think uh many people define software factories in various different ways. 00:04:30.480 --> 00:04:34.959 Uh fundamentally, it's what organizations use to produce code. 00:04:35.120 --> 00:04:40.319 And there's various various levels of autonomy uh that exist in software factories. 00:04:40.399 --> 00:04:47.439 So there's anything from the first level, which is giving developers uh coding agents, right? 00:04:47.600 --> 00:04:48.959 And how do you secure that? 00:04:49.120 --> 00:04:56.319 How do you allow developers to be able to produce code where they're the ones directing uh the whole process end-to-end? 00:04:56.800 --> 00:05:10.480 The second level of autonomy, which is uh you are at that point having a bit of a workflow, and in that workflow, you're using agents uh to produce code, to test code, to do various parts of your implementation. 00:05:10.720 --> 00:05:14.480 And then uh, but the second part is that the workflow is predefined. 00:05:14.879 --> 00:05:28.959 And the third stage of software factories is where the actual there is no predefined workflow, it's actually the agent who is making the decision on the next step to take, the right things to do, the right agents to call, and so on and so forth for that. 00:05:29.120 --> 00:05:42.480 So, and we're seeing companies be in various stages of that maturity, and actually what we're seeing is we're seeing that the depending on the task, it might make sense to be on the on a various levels of that maturity curve. 00:05:43.199 --> 00:06:05.839 We're yet to see many organizations that have made it all the way to the third one and have done that successfully, but we're seeing companies be on that journey, and what we wanted to do is we wanted to put a paper together, kind of applying the principles that Ian talked about of agent baseline, on how to think about how to build software factories utilizing the principles we outlined in the paper. 00:06:06.079 --> 00:06:09.199 Yeah, and so for people who I mean this was released this week, right? 00:06:09.279 --> 00:06:10.319 Uh in fact Tuesday. 00:06:10.560 --> 00:06:14.480 So, yeah, it is it is, I say this week, like like this is being live streamed. 00:06:14.639 --> 00:06:18.639 At the time of recording, it is the 15th of October on the on the on the Thursday. 00:06:18.720 --> 00:06:21.199 This was released on the on Wednesday, the 14th. 00:06:21.439 --> 00:06:25.120 Um now, in this, there are like you just mentioned, there are many levels. 00:06:25.279 --> 00:06:26.399 There are three levels. 00:06:26.480 --> 00:06:29.920 There's also you know

six kind of pillars of of what people should be thinking about. 00:06:30.000 --> 00:06:31.439 We're gonna go a little bit deeper into that. 00:06:31.600 --> 00:06:38.399 But for folks who want to uh read all about this, uh agentbaseline.org, and you can take a look at the the V1 of the reference architecture. 00:06:38.480 --> 00:06:40.160 And I think this is open for comment now. 00:06:40.240 --> 00:06:44.240 So please take a read of that, have a look at that and and give your feedback um to that. 00:06:44.879 --> 00:06:46.240 The team are very interested in that. 00:06:46.399 --> 00:06:52.879 But let's jump into this a little bit into a little bit more depth and I'll and I'll maybe pick on a few a few people here based on your experience. 00:06:53.040 --> 00:06:56.240 Um so Ellie, um sandbox network boundaries. 00:06:56.399 --> 00:06:59.600 Um talk to me as if I've never set one of these up. 00:06:59.680 --> 00:07:03.040 Why is it important and and and explain to me the uh what I need to do? 00:07:03.680 --> 00:07:07.040 Uh I uh uh uh hugging face? 00:07:07.360 --> 00:07:10.079 I'm hearing I've I read I read something about that. 00:07:11.120 --> 00:07:13.040 So yeah, so I mean don't further. 00:07:13.279 --> 00:07:18.240 Uh maybe take a step back and think about what we are actually doing over here. 00:07:18.399 --> 00:07:23.040 So if you think about agents, uh on one hand, they're just software. 00:07:23.120 --> 00:07:36.639 Yeah, on the other hand, they're directed by undeterministic uh uh actor or undeterministic thing that they're trying to use, which is fundamentally probabilistic in what it's trying to do. 00:07:36.959 --> 00:07:55.360 And while in most scenarios and the wide majority of scenarios, uh an agent will do exactly what you ask it to do, in some percentage of use cases where the percentage is not zero, it will do things that are not your intended action. 00:07:56.000 --> 00:07:59.759 And it might be in most in most situations, it would be in good faith. 00:08:00.079 --> 00:08:02.800 It will try and do it because it's trying to achieve a task. 00:08:02.879 --> 00:08:13.759 It's it's it's really good at pushing through and trying to achieve a task, even if it means potentially, you know, doing some malicious actions. 00:08:14.000 --> 00:08:31.360 So, what you're trying to do, you're basically trying to take an actor that could potentially be malicious and could do it at very high speed and do it in swarms, and you're trying to give predictable uh things that this actor can and cannot do. 00:08:31.439 --> 00:08:37.919 So, what you're trying to do is you're trying to manufacture trust of an actor that is fundamentally unpredictable. 00:08:38.080 --> 00:08:45.600 So the way we think through this is what's needed is is needed, what you need is you need a runtime that can enforce boundaries. 00:08:45.919 --> 00:09:03.519 And the boundaries are you can't get to any network you want to get to, you can't get to secrets you shouldn't be getting access to, you can't get to uh uh folders or I don't know, keys that are stored in your local machine in order for you to be able to trust that particular agent. 00:09:03.600 --> 00:09:22.080 So, in our mind, in order to manufacture trust, in order for you to be able to have autonomy for those agents, you need to put them in environments where uh you they can only do a certain set of tasks, which are predetermined ahead of time. 00:09:22.399 --> 00:09:36.159 And in order to get that autonomy, um it it potentially potentially will need certain access to things like keys and credentials, but but Ian, why should an agent never hold a credential? 00:09:36.240 --> 00:09:42.320 And what is the what is the way we can allow autonomy for agents without them having that access? 00:09:42.639 --> 00:09:54.000 Yeah, I mean I think fundamentally, if you uh using Cloud Code, Pi, you know, harness on your computer today, uh running inside your user space on the operating system, and you give it your your secret. 00:09:54.240 --> 00:10:06.720 That secret, whether it's like uh let's say it's a Stripe API key or keys to AWS, there's nothing that prevents that agent from performing any action those keys enabled it to do, but that may not be actually your intention of what you want the agent to do. 00:10:06.799 --> 00:10:19.039 And so there's a very famous incident called Pocket OS, um, where unfortunately, you know, the user of the agent wanted the agent to do something in staging, it found keys in production, ended up leading a production database caused a significant amount of damage. 00:10:19.279 --> 00:10:20.159 This happens all the time. 00:10:20.240 --> 00:10:27.840 There's a lot of stuff going on in personal agents today where the agents are doing things, the personal agent's instinct, muse, and friends are going on doing things the user didn't necessarily intend. 00:10:28.000 --> 00:10:31.679 So at the end of the day, it's a question of like what's the boundary and how do I prevent it from doing things. 00:10:31.759 --> 00:10:40.879 And traditionally in today and into the future, it's always what permissions does this agent have in the context of what it's attempting to do, and what how is it passed that around? 00:10:40.960 --> 00:10:49.279 I think the other component of it is you don't want the agent to be able to find keys it shouldn't have access to, but also you don't want it to accidentally leak keys it has access to. 00:10:49.440 --> 00:10:59.440 So an agent's not particularly smart, and it could just write a key into disk, or it could send a key off by Google query parameter or put it on the open web without realizing it. 00:10:59.600 --> 00:11:04.559 And so from that perspective, you have to, you know, put in a sandbox, make sure there's no secrets inside the sandbox. 00:11:04.639 --> 00:11:08.559 Uh and then most importantly, you want to have that on the egress point as traffic flows out. 00:11:08.639 --> 00:11:13.120 That's where you

really want to have your enforcement and also do add any type of secrets that are required for oldbound traffic. 00:11:13.600 --> 00:11:17.759 That way you don't have this issue where the agent's leaking keys or using keys, you can also give it a. 00:11:19.840 --> 00:11:20.159 Exactly. 00:11:20.320 --> 00:11:23.759 And then you also have the the beauty of being out of an authorization boundary. 00:11:24.000 --> 00:11:28.399 So many keys today, of course, give you more access to do things than they should. 00:11:28.559 --> 00:11:33.200 And so you can also implement authorization boundaries to scope those keys down to exactly what they need to do at any point in time. 00:11:33.519 --> 00:11:35.440 So it's almost like a child, it's almost like you're treating a child, right? 00:11:35.519 --> 00:11:43.120 You're giving it boundaries, uh, whether that's whether that's you know, giving it containment or or you're you're stopping it from getting access to things it shouldn't do. 00:11:43.519 --> 00:11:45.919 Um Marcelle, let's let's talk about supply chain. 00:11:46.000 --> 00:11:49.759 This is another this is another key piece of this of this of this reference architecture. 00:11:49.919 --> 00:11:53.919 What does supply chain mean when the thing picking these dependencies is an agent? 00:11:54.000 --> 00:11:55.679 What is it how how does that relate? 00:11:56.000 --> 00:12:01.120 You know, the way I see it is what is the sort of you have to you have to think what you're trying to protect, right? 00:12:01.200 --> 00:12:14.000 It's like so like when you have, for example, you put an agent on some sort of sandbox environment, like uh the risk of the agent installing a malicious package there is bad, but it's very different than you know, like you don't have any sandbox uh whatsoever, right? 00:12:14.080 --> 00:12:18.799 You're basically installing a malicious package on like you know a developer machine, for example, which is a formidable target today. 00:12:18.879 --> 00:12:26.399 So for me, when I think about supply chain, I think about okay, what is this coding agent actually pulling in and what it's actually using as a way to like uh communicate, right? 00:12:26.480 --> 00:12:32.480 So like you know, when you think about MCP servers, when you think about uh skills, like what are the skills that you know is actually loading. 00:12:32.639 --> 00:12:46.960 And what a lot of people don't realize these days is that you know, like when you set up these coding agents to do a task, and one of the tasks may involve like cloning a uh third-party repo, like an open source repo, and then suddenly that open source repo you know has configurations of agents themselves, right? 00:12:47.039 --> 00:12:53.120 You know, like you know, skills and and uh and and mcp server connectors, maybe the agent is gonna start you know actually consuming those as well. 00:12:53.200 --> 00:12:57.039 So like you have this sort of supply chain that is like almost dynamic that is happening. 00:12:57.120 --> 00:13:02.720 It's not like you can just look at your package.json and see what package is actually like being being pulled from that particular application. 00:13:02.799 --> 00:13:08.080 So we think of it as like different types of risk that you're trying to contain uh when when you have that. 00:13:08.320 --> 00:13:14.159 And um one of the very interesting things that happen uh is this shy wooled uh warm, right? 00:13:14.240 --> 00:13:23.919 Where basically it was a very specific attack that at some point actually just spawns a local MCP server and try to actually do exfiltration of like you know sensitive data based on that. 00:13:24.000 --> 00:13:36.399 So like we actually always should be treating things like you know untrusted MCP servers that the agent is connecting to as like you know a potential like you know severe uh threat factor to to kind of really uh mitigate uh immediately. 00:13:36.480 --> 00:13:42.000 So those are some of the things we think about when you think about uh supply chain of of these agents running. 00:13:42.559 --> 00:13:54.240 Maybe uh just to continue your analogy of kids, you know, uh one way to think through this is like if you have a kid, you have to be worried about the things that it consumes. 00:13:54.399 --> 00:13:54.879 Yeah, right? 00:13:55.039 --> 00:14:07.279 So supply chain, it can consumable like the fact that that that you've put an agent inside of a sandbox does not mean that it can't consume things that would make it do things that are pretty that that are very bad. 00:14:07.360 --> 00:14:15.519 You have to you have to worry about the supply chain, you have to worry about the things that it can do on the on the on uh when it does actions. 00:14:15.759 --> 00:14:21.440 But do assuming that just sandboxing is enough, uh, in my mind is completely insufficient. 00:14:21.600 --> 00:14:21.759 Yeah. 00:14:22.720 --> 00:14:28.240 We we know that you know agents are very more and more happy to escape sandboxes, right? 00:14:28.320 --> 00:14:37.919 Like, you know, as soon as you give it some particular access to a machine which is insecure themselves, right, they basically just port themselves into that and uh and actually start doing damage from that instead of like the sandbox environment. 00:14:38.159 --> 00:14:42.879 So let's continue this analogy with uh treating agents as kids. 00:14:43.039 --> 00:14:46.480 So you remember someone has to orchestrate the children. 00:14:48.000 --> 00:14:48.720 This is where we're going. 00:14:48.879 --> 00:14:48.960 Yeah. 00:14:49.679 --> 00:14:52.720 I often refer to myself as a babysitter when I should actually say parent. 00:14:52.799 --> 00:14:55.519 So but yeah, orchestrator I think is really my new term. 00:14:56.159 --> 00:15:06.320 Um okay, so Ian, the the the the paper the paper admits that sometimes it you know we still don't know how to safely work out an agent's task scope from untrusted input. 00:15:06.480 --> 00:15:13.279 And I

think this is uh I I love referring to this almost like as control plane and data plane and how they actually how they actually mix. 00:15:13.440 --> 00:15:22.720 And I guess for the kid, this is um you know whether they take instruction from a from a responsible person or maybe an irresponsible person and how they actually how they actually determine the difference. 00:15:22.879 --> 00:15:38.960 So given that we the even the paper admits and we still admit it it's very hard to be able to distinguish between that, what is a what is a reasonable answer of actually how we can we can help ourselves in in determining what's what you know what's a responsible way to deal with this potentially malicious input or or untrusted data? 00:15:39.200 --> 00:15:53.120 Yeah, I think I think the the admission in the paper, and I think this is true to synagentic security, is we have to accept the fact you can't trust the agent, you can't trust the inputs, and you want to control um, you do want to control the inputs can come from, but it's an almost an unsolvable problem. 00:15:53.360 --> 00:15:54.559 Some of the inputs can be controlled. 00:15:54.639 --> 00:15:57.200 For example, you know, the base image inside the sandbox. 00:15:57.279 --> 00:15:59.200 Like what's the environment inside the sandbox? 00:15:59.360 --> 00:16:01.679 Are there potential supply chain vectors coming in? 00:16:01.840 --> 00:16:05.600 Well, something that's an example that's very difficult to control is like uh prompt ejection. 00:16:05.840 --> 00:16:06.720 Yep, right? 00:16:07.120 --> 00:16:10.399 It's I don't know if we'll ever be able to actually solve that problem specifically. 00:16:10.480 --> 00:16:15.679 And so instead, what you have to say is you want to basically say, we're not gonna have it be wide open to everything. 00:16:15.840 --> 00:16:40.720 Uh, we're going to create hardened boundaries on everything we can, and where we can't, we're gonna try and pursue the principle of least privilege and have, to the best of our ability, have um over time, we're gonna have agents that are actually monitoring other agents, and those agents are monitoring other agents are gonna be very responsible for trying to ensuring aligning that the current agent's aligned to the task and it doesn't get the same context, the same openness to prompt injection that this one does. 00:16:40.799 --> 00:16:43.039 And that's how you kind of keep those two systems in alignment. 00:16:43.279 --> 00:16:48.080 And I think broadly speaking, the best way to think about it is nothing about this is trustable. 00:16:48.240 --> 00:16:59.679 So the only thing you can focus your trust on is how do I create as much of a hardened boundary around the untrusted thing so that I can harness the value of an agent, which is really this do while loop that's just gonna experiment things on a hypothesis over and over again. 00:16:59.919 --> 00:17:08.799 Want that to enable it and get as much of a sandbox or playground as you will, for it to try building a sand castle and iterate its way to success in a non-destructive fashion. 00:17:09.200 --> 00:17:10.240 A couple of interesting things then. 00:17:10.319 --> 00:17:14.000 So least privilege, this is this is something that comes up again and again and again when we talk about AI. 00:17:14.079 --> 00:17:20.319 And it's just like it's a it's a it's just a good bra, it's a really good practice that we should be doing anyway, but we shouldn't stop doing it because of agents. 00:17:20.480 --> 00:17:25.759 And the other thing that I really like you saying, so that that that stops things from blatantly doing things it shouldn't do. 00:17:26.160 --> 00:17:36.079 And the second piece that you kind of mentioned is almost like having an agent as a judge to make sure it's it's it's likely doing the right thing, not just doing something completely uh ridiculous. 00:17:36.480 --> 00:17:47.599 Well, and I think one of the important things that actually come out in the last four weeks or so that I think is important, and I think we'll define the next 12 months of like of agents and what we're going through is decision models. 00:17:47.759 --> 00:17:47.920 Right? 00:17:48.079 --> 00:17:58.559 I think those those actually give us a capacity to have something that's far cheaper than a language model and is better atuned to making you know decisions on whether something is aligned with intent or not. 00:17:58.799 --> 00:18:02.079 Yeah, with confidence levels as well, which is super interesting, super important. 00:18:02.319 --> 00:18:09.920 Um Ellie, if um one of the things here that we that we talked about is is is high impact actions that that need human approval. 00:18:10.079 --> 00:18:12.960 This is always a this is always a fun um boundary. 00:18:13.119 --> 00:18:17.279 How do you how do you stop that um, I guess, for things that need human approval? 00:18:17.359 --> 00:18:25.440 How do you stop that from becoming a button that people just end up blindly clicking without without actually reasoning about it or looking at it in that much depth? 00:18:25.680 --> 00:18:34.559 So I think like what we've all seen uh I mean manifest itself from uh early days of cloud code. 00:18:34.799 --> 00:18:38.880 Uh like we're talking in this is this is October of 26. 00:18:39.200 --> 00:18:42.400 Early days of cloud code was maybe like a year ago. 00:18:42.720 --> 00:18:46.960 Uh but in AI years, this is this is a long time ago. 00:18:47.119 --> 00:18:59.680 Uh the the uh approval fatigue one would have from like the uh I I call it there is two modes that agents operated in up until recently. 00:18:59.839 --> 00:19:09.759 I call them the YOLO mode, uh like you just let it rip, or the CYA mode, which is the mode like you're basically approving every single action. 00:19:09.920 --> 00:19:14.319 So like someone could eventually blame you for saying yes to that. 00:19:14.559 --> 00:19:33.279 But once once you are, you know, like the the reality is like

as soon as you start asking people to the yes, yes, yes, yes, yes part of this, uh you I mean, are are you know it's it's been long known that people don't usually read any warnings you throw their way. 00:19:33.440 --> 00:19:35.920 It's even better if you throw it to them every like minute. 00:19:36.079 --> 00:19:40.079 The chances of them reading any of that stuff is almost non-existent. 00:19:40.240 --> 00:19:55.599 So like the fatigue of approving every single thing is becoming it is very real, and this is why people are uh looking for ways to run things with autonomy and not approving every and especially and the models have become much better, yeah. 00:19:55.680 --> 00:20:19.920 Right, where they can do tasks that are much longer, but you still want to maintain control and you want to be to know that this particular model will not do things that will cost you your job, uh, you know, uh, or other very adversarial things that could happen if an agent escapes or an agent is allowed to do things that it's not supposed to be doing. 00:20:20.960 --> 00:20:24.960 Um let's talk a little bit about skills. 00:20:25.039 --> 00:20:30.880 Um now, how much should I guess security teams care about skills and you know, poison skills? 00:20:31.039 --> 00:20:32.799 Is that a supply chain problem? 00:20:33.759 --> 00:20:40.720 I mean, I think if you take a step back and you think about why skills are important, it's to basically do continuous learning of the agent, right? 00:20:40.799 --> 00:20:49.359 It's just like you know, you can't you we haven't really solved like you know, in terms of like how you do model training of like how do you actually get like continuously get the model to become better and better. 00:20:49.519 --> 00:20:58.559 So like skills are this sort of like matrix-like moment where you know just say I want to fly a helicopter, just put a skill, and and you get it get to fly a helicopter, right, as as a human. 00:20:58.880 --> 00:21:08.240 So I think if you go back to understanding what that means, is it's like you know, you can actually teach the agent to do something very well or not so well, right? 00:21:08.319 --> 00:21:18.799 So like if you actually have someone tempering your skills, you can actually add it to like download certain binaries and execute them as they think they are actually doing that as a prerequisite for you know doing a particular action. 00:21:18.960 --> 00:21:26.000 So we have to be really careful about what are the skills, what is the skill actually telling or trying to induce the agent to actually start taking terms of action. 00:21:26.160 --> 00:21:37.680 So I think of them as sort of like this learning or or context that you're adding into the into the agent, and then it's really up to you to kind of understand, okay, is this really something that you actually want to have the agent doing? 00:21:37.920 --> 00:21:50.640 And if you have something where it's very tight with like the alignment of the skill of like

you know, actually going to follow, and because you can also run scripts uh with some of these skills, you know, as you're kind of loading them, uh that is probably kind of you know like a real supply chain problem there, right? 00:21:51.119 --> 00:21:57.839 You're actually running uh people's code in their you know environments without them even wanting, or without the coding agent doing anything. 00:21:58.000 --> 00:21:59.759 So there's all these particularities about you know the way. 00:22:00.079 --> 00:23:38.480 you all the skills the way you actually like get them to to like change the the way the the agent is is is is operating that actually should should make it a concern for for you yeah cool i i've got two more questions um i'm gonna i'm gonna end i'm I'm gonna be very kind and and tell you all about my last question I'm gonna ask you all um what for people who read read this entire uh uh reference material and say yeah I totally agree with this what should they when they come to work on a Monday morning what's the one thing that they should do straight away so that's my that's gonna be my ending question so you can think about that each and I'll ask each one of you that but what I want to know is we have sneak here we have uh uh keycard we have Docker and actually me from Tesla who who also reviewed the dock um I'd love to know what is the you know you're you're all you're all different companies different points of view what's the one thing that you all disagreed on what's the hard what was the hardest thing to find parity around what do you think what what was it Ellie I mean like uh it's easier to answer what we all agreed on but that's why I asked the other one I got it we disagree on the least I think the thing that like you know I think when in writing the paper the hardest component and it took me like actually about a large amount of time after everyone stopped talking and I was kind of left with the pen to sit back is like how do we take this concept of the software factory which is like one nebulous at best right who who knows what that is and then two how do we break that down into sort of the different tiers and levels of sophistication so that you know the next question you ask is how do I map to where I am today? 00:23:38.880 --> 00:24:11.359 Because I think that the the truth is is there's so much I won't use the word misinformation but there's certainly a lot of like marketing going on around around these concepts like from vendors and from people who are doing it and saying hey I'm merging 90% of my code automatically and it's really hard I think for a lot of people to say well I'm really like we're just like we yeah we got co-pilots and we got some cloud code but like how do I get to the next level and I think at the core of it it was figuring out how we distill I would say it was a disagreement but it's really how do you distill it down in a more simplified stepwise manner. 00:24:11.519 --> 00:24:20.400 Yeah because

in in you know we have we have great people from massive institutions come and contribute to the paper but really if yeah you gotta have a place to start and so I think that was the core of it. 00:24:20.480 --> 00:24:24.400 I think what like most controversial how to I don't think it was even controversial. 00:24:24.480 --> 00:24:26.079 I think it was conversation. 00:24:26.400 --> 00:24:33.039 How do we make the paper I like the way I like the way you're arguing about what was the most we're both we're both Canadian for me. 00:24:33.359 --> 00:25:00.480 Yeah both Canadian right yeah yeah I think lending on terminology and lending on the on the levels to certain degree to make the pay because like yes there is lots of some people that have a lot of sophistications and they have financial reporting goals and they have you know auditability goals and that is great awesome there's a lot of that happening and they need very particular things that that cost lots of money and all of those things. 00:25:00.720 --> 00:25:35.680 On the other hand you have a three-person startup that is starting to build things how do you make a paper that that contains principles that contains uh guidance that is applicable to both yeah and you can start at any level and you can make it useful and benefit from at least understanding to a large degree what are the things you should be concerned about what are the what are the types of things you should be looking for in order to try and get more uh safe and secure in what you're trying to do. 00:25:37.359 --> 00:25:39.440 Cool so we're gonna wrap with that last question. 00:25:39.680 --> 00:25:43.039 So people are reading this they love it where should they start? 00:25:43.119 --> 00:26:00.000 Let's start with you Marshall what where should they start on Monday morning what's the first thing they should do I think uh the way I put it is I mean going back to the kids I think that sometimes these are kids with chainsaws right like playing with chainsaws so I would basically consider like think about what is the chainsaw you're using. 00:26:00.079 --> 00:26:40.000 Like is it for example do you already have like a container uh environment for like you can actually run your agents like if you don't have like cover the basics cover the basics of like you know not exposing secrets in plain tags not like letting your the agent actually like figure out things that are that are not really shouldn't be easy to figure out so I would say cover that get the egress going like find like control points where you actually can start to more safely and responsibly play with uh play with this technology and kind of keep evolving as maturity so that's that's how I would think about it in terms of like where are you I I'm quite far away from you here much but it sounded like you just said be careful with the chainsaws your children are playing with because that was did I hear that correctly okay that's not an analogy I've heard before but I love

it. 00:26:40.160 --> 00:26:40.480 I love it. 00:26:40.799 --> 00:27:19.519 Ian same question I mean I think the question is okay are you even at level one which is like do I have a common setup for people inside my organization to use agents whether in the cloud or locally that they're all sharing and that that's a that's the beginnings of a system right I think that's the first step is what we know about this entire space is it's it bet it doubles down a lot of the best practices of the last generation which is the more consolidation you have around the same workflows the more consolidation you have around the same context sharing around the same core systems the better these things are that's both from a security standpoint but it's also deeply from a usability standpoint the amount of value you can extract. 00:27:19.599 --> 00:27:20.640 And so I would start there. 00:27:20.720 --> 00:27:22.559 And then I would say okay how do are we there? 00:27:22.640 --> 00:27:51.279 And if we're not how do we get there and then how do then then you get to the evolution of like okay let's go from level one to level two oh and eventually we want to have this you know reactive proactive factory that is you know driving the business I would I would just start with like it's almost like a self-reflection and understand what level of maturity you are where you are more mature where you're less I think I think for most people listening to our customers is they've gone out and they've bought seven different coding agents and they have cloud agents, they have local agents there's this buffet in their organization. 00:27:51.440 --> 00:27:56.400 But the truth is in order to make this stuff really hum you kind of have to go through the exercise to start to consolidate. 00:27:56.640 --> 00:28:01.039 Right right I think I might be a little biased in answering the question. 00:28:01.279 --> 00:28:28.400 First totally agreed but I think fundamentally you want to start looking at containment so how do you put agents in some uh boundary where you can control uh that you can start implementing other uh other uh other aspects of the paper on like networking ingress or other components that are important because it is really hard to do so if the agents are not contained. 00:28:29.039 --> 00:29:12.240 Also so as that call to action that you know hopefully after you've heard a lot of this most important thing go to agentbaseline.org take a read of that you know that that uh reference architecture that v1 please take a read and give some feedback give us your you know give us what worked what didn't what was missing let us know because this is going to be an evolving document and it's what it's you know what we need to be able to make it useful uh for the masses um there's also uh an agent baseline slack I believe right so uh how how how should people go to join that agent baseline.org and following agent baseline.org join the slack drop some comments drop some GitHub issues love feedback and think it's open for a reason it's community motion for a reason I don't think we're experts um but this is an opinionated take and we're we'd love to see the feedback. 00:29:12.640 --> 00:29:23.920 Awesome so do check out that uh that document give us that feedback and uh for now uh Ellie Ian and Marcello thank you so so much for all your wisdom all your insight and uh thanks for putting together that paper as well. 00:29:24.480 --> 00:29:25.279 Thank you amazing. 00:29:25.440 --> 00:29:26.880 Hope you enjoyed that thanks for listening. 00:29:27.039 --> 00:29:29.119 Tune in to the next episode soon bye for now. 00:29:29.519 --> 00:29:34.240 The AI Native Dev is brought to you by TESL the package manager for skills and context. 00:29:34.319 --> 00:29:37.359 Your hosts are Guy Pajani and me Simon Maple. 00:29:37.440 --> 00:29:39.119 Our producer is Tom Dowler. 00:29:39.279 --> 00:29:46.160 The AI Native Dev is not just a podcast it's a community and we host monthly meetups at the TEL offices in central London. 00:29:46.319 --> 00:29:51.119 Visit Tesle.io forward slash community to learn more and I hope to see you there.