Dexter Horthy: Why We Stopped Trusting AI to Write the Plan의 전체 녹취록

--> 00:04:22.480 Um, it's very interesting. 00:04:22.639 --> 00:04:31.839 The the biggest change between summer of 2025 and now is actually like the planning metagame. 00:04:32.079 --> 00:04:34.160 Um, I don't think a lot of people realize this. 00:04:34.240 --> 00:04:40.160 A lot of people just write plans because they put plan mode in the thing or they've read something online and they found some skills and hey, we're gonna make a plan. 00:04:40.399 --> 00:05:01.279 Um, and I think depending on where you are in your journey and like the how the models have changed, I in retrospect I realized that like writing a plan file in the summer in like July of 2025, the number one most high-leverage thing that that got you was it was a really reliable way to get a model to work for longer. 00:05:01.680 --> 00:05:01.839 Yeah. 00:05:02.240 --> 00:05:03.920 It was like, okay, we're gonna do everything on this plan. 00:05:04.160 --> 00:05:06.800 And to kind of follow instructions, I think, a little bit at the time, right? 00:05:06.879 --> 00:05:25.920 It was uh well, it didn't always follow instructions, but you had uh you also had things like uh spec kit, for instance, who that was trying to like really micromanage uh the uh the agent and sort of like say, okay, not only is this the plan, but at any time, just come back to me at the end of this step and I will spit out the precise set of additional plans that you're gonna do. 00:05:26.079 --> 00:05:34.000 Uh but I think it got them uh yeah, it got them to sort of kept kept things a little bit on track, or at least more than it was if you didn't plan. 00:05:34.480 --> 00:05:38.000 And it it was a way to like also like make the work a little resumable, right? 00:05:38.160 --> 00:05:52.079 It was it was it it was the first version of like first the compaction of intent of like, hey, I'm gonna take this whole thing that I want to build, which could be specified as like every line of code I want to write, yeah, or you could put it into this file as a couple hundred lines of markdown, it's like here's what we're gonna do. 00:05:52.240 --> 00:06:03.360 And then you could keep that up to date with what was done, what changed, and so you could resume from a clean context window later, which again, also in July of 2025, like the models were not that good at long context. 00:06:03.439 --> 00:06:08.720 I coined this term called the dumb zone or whatever it was, which at that time was around 100,000 tokens. 00:06:08.800 --> 00:06:11.120 If you went over a hundred thousand tokens, you got way worse results. 00:06:11.279 --> 00:06:12.720 Obviously, that number's higher now. 00:06:12.959 --> 00:06:13.199 Yeah. 00:06:13.360 --> 00:06:24.720 Although there is probably still a dumb zone, like there's still a there's still a spot in which like I still like I reuse that term on it, and I feel like it has uh it has an element of like you know, like too much focus is no focus, right? 00:06:24.800 --> 00:06:39.120 You know, when you have sort of too much information in it, then finding the area of attention is still something where if you're intentional about it, by yeah, yourself making the decisions around what's in the context window and what's not, then you can you're more likely to achieve the goal that you're trying to achieve. 00:06:39.600 --> 00:06:40.000 Yeah. 00:06:40.240 --> 00:06:45.839 So anyways, getting back to like our journey is like we had just come away from um Sean Grove had given this really interesting talk. 00:06:45.920 --> 00:06:46.800 I recommend watching it. 00:06:46.879 --> 00:06:52.399 He was a researcher at OpenAI at the time, and he was talking about basically like the spec is the new code. 00:06:53.120 --> 00:07:06.240 Um, and this idea that like the way we think about assembly now is the way we're gonna think about code in the future, and the spec will be the thing that is verifiable and executable and compiles into code. 00:07:06.399 --> 00:07:06.560 Yeah. 00:07:06.879 --> 00:07:16.959 And that like checking in the code, uh, he made this he has this really good analogy of like, hey, you write a bunch of Java code and then you compile the the jar, right? 00:07:17.040 --> 00:07:20.240 And the jar goes into production or whatever, whatever your compiled asset is, right? 00:07:20.399 --> 00:07:22.079 And then you commit the code to the repo. 00:07:22.399 --> 00:07:26.720 It says in the future, like you'll basically just commit the specs. 00:07:26.879 --> 00:07:40.800 Um, but right now, all of the interesting stuff from how people were using coding agents in June of 2025 was basically you send a bunch of prompts and then you get your result out, and then you commit that, and then you push that, and that becomes part of production. 00:07:40.879 --> 00:08:06.399 Yeah, but like that all of the interesting, useful, more high-leverage stuff is actually all the prompts you sent in that conversation that should be recorded somehow, and that like having a long chat with an agent and then throwing away the chat and shipping the code is gonna be the new version of like what if we uh wrote you know a bunch of Java code and then compiled the jar and checked in the compiled jar and threw away the code that we wrote. 00:08:06.720 --> 00:08:06.959 Right. 00:08:07.199 --> 00:08:07.839 Yeah, yeah. 00:08:08.079 --> 00:08:15.920 I think I mean I I very much kind of uh remember that it needs to be sort of held uh frankly, like this conversation is not just sort of past tense, right? 00:08:16.000 --> 00:08:27.120 Like I think today there's a lot to be claimed, and I know this is very much kind of in your world, which is like those are the very artifacts that you're supposed to be reviewing and discussing and preserving on at the uh uh uh at the time. 00:08:27.439 --> 00:08:44.720 But I I think so so so you were like touting, okay, we want to do like uh uh context engineering and kind of capture all of these sort of intents on it, but I guess what was the what was

the advocacy then like at the time when you were guiding people to say, hey, be amazing with sort of agentic development? 00:08:44.799 --> 00:08:47.600 Like what is it that that you should do? 00:08:47.919 --> 00:09:09.279 Uh and I'm and I'm interested because like I I know that you kind of went from like almost it feels like you went from like a context engineering kind of advocate to to almost like uh you know an antagonist to the uh to the group on it, that it's like a little bit more like you really need to be uh uh more more defined and more reviewing, and I've heard you say now that you should review the code for now. 00:09:09.440 --> 00:09:13.120 So I guess what's the what has changed? 00:09:13.519 --> 00:09:22.720 Yeah, I mean this thing happens with models as new models come out and get smarter, you you never know when the singularity is, and the only way to like stay ahead is like try to live in the future. 00:09:22.879 --> 00:09:35.039 And so I said, Okay, this is a thing people are saying might work, and people in the lab, you know, I think there was a thing that was happening last year a lot where it's just like, oh the labs have way more alpha than any of us, so if they say this is a thing you should do, then you should go do it. 00:09:35.200 --> 00:09:40.000 Um and so the OpenAI said, Oh, this is working for us, then uh then we wanted to try it. 00:09:40.080 --> 00:09:48.320 We wanted we wanted to be living in the future, and so we said, Cool, let's try this, let's try like we'll still have pull requests, but like we'll just say, like, cool, if you read the plan, like the code's not as important. 00:09:48.399 --> 00:09:50.320 Um, ton of ton of issues with that. 00:09:50.399 --> 00:09:53.759 Uh, first of all, the plans contained all of the code that was gonna be written. 00:09:53.919 --> 00:09:58.399 Uh, and so you would read this plan of every single every single it was like literally like inline diffs. 00:09:58.480 --> 00:09:58.559 Yeah. 00:09:58.960 --> 00:10:10.480 Read the plan, and then you would go uh you would go write the code, and then if you didn't read the code, you would basically if you wanted to also review the code because there was drift, or you you know, you were larger, we were three-person startups, so we could have this kind of option. 00:10:10.559 --> 00:10:13.519 We were pre-product market fit, it's like, okay, go, go, go, go, go, go. 00:10:13.679 --> 00:10:24.080 Um, but we kind of recommended this to some larger teams, even teams with like 50 developers, then they would say, Yeah, we were hit the plans, and then the person went and did the implementation, and then the implementation would drift, and so they would have to review the code too. 00:10:24.159 --> 00:10:27.519 They weren't ready to let go of reviewing the code, which uh good for them. 00:10:27.679 --> 00:10:33.679 Uh, but it was actually they were actually doing more work, they were reviewing basically the same, almost the same shape of code twice. 00:10:34.000 --> 00:10:52.799 Yeah, depending on the sort of the uh like the detail level of the sort of the spec, but either way, the and combined with like some sort of sense of discomfort because now you're reviewing someone else's code, and so you're kind of uh familiarity and control over it, uh it's oftentimes like harder to apply your opinions uh when editing than it is when you kind of write it in the first place. 00:10:53.200 --> 00:10:53.440 Right. 00:10:53.600 --> 00:11:07.600 Once yeah, once once the code is written, you're kind of committed to a path, and it's like much more work to change it to the point where sometimes it just makes sense to throughout the implementation, like drop a new prompt in or a new or redo your plan and then and then go again. 00:11:08.240 --> 00:11:16.720 Um but uh yeah, we did this experiment, we ran this for like four or five months before we realized like, oh, this is this code base is actually unusable now. 00:11:16.799 --> 00:11:18.240 Like it became slower and slower. 00:11:18.320 --> 00:11:22.320 There was like every single change caused, you know, change over here caused a regression over here. 00:11:22.399 --> 00:11:27.840 I think this is also one of the things we were like, what if we let the model own the system architecture too? 00:11:28.000 --> 00:11:29.440 Like human in the loop reviewing it. 00:11:29.600 --> 00:11:30.639 It was like, hey, how would you solve this? 00:11:30.799 --> 00:12:03.120 Well, well, I'd make a Golang daemon and then I would use a Unix socket, and then I would connect these things, and then when we run the MCP, we ended up with this like system of like six different services where you have like a desktop app that talks to a socket over Rust, and then that that's like a Golang daemon on the back end, which then launches and manages Claude code sessions, which then launch MCP servers to handle the approval system, and then those MCP servers call back to the like front end and like put things in a SQLite database via Unix socket, and then that shows in the U and the user approves something that goes back to the MCP, back to Claude, back to the Golang daemon, and back to the UI. 00:12:03.200 --> 00:12:11.200 It was just like this incredibly complex system that the models developed uh if you if you let them decide how it should be architected. 00:12:11.759 --> 00:12:12.799 How much is that though? 00:12:12.879 --> 00:12:19.120 Uh how much is it that the models don't know how to do it versus you just sort of saying like, hey, figure out an architecture? 00:12:19.200 --> 00:12:21.279 Like how much of that can be prompted? 00:12:21.440 --> 00:12:27.519 And I know we're sort of still in a little bit of like a past tense, so maybe some of this like we need to talk about like a current perspective on it. 00:12:27.919 --> 00:12:36.480 But I think there's a difference between saying, hey, models can't architect and models can't architect yet, or they can't architect if you

don't give them enough guidance. 00:12:39.279 --> 00:13:00.159 Yeah, and I I think I think like fast fast forwarding a little bit, so like we ran this experiment and then we decided no, this is bad, and then we literally rebuilt the whole product from scratch, and the first two weeks of that, my co-founder Kyle was literally in VS Code, typing every not even cursor, like typing every character by hand to plumb out the data architecture for the new version. 00:13:00.399 --> 00:13:01.440 Yeah, exactly. 00:13:01.600 --> 00:13:11.039 Um But I'm really glad like those those patterns have stayed really, really solid, and like especially if you're starting a new project, the decisions you make early on just constantly cascade into the future. 00:13:11.120 --> 00:13:33.840 And we've we've changed some basic things, but the core ideas behind it are still really strong, and we built on like Electric SQL and durable streams and have this kind of unidirectional data flow uh where like clients don't have to like there's no signaling logic from the back end and there's no like signal receiving logic on the front end that's all just handled by the sync engine, uh, which which was a much better way to do it. 00:13:33.919 --> 00:13:41.440 And I think any any piece of software built today should use like a real-time sync database where clients can just like connect and and get updates live. 00:13:41.679 --> 00:13:54.559 Um but uh anyway, so we we basically were like, okay, we need to read the code, we need to understand the architecture, and then also like there's so many slot patterns that get in and that these models will left unattended, will create a system. 00:13:54.720 --> 00:14:12.159 I don't know, there was this thing, I think it was Donald Knuth or one of these like OG like C language guys had this had this uh had this sort of uh principle that was like uh in order to debug code, you need to be like much smarter than you were when you wrote it. 00:14:12.320 --> 00:14:20.799 And so if you write code that is like the smartest code, the most clever thing that you can come up with, you are by definition not smart enough to debug it. 00:14:21.120 --> 00:14:38.559 And so the thing you're building always has to be more simple than than than than your actual peak uh peak intelligence, uh, which I don't know, I think just in the news last week was like Steve Yege, who spent the last year running around every podcast on the internet and saying, like, hey, uh, and I love Steve. 00:14:38.639 --> 00:14:42.080 Steve's a fucking brilliant guy, uh, a huge fan. 00:14:42.240 --> 00:14:52.879 Uh, but I am gonna call him out for going running around the internet and telling everybody, hey, if you're still trying to read the code, you're not gonna make it, and uh like just let the models cook kind of thing. 00:14:53.039 --> 00:15:04.480 Uh and then last week kind of said, Yeah, I used Fable to build this thing, and Fable made a system so complicated that it itself could not debug it, and you really have to keep an eye on these things and like understand the system you're building. 00:15:04.960 --> 00:15:16.080 So, so the sort of the counterargument to this, like I I I I kind of understand and I I relate, and we sort of still have like a Tesla as we sort of build, and we are sort of a team of sort of 20 odd people on the engineering side. 00:15:16.399 --> 00:15:23.279 So, you know, we do need some sort of supervision by the way, like still compliance systems require kind of a name to every sort of code review and all that. 00:15:23.360 --> 00:15:34.399 So we have a lot of like um uh AI-assisted code review, and some changes are allowed to sort of go autonomously, and there's some supervision of complexity, et cetera, et cetera. 00:15:34.559 --> 00:15:37.519 Uh so I wouldn't sort of say, hey, entirely lights off. 00:15:37.679 --> 00:15:42.480 But I think the the counter argument to reviewing the code is is that you're stuck. 00:15:42.639 --> 00:15:49.279 That you know, when you when you uh review the code and you apply uh a change, right? 00:15:49.440 --> 00:16:06.240 Or you sort of you know give more specific instructions or open VS Code and actually kind of handcraft you know some sort of portion of it, then what you're not doing is you are not building a mechanism or a system or an automation that enables the creation of code that is that is to your preferences, right? 00:16:06.320 --> 00:16:15.360 That is, you know, when you talk about those specs, like I I always think of specs as this kind of this slider uh between determinism and adaptability. 00:16:15.519 --> 00:16:41.200 You know, it's sort of like how many degrees of freedom do you want to give the uh the the LLM, the agent, uh, to to build with, like these are the decisions that you have made, and and within those you should be allowed to play, or like where you're enabling the agent to play, and you need to define the sort of the outcomes that you want to see out of it, and you should be comfortable with sort of not caring as long as the outcomes are are achieved. 00:16:41.360 --> 00:16:51.039 Uh and so if you're if you're kind of building towards that, um you would um like to achieve that result, you have to codify your intent. 00:16:51.279 --> 00:16:58.639 And the way that you would do that is not by reviewing the code, but rather by continuously interacting uh with the system. 00:16:59.279 --> 00:17:13.839 There's like a balance to it, like this idea of like plan before you build, like the only reason to do planning or alignment or whatever it is, whether it's you and the model or you and the model and your team, is like because you have this factory, and this is pre-AI, right? 00:17:13.920 --> 00:17:23.839 But there's all these pieces, and I've I've talked about this before, so I'm not gonna go in depth on this, but you have people making stuff, and then someone builds it, and then you have a review thing, uh, and then you have a rollout and you have

monitoring and you have all this stuff. 00:17:24.000 --> 00:17:34.880 But basically, this idea here is like what people learned 30, 40, 50 years ago is that this build step is like can take hours or days, and the review step can also take hours or days. 00:17:35.039 --> 00:17:48.799 And so we as humans before AI, we're like, okay, we're gonna talk about the architecture, we're gonna do the sprint planning thing, we're gonna align with everybody on the team to make sure that this is at least directionally correct, in the hopes that we can decrease the chance of rework and reduce the review time. 00:17:48.880 --> 00:18:01.920 Uh, because if we talked about it ahead of time, and we've all done this, everyone's read a perfect PR where they're like, yeah, we talked about this for an hour, and now this thing is a thousand or two thousand lines of code, but I I it's it's following our patterns and everything is correct, so it's easy to review. 00:18:02.160 --> 00:18:04.720 When people say we are drowning in PRs, we have too many PRs. 00:18:04.880 --> 00:18:12.799 My pushback is like, no, you don't have too many PRs, you have too many bad PRs because you're letting people turn off their brains uh when they're coding. 00:18:13.039 --> 00:18:17.599 Um, and so like the the other side of this though is like you don't want to overplan, right? 00:18:17.759 --> 00:18:22.559 So like I can YOLO a two-sentence prompt and there's a 50% chance I'll have to rework it. 00:18:22.640 --> 00:18:26.640 I when I say rework, I mean like I'm gonna go back and forth with the model a couple more times. 00:18:26.720 --> 00:18:30.400 Please don't YOLO a two-sentence prompt and not read the code and send it to someone else to review. 00:18:30.480 --> 00:18:32.640 I'm assuming you're gonna actually repolish it. 00:18:32.880 --> 00:18:35.759 Um or you could like hand write a detailed spec. 00:18:35.839 --> 00:18:39.759 You could sit there at a meeting and spend a bunch of time and spend five five hours writing a spec. 00:18:40.000 --> 00:18:41.599 And maybe now there's a 10% chance. 00:18:41.680 --> 00:18:45.519 You may still have to change some stuff, but um but like it's it's less. 00:18:45.759 --> 00:18:55.039 Or you could do the extreme of like write all the code by hand, uh, and then there, well, there's a zero percent chance you'd have to rework it because of something the agent got wrong because you wrote it. 00:18:55.279 --> 00:19:10.319 You may still have to rework it that we've had code review for you know 30, 40 years, but um it's it's you're not gonna have to change it because of something the agent because you left the agent some uh some some leeway to uh to to make decisions on some of the smaller details. 00:19:10.720 --> 00:19:13.279 And I I think about this in terms of expected pain. 00:19:13.359 --> 00:19:18.480 Like, what is the chance you'll have to change uh something later and how painful is that? 00:19:18.640 --> 00:19:30.880 Like I never argue with a model about like how what color a button is gonna be, or something like that, because I know that if it's not the color I like, I'll just send one prompt and it will fix it. 00:19:31.039 --> 00:19:35.200 But if like the database schema is wrong, then we're probably gonna have to throw it out and restart over. 00:19:35.279 --> 00:19:37.279 And so I want to get aligned on that first. 00:19:37.440 --> 00:19:37.599 Yeah. 00:19:37.839 --> 00:19:47.839 And so the idea is like you have to build this intuition of like based on the size of the thing and the complexity of the code base and how good the code is today, and how many bad patterns are in there that the model might follow. 00:19:48.000 --> 00:19:50.000 Like, where do you want to land on this curve? 00:19:50.160 --> 00:19:52.240 And that's that's what I think about like today. 00:19:52.400 --> 00:19:54.240 Planning is all about leverage. 00:19:54.400 --> 00:19:56.720 How do you find the sweet spot for the task? 00:19:56.880 --> 00:19:59.440 And you're gonna get wrong sometimes, that's why I say like get a lot of reps. 00:19:59.599 --> 00:20:11.839 But this is this is kind of how I think I think I think it's similar to what you said, but just kind of visualized in terms of like what are plans for um is to is to kind of at every stage of your human layer. 00:20:12.000 --> 00:20:13.599 We have a bunch of different planning workflows. 00:20:13.759 --> 00:20:16.319 We have like just do a design doc and then go ship it. 00:20:16.400 --> 00:20:18.480 We have like design and then the order we're gonna do it. 00:20:18.559 --> 00:20:29.440 We have like a PRD flow that's like three or four stages, but depending on the size of the feature, yeah, you want to have one or more stages where you are just like slicing off 50% of the potential bad outcomes. 00:20:29.599 --> 00:20:34.160 You're narrowing the outcome space in a way that is like a good use of your time. 00:20:34.319 --> 00:20:34.480 Yeah. 00:20:35.119 --> 00:20:37.759 I think um, well, I have a whole bunch of like uh thoughts on it. 00:20:37.839 --> 00:20:45.440 Like I think you know, direction we're sort of aligned on it, like you have to control a little bit the uh or you have to decide what you want to control for and what do you not want to control for on it. 00:20:45.599 --> 00:20:48.000 Um one uh I have a set of questions here. 00:20:48.160 --> 00:20:55.680 One is like how much of that planning uh is around kind of guiding the decisions versus expressing your intent? 00:20:55.839 --> 00:21:00.160 Because I think when you're when you're when you're planning, you say, well, I want the architecture to be like that. 00:21:00.240 --> 00:21:05.039 And implicitly it's well, because I want to make sure that it you know scales cheaply. 00:21:05.200 --> 00:21:13.920 I want to make sure that it is you know resilient to like uh additional kind of uh whatever data fields on it, you know, whatever it is. 00:21:14.000 --> 00:21:15.200 Um but you don't say that.

00:21:15.279 --> 00:21:18.240 You sort of say, I want the architecture to be like that. 00:21:18.400 --> 00:21:27.920 Uh, and so you're you're you're sort of limiting to your own creativity of how you would implement, but you actually never capture your your kind of actual intent of why is it? 00:21:28.000 --> 00:21:32.799 It's one of these, you know, ask why 17 times uh to to get to it. 00:21:32.960 --> 00:21:45.359 So I guess when you're when you're going through that planning, how much are you trying to extract and do you separate the uh what do you want to achieve versus how the implementation should look? 00:21:46.240 --> 00:21:59.759 I mean, I think there's a reason why Matt Pocock's skills and specifically like the grill me skill has uh the most it's in like the top twenty five stars on any GitHub repo of all time, and it's like the top. 00:22:00.319 --> 00:22:10.079 Three stars on GitHub for repos that like mostly just contain skills is because like people are bad at specifying their intent. 00:22:10.319 --> 00:22:28.000 And uh and I think the the a good AI assisting assisted planning process involves like having the model get enough context to understand what's going on and enabling a user, the fastest way to get a good description of your intent is to just like off the cuff say kind of what you want. 00:22:28.160 --> 00:22:42.720 I'll turn on voice mode and I'll ramble for 60 seconds, and the things I know I want, like, oh okay, we're gonna this has to scale to end users, or this cannot use this pass that has to use this, like it has to use Redis and not the Postgres database because I know the speed like anything you know you should specify. 00:22:42.880 --> 00:23:09.119 But the the the onus of like, hey, I need to sit there and just like think for six hours and decide exactly what I want, can it's it's much easier and much faster to be able to take a very rough, like large, unspecified thing, under specified thing, and use AI to you know relentlessly interview the user as as to hey look, here's what you asked for, like here's some alternatives, are you sure you don't want these? 00:23:09.279 --> 00:23:15.599 Or like, hey, you asked for this, but there's a ton of open questions there, like here's what I think they are, and like walking down that decision tree. 00:23:15.839 --> 00:23:16.000 Yeah. 00:23:16.240 --> 00:23:21.920 So I think when you're when you're so I I think we're in agreement over there that you you like it's hard to extract sort of things out of it. 00:23:22.079 --> 00:23:37.200 But when you when you think about like reviewing the code and uh and the likes, I guess what I what I struggle with a bit is this uh okay, like if you extracted the intent correctly and you have defined your kind of verifications correctly, right? 00:23:37.279 --> 00:23:42.960 You you know what you're looking for, which are are are both big ifs, they're both hard to do uh on it. 00:23:43.440 --> 00:23:44.559 Do you still need it? 00:23:44.640 --> 00:23:46.880 Like the models are getting pretty good, you know. 00:23:46.960 --> 00:23:50.880 Let's put aside, you know, that is hard to do, let's put costs aside for a sec, right? 00:23:50.960 --> 00:23:55.519 Like let's assume you can kind of iterate and you can define because you've defined what good looks like on it. 00:23:55.839 --> 00:23:59.519 Like, do do you think like is it uh what's the limitation over here? 00:23:59.599 --> 00:24:10.160 Is it is it actual like models are just not good enough uh yet, in your view, or is it uh an I don't know if you have like a guess as to like when would they be? 00:24:10.319 --> 00:24:28.160 Uh or is it uh or is it more uh or is it something else, you know, that it's like uh that that is sort of that that like that causes you not to sort of say like what you really need to do is like invest in the capturing of the intent and the verifiers, and instead you're saying you should review the code. 00:24:29.839 --> 00:24:40.880 I I there's like this there was this post from the Haskell community like six months ago that was like a sufficiently detailed spec is code. 00:24:41.200 --> 00:24:41.920 It's indistinguishable. 00:24:42.799 --> 00:24:42.880 Yeah. 00:24:43.039 --> 00:24:44.720 But those are like they are implementation specs. 00:24:44.799 --> 00:24:48.960 Like these are they're talking about cases in which I am dictating an algorithm to you, right? 00:24:49.119 --> 00:24:49.599 Yeah. 00:24:49.920 --> 00:25:11.759 And so yeah, if you give I mean the the the reason why I think specs are so powerful and people maybe get a little bit like one-shotted by a good spec with verification is like if you give the model a good way to get feedback on what it's doing, uh, and to like a hail climb against some objective that it itself can measure, whether it's qualitative or quantitative, yeah, um it will move mountains for you. 00:25:11.839 --> 00:25:16.559 And it's a again, just like plans were a great way to get a model to work unattended for longer. 00:25:16.720 --> 00:25:19.599 A good spec and good, we call it like back pressure, right? 00:25:19.680 --> 00:25:24.319 Is how do you give the model the ability to get feedback on what it did without a human having to do that part? 00:25:24.400 --> 00:25:24.640 Yeah. 00:25:24.799 --> 00:25:31.279 Um, you can get a lot more done and you can do more in parallel because that thing's gonna go work for two hours and go kick off another one. 00:25:31.440 --> 00:25:31.920 Right. 00:25:32.240 --> 00:25:35.200 I think the so I think I I I think that I'm entirely aligned with. 00:25:35.279 --> 00:25:49.519 So like I think the difficulty as I interpret what you've sort of said, and sort of maybe like a bit of like uh to my sort of self-view over here as well, which is like in a perfect world, you'd be able to express intent very well, and you'd be able to express uh your verifiers, right? 00:25:49.599 --> 00:25:52.400 Of like how is it that you would verify that it's right

very well. 00:25:52.480 --> 00:25:54.240 And when you do, you don't need to review the code. 00:25:54.319 --> 00:26:00.079 Like if you're if you're able to do that, then you will achieve uh something in which you can allow autonomy, right? 00:26:00.160 --> 00:26:06.720 Like what is the level in which you'll allow like what do you need to see to be able to allow autonomy? 00:26:07.680 --> 00:26:13.920 Yeah, um, I mean I I think it's a great way to kind of like prototype and understand what's possible. 00:26:14.000 --> 00:26:22.160 Like I'm working right now on just a little experiment, and I don't actually think this is like a good there are I I'm working on a like code search harness basically. 00:26:22.319 --> 00:26:29.920 Um and I know like Tejus at Morph LM built a custom model just optimized for code search um called warp prep. 00:26:30.000 --> 00:26:33.839 But what I have been working on is like a code search harness for Jev. 00:26:34.480 --> 00:26:39.119 Um so like can you get Jeb to select the next correct action? 00:26:39.200 --> 00:26:47.920 Like basically using Jeb's classifiers as tool calls of like cool, next I want to read this file, or the final answer of like, hey, yeah, now we have enough context to answer the question or whatever it is. 00:26:48.480 --> 00:26:53.359 And the problem is Jeb only has a 32k or 64k like input context window. 00:26:53.440 --> 00:26:56.720 I forgot, I think 64k is some asterisk on it. 00:26:57.119 --> 00:27:03.920 Um but um that means that you cannot put an entire code base in there, right? 00:27:04.079 --> 00:27:28.240 Uh, or you cannot say here's all the information we have so far, and so you have to get really creative about how do you split things up and how do you merge them back together, and so I'm kind of taking like an evolutionary algorithm sort of approach of like, hey, cool, go do a bunch of subagents, have them try different things, try them hill climb different dimensions of the Pareto frontier for cost and speed and accuracy and 45 different code search evals right now and more coming. 00:27:28.480 --> 00:27:31.440 Um I'm not reading any of that code. 00:27:31.680 --> 00:27:47.359 Um and if it actually works and if seems good, then I will probably like take the like root of it and be like, cool, like explain this algorithm to me, turn that into a spec, and then re-implement that with actual like clean architecture, because there's just slop everywhere right now. 00:27:47.440 --> 00:28:14.640 Like the code is just like doing all this crazy regex matching and splitting, and uh so I I think I think not reading the code in order to prove what's possible and to prototype is is very feasible, but I mean I think everyone says, I don't know, there was this thing that people said in like the Sonnet 3.5 Opus 4 kind of days, which was like, yeah, sure, the code is slot, but I'm not gonna read it because like by the time I need to fix it, that'll be GPT-7's problem. 00:28:14.960 --> 00:28:33.759 And

the challenge is like, folks, we're at GPT-6, and every really good builder I know has mentioned that like like you know, Astra got much better at computer use, it got much better at video game slot, it got much better at Blender, but it didn't actually get that much better at writing code that is maintainable. 00:28:33.920 --> 00:28:40.079 Um, and I think this is because uh to train a model at something, you need RL environments for it. 00:28:40.160 --> 00:28:43.839 Uh and to have a good an RL environment, you need a Oracle. 00:28:44.000 --> 00:28:49.920 You need a thing that can verify whether the product meets the needs of the user. 00:28:50.079 --> 00:28:54.480 And maintainability, you don't find out something is unmaintainable until two months later. 00:28:54.640 --> 00:28:58.319 Uh there's no fast oracle for software maintainability. 00:28:58.480 --> 00:29:04.079 Um, I think is uh I was riffing about this with Addy Osmani, who was at Google for a while, um, and that was kind of his take. 00:29:04.160 --> 00:29:09.200 It's like, yeah, maintainability has no fast oracle, and it doesn't seem like there was this benchmark. 00:29:09.279 --> 00:29:10.640 Have you heard of slop code bench? 00:29:10.799 --> 00:29:22.799 I'll give you the very high, very high level, which is like you give a model a code base, you give it a challenge, you say, hey, go go write this feature, and then you have the code base, and then you say, Cool, now go write this feature, and it has to inherit its code base at every step. 00:29:22.960 --> 00:29:27.759 And it's like, now go write this feature, and like basically what the benchmark shows is one, it's very undersaturated. 00:29:27.839 --> 00:29:31.519 Like, I think GPT-5.5 in May got a 14.8%. 00:29:32.559 --> 00:29:35.599 Uh, and they're getting ready to publish the results for Astra. 00:29:35.680 --> 00:29:42.480 Uh, I think uh I was helping them run it with some inference and stuff, but like um it's not my benchmark, by the way. 00:29:42.559 --> 00:29:44.480 This is like our lab out of University of Wisconsin. 00:29:44.559 --> 00:29:48.400 Uh but we've been like, you know, providing providing runs and doing stuff like this. 00:29:48.480 --> 00:29:52.720 And Astra got in the first run we did was like 16.3%. 00:29:53.759 --> 00:30:00.160 So like the models are getting better in this dimension of like, can they maintain a code base over time? 00:30:00.720 --> 00:30:03.039 But not as fast as everyone thinks. 00:30:03.200 --> 00:30:05.599 Like I said, we said GPT-7 would solve all of this. 00:30:05.680 --> 00:30:17.920 We're GPT-6 and like we are getting we are RLing and hill climbing in other domains, but like the frontier of like model maintaining code base over time does not appear to be getting better that quickly. 00:30:18.240 --> 00:30:21.039 Hey everyone, hope you're enjoying the episode so far. 00:30:21.279 --> 00:30:29.440 Our team is working really hard behind the scenes to bring you the best guests so we can have the most informative conversations about agentic development. 00:30:29.599 --> 00:30:35.039 Whether that's talking about the latest tools, the most efficient workflows, or defining best practices. 00:30:35.200 --> 00:30:38.799 But for whatever reason, many of you have yet to subscribe to the channel. 00:30:39.039 --> 00:30:45.759 If you're enjoying the podcast and want us to continue to bring you the very best content, please do us a favour and hit that subscribe button. 00:30:45.920 --> 00:30:53.279 It really does make a difference and lets us continue to improve the quality of our guests and build an even better product for you. 00:30:53.599 --> 00:30:55.200 Alright, back to the episode. 00:30:55.839 --> 00:30:57.920 I love this conversation, I think about this all the time. 00:30:58.079 --> 00:31:06.319 Uh I feel like the counter-argument to an extent is sort of this notion of sort of of maintenance agents, right? 00:31:06.400 --> 00:31:15.440 And I I think in general, like through the TESL journey, right, we started talking about sort of spec-centric software development, spec-driven software development, and then we moved into this sort of context and skills. 00:31:15.519 --> 00:31:30.400 Uh and to me, the the sort of the premise of that evolution was that uh is maybe the acceptance that when you think about an amazing developer on your team, uh they're not amazing just because they know what you are building, but also because they understand how you are building. 00:31:30.559 --> 00:31:33.039 And really, to me, context is sort of the breadth of it, right? 00:31:33.119 --> 00:31:37.440 I think like you look at skills, for instance, they will have, I think, three primary categories. 00:31:37.599 --> 00:31:43.839 They will have uh uh specs, which are more about like how does your product or another product work on it so they can build it. 00:31:43.920 --> 00:31:58.559 They will have um um uh uh uh workflows around how do you want to sort of perform you know some action on it, and then they will have opinions, they will have policies, best practices, and like choices where it's not necessarily correct or incorrect, but it's just your choice. 00:31:58.720 --> 00:32:01.440 We got in the room and we agreed this is our security policy, right? 00:32:01.519 --> 00:32:03.680 This is our sort of uh uh balancing act. 00:32:04.000 --> 00:32:10.720 Or we say I I don't know, we had a couple of these at Sprout Social my first when my first gig out of college, and it they they called they just had a list of things. 00:32:10.799 --> 00:32:13.519 It was like, do not wrap APIs in other APIs. 00:32:13.920 --> 00:32:14.160 Exactly. 00:32:14.400 --> 00:32:16.400 Like minimize code's traversal depth. 00:32:16.559 --> 00:32:24.319 And uh my boss at the time, this guy Alan, crazy smart dude, he had this thing of like three, he called it three copies press hard. 00:32:24.559 --> 00:32:32.160 Is like at the top of your endpoint, you always have like a okay, log a

thing and then send a metric and then do another thing. 00:32:32.319 --> 00:32:38.000 And every engineer wants to like collapse those into a function, like an abstraction, like send metrics. 00:32:38.160 --> 00:32:48.400 And his take is just like, no, just duplicate the three lines of code everywhere because as soon as you put an abstraction over it, it ends up like leaking a bunch of stuff and it becomes very complex and it gets a bunch of responsibilities. 00:32:48.559 --> 00:32:50.400 Just call the three different methods in line. 00:32:50.559 --> 00:32:53.759 And this is what you do when you go to the car shop or the car mechanic, right? 00:32:54.000 --> 00:32:56.480 You you you they say three copies, press hard. 00:32:56.880 --> 00:33:09.119 You write down, you sign the thing, and then one goes to management and one goes to one goes to processing and one one you keep, uh, and they've just figured this out that it's like, no, we're not gonna take one thing and then like make copies of it or whatever. 00:33:09.200 --> 00:33:13.279 It's just like, yeah, it seems a little weird and archaic, and like, okay, you could be smarter about it. 00:33:13.440 --> 00:33:16.880 Anyways, yeah, these are the kinds of opinions that like are very hard. 00:33:17.039 --> 00:33:20.559 This taste that engineers have over five, ten, twenty years. 00:33:21.039 --> 00:33:21.440 Yeah. 00:33:21.680 --> 00:33:23.920 We haven't figured out how to put that in a model yet. 00:33:24.000 --> 00:33:26.400 We just haven't figured out how to give models that that taste. 00:33:26.799 --> 00:33:34.559 So I guess uh I guess what I'm asking is okay, so you can't put that in a model, and and I don't know if they ever will, because like models almost by definition, they are generic intelligence. 00:33:34.640 --> 00:33:36.880 The one you use and the one I use are the same. 00:33:36.960 --> 00:33:41.759 Uh and so even if they are or less smart, they're they're they're the same opinion. 00:33:41.839 --> 00:33:48.799 You know, they will uh they will run, you know, design is probably the easiest examples of that, you know, like like you might like dark mode or you might hate dark mode, right? 00:33:48.880 --> 00:33:50.240 Like, you know, you might sort of choose that. 00:33:50.400 --> 00:33:54.640 And so uh but but you can't put it into the model, but you can put it into context. 00:33:54.799 --> 00:33:58.720 And so you can define, and if you've captured that, and that to me is another planning aspect, right? 00:33:58.799 --> 00:34:13.199 Like you've planned how you want to build, and now you can run a whole bunch of uh maintenance agents that would run on that code base that would review code that was created by an implementer agent uh who took in more like uh feature requirements, right? 00:34:13.280 --> 00:34:18.559 Like sort of requests on it and will ensure, and you codify your taste, your preferences. 00:34:18.719 --> 00:34:31.360 I guess why isn't that the way to achieve uh a non or

a low slop code uh and kind of invest in that path versus reviewing the code? 00:34:31.920 --> 00:34:49.760 Yeah, I mean I I could see I could see a world where I mean the the way this usually starts is people just add a hundred end up adding one after another after another of like bullet points into their cloud empty or their agents empty, of like always do this and never do this, and then you end up with a bunch of instructions and like most of them don't get followed because it's too much context. 00:34:49.840 --> 00:35:04.000 And so, like, yeah, you could probably I mean, so we do this thing internally at Human Layer where we have uh deterministic detectors for some of them deterministic, some of them are non-deterministic, but very simple of like we don't like this pattern, the desired end state looks like this. 00:35:04.159 --> 00:35:05.920 Yeah, uh, there's a bunch of code that looks like that. 00:35:06.000 --> 00:35:13.199 Yeah, uh whether it's like React Doctor deterministically pulling warnings out or ox aux lint pulling warnings out. 00:35:13.360 --> 00:35:17.760 It's like every night we have four or five different um people call them agents. 00:35:17.840 --> 00:35:21.119 It's the same harness with a different user message, basically. 00:35:21.199 --> 00:35:21.440 Right. 00:35:21.599 --> 00:35:23.599 Five agents if you if you if you must. 00:35:24.000 --> 00:35:27.599 Five sort of uh skill-powered workflows or sort of uh prompt power workflows. 00:35:28.320 --> 00:35:28.800 Exactly. 00:35:28.960 --> 00:35:37.360 And they run on crons and GitHub actions, and every morning we wake up to five pull requests that make our code base a little bit better along some dimension. 00:35:37.519 --> 00:35:37.840 Yeah. 00:35:38.079 --> 00:35:44.559 Um I think the challenge is like you have to know exactly like you said, you have to know exactly what you want. 00:35:44.639 --> 00:35:47.119 You have to codify all of your opinions somewhere. 00:35:47.280 --> 00:35:48.960 You probably can't be in your agent's MD. 00:35:49.039 --> 00:35:54.880 Maybe you can put them in a skill or you can break them up into a set of skills, but the the more opinions you have, the more clever you have to be. 00:35:54.960 --> 00:35:56.079 This is context engineering. 00:35:56.159 --> 00:36:04.320 This is always the thing, is like the more clever you have to be about like splitting them up and parallelizing and kind of like orchestrating multiple different things to go do this. 00:36:04.559 --> 00:36:04.719 Yeah. 00:36:05.119 --> 00:36:10.079 So I get the point around sort of the need to uh review the codes today because of the limitations. 00:36:10.159 --> 00:36:12.559 And as I mentioned in TESL, we still sort of review the code. 00:36:12.639 --> 00:36:21.119 Uh but to an extent, the sort of the bet that we're making in our own internal software factory uh is that really it's all about extracting correctness out of observations. 00:36:21.199 --> 00:36:35.280 Uh and so what we

do is we take today maybe some amount of sort of a risk or burden uh that really forces the process of not writing uh the code and sort of reviewing the code, and then you interact with the systems. 00:36:35.360 --> 00:36:59.199 There's exceptions to that, there's asterisks next to it, but by default, it has to be kind of a feedback that you provide about what is wrong, you know, as part of pull requests, as part of uh systems, uh, and sort of these sort of continuously uh improving skill-powered workflows, so like these feedback loops that uh that really uh help feedback into the system. 00:36:59.360 --> 00:37:19.760 And I think there's a good case to be made that today we are uh maybe making more effort in sort of uh in sort of investing and honing those skills uh on the on the sort of the the bet, which is always TBD, uh that it will uh it will uh set us up better, and we're sort of seeing some compounding speed out of that. 00:37:20.000 --> 00:37:22.480 So I I don't know how uh how you think about that. 00:37:22.719 --> 00:37:26.559 Uh and and how does that align or conflict with like reviewing the code? 00:37:27.199 --> 00:37:29.840 Yeah, I mean we are very bullish on this compounding thing. 00:37:29.920 --> 00:37:40.079 I I think probably like the anti-pattern I see in a lot of orgs, and most of them are probably larger than yours or mine, is is someone gets excited and they say, I'm gonna build the software factory. 00:37:40.159 --> 00:37:46.880 And they kind of throw everything out and they say, Cool, let me design this whole thing from scratch, and it's gonna have this compounding thing, and we will never have to read the code and whatever it is. 00:37:46.960 --> 00:37:59.840 And like I think that's uh like any piece of software, the big design up front is like is always gonna lose to the like iterate fast and get 1% better every single day, or what even 1% better every single week. 00:38:00.000 --> 00:38:11.840 So I think like the idea of having having a process thing is is like you cannot afford to put slop code in your code base because every every time you let bad code in, you are making the entire system worse. 00:38:12.480 --> 00:38:26.639 People talk about software yeah, people talk about software factories, but the the the there is a big difference between a traditional factory and a software factory, and that like in a factory, if a car comes out of the end of the line and it's bad, like that's cool. 00:38:26.719 --> 00:38:29.519 Whoever was gonna buy that car or whatever is gonna is gonna deal with it. 00:38:30.079 --> 00:38:40.639 But like if you ship bad code in your software factory, it degrades every future piece of work that goes through that factory because there's more bad patterns for the model to see. 00:38:40.800 --> 00:38:51.599 So, like you do want to continually be making progress towards the you have to want to have some ramp where you can make the factory better every single day or every single week. 00:38:51.760

--> 00:39:00.800 Um, but it's like the the number one priority is like stop the stop the bleeding or or stop the you know, close the windows on the house before you turn on the air conditioning. 00:39:02.159 --> 00:39:02.639 Yeah. 00:39:03.360 --> 00:39:04.159 Exactly. 00:39:04.320 --> 00:39:18.960 Um but no, this idea of like cool, you know, we have humans in the loop whose job it is to make sure that the code is good and that nothing goes into prod or into the code base that is like going to make the quality worse and going to make future quality worse because the agent will see that and think it's okay. 00:39:19.119 --> 00:39:31.039 Um, but a byproduct, the kind of like exhaust of that process is you get a ton of like real examples that you can use to feed improving the factory and make it a little bit better. 00:39:31.199 --> 00:39:36.239 Like anytime a human has to put a comment on an agent written PR and say, hey, this is wrong, we don't do it this way. 00:39:36.559 --> 00:39:37.840 Agent goes and fix it, it's fine. 00:39:38.320 --> 00:39:40.079 You need to be like siphoning that off. 00:39:40.239 --> 00:39:50.880 Every time someone sends, even in a prompt, you should be reviewing the session traces, maybe not reading them verbatim, but analyzing the session traces of everyone on your team and saying, where are people getting frustrated? 00:39:51.039 --> 00:40:04.480 Where are they telling the model they did something wrong, and then figuring out how to improve the entire system, whether it's the base skills, whether it's the code review skill, whether whatever it is, um, so that you know, if you're not getting better, then you're getting worse. 00:40:04.639 --> 00:40:13.840 And so, yes, I 100% agree on this idea of compounding engineering or or or whatever you want to call it, of you you do always want to be working on the thing that builds the thing. 00:40:14.000 --> 00:40:22.880 And if you do a good job of this, then you're sliding scale of like you're spending less and less time actually reviewing individual code and more and more time doing things that are going to improve. 00:40:23.119 --> 00:40:27.199 You're you're basically shifting from like caring about your position to caring about your velocity. 00:40:27.519 --> 00:40:30.320 Yeah, yeah, from kind of building the car to building the factory, right? 00:40:30.400 --> 00:40:31.760 You're sort of uh producing that. 00:40:32.079 --> 00:40:34.480 And how uh so so that's a really good point, though. 00:40:34.639 --> 00:40:50.559 I think one takeaway from what you're saying is like yes, you should improve the sort of the context eventually that guides the factory and their production, but you should also remember that uh one of the primary, if not the primary context that eventually guides the factory is the actual existing code base. 00:40:50.719 --> 00:41:04.719 And so if that code base is full of slop, then uh then the reality is that that will be a pattern because the agents somewhat

legitimately default to mimicking the uh the uh the patterns that they see that they see within the code. 00:41:04.800 --> 00:41:18.960 Uh and so if you if you let that go kind of sideways too much, and it's it's sort of a counter a counter view to to uh to saying that tech debt is deflationary and that uh you can accumulate debt today. 00:41:19.119 --> 00:41:28.079 I mean that would maybe be true if literally every pull request uh that you're creating, you don't take in any code, you just sort of produce it uh from scratch. 00:41:28.159 --> 00:41:34.000 Uh but that is not realistic because you don't actually have that uh so really good point. 00:41:34.159 --> 00:41:41.360 Maybe so, going a little bit more future looking uh a little bit and and thinking about what are the artifacts that might stick around. 00:41:41.440 --> 00:41:45.920 So, like how long do you think code will still be the source of truth? 00:41:46.079 --> 00:41:54.239 Like we're talking about rebuilding, replacing, editing increasingly, but we still end up kind of capturing the actual decisions uh in the code. 00:41:54.480 --> 00:41:59.119 Do you foresee that two years from now that is that is still the case? 00:41:59.280 --> 00:42:07.360 Or do you think new artifacts emerge that are the primary and the code is more like the bytecode that you've produced? 00:42:07.679 --> 00:42:08.239 Right. 00:42:08.480 --> 00:42:10.880 Um yeah, I don't I don't want to predict the timeline. 00:42:10.960 --> 00:42:17.039 I think a lot of people's predictions have been either way too way too conservative or way too liberal. 00:42:17.199 --> 00:42:34.159 And um, you know, our job is work with a lot of work with you know, Fortune 500 companies, publicly traded companies, you know, large private companies, and like the lot the the the the reason why they love working with us is because we're like cool, like do not mortgage your code base for the future. 00:42:34.320 --> 00:42:41.119 Like, yes, like there may be like a 1% chance that you miss out on you know some crazy, crazy upside. 00:42:41.280 --> 00:42:49.039 Um, but we are going to let you move as fast as possible without uh without giving up control or giving up taste. 00:42:49.280 --> 00:43:01.199 I was at a dinner in San Francisco probably six weeks ago, and um, you know, we were talking about software factories and loops engineering or whatever we were calling it back then, and this question was like, cool, raise your hand if AI is writing 100% of your code. 00:43:01.360 --> 00:43:02.719 Like everybody, hand up. 00:43:02.960 --> 00:43:06.159 Raise your hand if you're still reading the code, and it was like 50% of the people. 00:43:06.320 --> 00:43:15.760 The other 50% have been like, no, I made 70 Linter rules custom by hand, and I I caught every single anti-pattern, or whatever whatever, whatever your approach is that gives you the confidence to ship it. 00:43:15.920 --> 00:43:19.440 Uh, but then the question came is like, how long until we

don't review the code? 00:43:19.599 --> 00:43:23.840 And I was like, uh maybe one, two years, it's possible that that could be the case. 00:43:24.159 --> 00:43:26.159 Um, and then there was a really interesting question. 00:43:26.239 --> 00:43:31.199 I was like, Do you ever think there will be a time when you don't review anything at all? 00:43:32.400 --> 00:43:48.239 And uh one of the founders there, um who is a veteran of a lot of this stuff, kind of said one of the most interesting things that I think about almost every day now, which is like there will probably always be alpha in reviewing something. 00:43:49.039 --> 00:43:50.079 We don't know what it is. 00:43:50.159 --> 00:43:51.440 Uh I can't say what it's gonna be. 00:43:51.519 --> 00:43:53.360 It's probably gonna change over the next two years. 00:43:53.599 --> 00:44:06.559 But if you don't review anything, then you're just letting the AI cook and then you're gonna get the Same product, the same email, the same company that everybody else gets. 00:44:06.800 --> 00:44:21.039 And so there has there will always be something if you want to rise above the median and not just be I think I think like the the generic like token farm of just like hey look yeah we spent tokens and we create value is like very undifferentiated. 00:44:21.119 --> 00:44:28.800 So I think I think there will always be a place for smart, creative, thoughtful, caring humans to be reviewing something. 00:44:28.960 --> 00:44:31.280 For now, I think it's still the code, but eventually it might be something else. 00:44:31.360 --> 00:44:32.960 I don't know when or what it's gonna look like. 00:44:33.039 --> 00:44:38.159 But there will always be something to review, and there will always be a thing that you can add to the add to the picture. 00:44:38.400 --> 00:44:38.639 Yeah. 00:44:38.800 --> 00:44:43.360 Yeah, I think we're sort of seeing a lot of that with the with like a claude presentations. 00:44:43.519 --> 00:44:59.679 You know, for a while, like uh, you know, everybody was amazed by how how how great it was that Claude like at the breakthrough of like it was creating these good looking presentations, and now every single presentation looks like a claude presentation, and you uh you just sort of yearn for for something that looks a little bit different. 00:44:59.920 --> 00:45:04.400 Uh but how um uh so so I I agree with that. 00:45:04.480 --> 00:45:08.159 I I think uh I think you have to differentiate, you have to capture those. 00:45:08.320 --> 00:45:16.960 I I do muse though about uh a better way to capture preferences, and I uh I think a lot about sort of outsourcing, right? 00:45:17.039 --> 00:45:38.559 You have a lot of like great companies that have sort of outsourced their whether it's their sort of uh uh sort of software stacks, or even more so like a lot of manufacturing companies, that they don't actually manufacture everything, they design the thing that will be manufactured, someone else is producing it, uh,

and so they they really kind of define the outcomes, but they work with these entities. 00:45:38.880 --> 00:45:55.920 Um I guess does that analogy kind of resonate with you, which is like create artifacts that document preferences, opinions, taste, uh and and for the most part outsource implementation to the agents? 00:45:56.880 --> 00:45:57.679 Yeah, so two things. 00:45:57.760 --> 00:46:02.239 Yeah, I think I think um absolutely like people say, oh, the the new moat is taste. 00:46:02.320 --> 00:46:24.880 I think that's really blurry, but like I do think it's safe to say like the moat you can have is understanding the problem better than anyone else, like knowing what features to ask the factory for, understanding your customers better than anyone else is incredibly valuable still, and that's again that's that's where you're you you can context engineer the agent in a way that none of your competitors can if you understand the problem better. 00:46:25.119 --> 00:46:26.400 I am ambivalent about taste. 00:46:26.480 --> 00:46:31.360 Like part of me feels that taste is just a preference that you haven't bothered writing down. 00:46:31.519 --> 00:46:43.199 Uh, you know, it's just uh like a I'll know it when I see it uh type thing is is um uh is something that just okay, like get the agent to see it often enough uh and it will capture it. 00:46:43.280 --> 00:46:58.880 Uh it's still it still is a differentiator, like it is still the case that if your written down taste is better than my written down taste uh for some purpose, uh and so for those purposes your product or your output will be better than mine. 00:46:58.960 --> 00:47:01.360 Uh and so there's still there's still value in it. 00:47:01.440 --> 00:47:06.159 I still think taste should be codified, and I think still it is I still think that it is the differentiator. 00:47:06.320 --> 00:47:11.360 Uh but it is uh it's not this elusive human versus AI difference. 00:47:11.440 --> 00:47:14.719 It's just a it's just a choice, uh choice piece. 00:47:14.880 --> 00:47:17.519 Um Well, what what about you, man? 00:47:17.599 --> 00:47:23.679 Like what do what do you think the future of reviewing things or building or factories? 00:47:23.760 --> 00:47:27.679 Like what what what spicy takes do you have or or mild takes? 00:47:27.760 --> 00:47:28.159 I don't know. 00:47:28.320 --> 00:47:50.000 Like I think there's there's there's the market for spicy takes is way too frothy right now, and I actually love a more middle of the road of like, look, yes, I think this is directionally correct, but actually like it's not as crazy as everyone says it is because everyone is incentivized to you know engagement farm on the internet, and you can actually you get paid money if you get people excited about something, regardless of whether it's true or not. 00:47:50.159 --> 00:47:53.920 But what are your spicy or or otherwise takes on like where this is going? 00:47:54.239 --> 00:47:59.119 I I've got a fairly kind of uh uh

skilled or context-centric view of the of the world. 00:47:59.280 --> 00:48:05.360 Like I I think the fact that like I think software engineering is a creative uh profession. 00:48:05.519 --> 00:48:10.480 Like you are creating software, uh, and the fact that it has engineering in the name is a mistake of history. 00:48:10.639 --> 00:48:16.079 Like it just happens to be that you had to be an engineer, you have to be an engineer to be able to create those. 00:48:16.239 --> 00:48:23.599 Uh and really what we want is to be able to capture taste and preferences uh and and and balances. 00:48:23.679 --> 00:48:29.599 Uh and I think those will be the kind of units of code, if you will, uh that you are that you are building. 00:48:29.679 --> 00:48:39.760 I thought this is my definition of what uh secure is, of what good design is, of what, you know, even the specifics of subscreen, like you can still get granular or not in terms of its preference. 00:48:39.920 --> 00:48:52.960 Uh and even some elements of like how controlling you are, whatever, whether you're Apple and like you want to design everything down to the sort of the last bit uh and and gram, or whether you're sort of fine with some more sort of generic OEM component in there. 00:48:53.039 --> 00:49:02.800 Um and so those are your choices, and those units will still be developed in the sense that you still need to know that uh uh which ones you are using and which ones depends on what. 00:49:02.960 --> 00:49:14.480 And you still need evals and tests to be able to know that when you modify them, they do not regress, and you still need to observe them and sort of see how they work and de-duplicate them and align them when you have multiple team members creating them. 00:49:14.559 --> 00:49:24.960 And so, like I I guess my belief is that software development would revolve around skills and that the role of code will be will diminish uh over time, that it has to be sort of duplicate. 00:49:25.039 --> 00:49:32.960 I don't know when it goes away, it's more about uh we've referred to at the beginning of uh TESL, we've had the sort of the notion of a spec and a shadow spec. 00:49:33.039 --> 00:49:38.159 You know, sort of the spec being the thing that you create and the sort of shadow spec being the capturing of the LLM's decisions. 00:49:38.400 --> 00:49:40.800 What is the difference between a shadow spec and code? 00:49:40.960 --> 00:49:41.760 I I don't know. 00:49:41.840 --> 00:49:44.480 It's like degrees of freedom uh for it. 00:49:44.960 --> 00:49:47.199 Um and and today you you already do that, right? 00:49:47.280 --> 00:49:55.519 You already use you know Java and you're delegating garbage collection, and it happens in different ways on different systems, and you don't care because you trust that it is good enough. 00:49:55.679 --> 00:50:00.000 Uh and so uh so I think it's sort of going into that world. 00:50:00.159 --> 00:50:05.199 And the the conversation we have right now is really about what is

the journey uh towards it, right? 00:50:05.360 --> 00:50:15.199 You know, what uh and what should you build today so you're not you know left stranded for tomorrow, but that still actually gets you somewhere today, that balance has been tough. 00:50:15.360 --> 00:50:16.239 Uh yeah. 00:50:16.480 --> 00:50:43.440 And so I I won't I won't say software engineer, or maybe I'll just say builder, but like as a builder, as a person who wants to create things of value, I think right now your your two biggest bottlenecks are how do I let the model pull as as efficiently and easily and like mentally like pleasant for me, the human, how do I enable the model to brain drain me as quickly as possible on all the things that are my tastes and preferences and things. 00:50:43.679 --> 00:50:45.599 And then I think I think we're in alignment there. 00:50:45.679 --> 00:50:56.800 And then if you're if you're still reading the code, there's like a a corollary of what is the fastest, most efficient, most visually effective way for a model to tell me what it has done. 00:50:57.039 --> 00:50:59.440 Like, yeah, maybe I might not read every line of code. 00:50:59.599 --> 00:51:18.559 Um, but uh between if I'm reading every line of code now, and someday I'm reading a spec, like between there, there will be, you know, laddering up of like, cool, don't tell me the code, but give me things, give me something that is, you know, 50% faster or 80% faster than reading every line of code that gives me almost as much confidence or almost as much understanding. 00:51:18.719 --> 00:51:36.719 How can you what are the like patterns where we can leverage how do humans think and understand things to both get that into the model's context as quickly as possible, and then let the model context engineer the human of like what's the fastest way that I can explain to this person what has been done so they can be competent that it works? 00:51:36.960 --> 00:51:39.119 Yeah, yeah, no, that's uh that's well said. 00:51:39.280 --> 00:51:42.320 So maybe one last question, a bit along those veins. 00:51:42.480 --> 00:51:47.280 Um I think the the advice you give today is different than the advice you gave 12 months ago, right? 00:51:47.360 --> 00:51:50.239 Sort of seeing the the evolution of the surrounding. 00:51:50.480 --> 00:51:55.280 Um what's your bet on what aspects of your advice will change 12 months from now? 00:51:55.920 --> 00:51:57.679 Yeah, uh that's a great question. 00:51:57.840 --> 00:52:03.199 And yeah, there are some things that we got right almost almost 18 months ago that I have not changed, right? 00:52:03.280 --> 00:52:05.920 Like the context engineering thing I think will always be true. 00:52:06.079 --> 00:52:06.719 Yeah, first principle. 00:52:06.880 --> 00:52:19.119 As long as we're using as long as we're using transformer-based LLMs, that's just like the math of how they work, and you will get better quality if you use less context, so you'll always be able to like engineer

things to get engineer things to get better results. 00:52:19.280 --> 00:52:22.239 Um what will change in the next 12 months? 00:52:22.400 --> 00:52:28.000 I think I spent a lot of time thinking about infrastructure. 00:52:28.239 --> 00:52:53.039 Um I think the entire software factory, this isn't gonna be like AI advice, but I think the like entire software factory stack will decompose into good interfaces and open components, kind of like what we saw happen in the Kubernetes world in the last like you know, 10 the last decade of like basically like the more open and portable you can make these ideas, the more likely they are to get broad adoption. 00:52:53.199 --> 00:53:04.719 Uh, and you need a certain amount of momentum to actually become the default solution, just like what happened to VMware, just like what happened with AWS, what happened with Kubernetes. 00:53:04.880 --> 00:53:11.760 Um and so I I think a lot of these like walled garden, like you can build a software factory entirely on Claude. 00:53:11.840 --> 00:53:22.239 You can use Cloud Code as the harness, you can use Claude Automations as your orchestrator, you can use Claude, like a managed code sandboxes, manage agent, whatever they call it, as your runtime environment. 00:53:22.400 --> 00:53:24.800 Uh, and you can kind of do this whole verticalized stack. 00:53:24.880 --> 00:53:39.599 And I just I I think that the advice I will keep giving people today is like I think DevTools and DevTools stacks are probably gonna stay or become even more open, open source, open protocols, whatever it is. 00:53:39.920 --> 00:53:45.760 Um and so I would I would encourage people to prepare for that. 00:53:45.920 --> 00:53:48.320 And uh you can build anything now. 00:53:48.639 --> 00:53:53.840 Uh don't uh don't totally know about on the code, especially if you're building critical systems. 00:53:54.239 --> 00:53:56.880 Amen to that in terms of the sort of the open composable systems. 00:53:56.960 --> 00:54:14.480 I had uh um early on on the podcast, we had uh Matt from uh Netlify, the founder of Netlify, and he uh and he was kind of making this passionate case which I relate to, which is that the web like had elements of it that were sort of closed, uh eventually the sort of the open web won, and I think we're all better off for it. 00:54:14.559 --> 00:54:34.800 And I think similarly in the kind of the world of agents, the closed ecosystems are very tempting uh because they they naturally align and all the different pieces kind of come together on it, but sort of this belief that eventually open and composable leads to greater innovation uh and greater flexibility for organizations, I think this composability will uh will win over. 00:54:35.679 --> 00:54:38.480 The ordering that's uh that's a whole different bully. 00:54:39.360 --> 00:54:56.559 Yeah, what what's what which chips are gonna fall in what order or dominoes, I suppose, and uh yeah, how how much of how much of a lead do these sort of frontier, the factories, the devons, the open AI, the anthropics of the world, how much of a lead do they actually have? 00:54:57.119 --> 00:55:09.679 Uh and how how it's like for them, it's like they are they're going to war every single day because I I think that open is catching up faster than anyone is ready to notice yet. 00:55:10.880 --> 00:55:13.280 Dex, thanks a lot for the great conversation here. 00:55:13.440 --> 00:55:14.159 Super, super fun. 00:55:14.239 --> 00:55:15.679 And you know, deep insights. 00:55:15.840 --> 00:55:20.880 Also, thanks in general for like sharing the the whole journey here and the learnings and the practices along the way on it. 00:55:20.960 --> 00:55:26.480 I think we we learn as a community, and uh you're definitely sort of a chief amongst the educators over there, and that uh is much appreciated. 00:55:26.880 --> 00:55:27.840 Oh, and thank you, man. 00:55:28.079 --> 00:55:40.239 I'm a huge fan of you and the team, and you guys were super early to this game, and you have, I mean, Patrick Dubois has like been a fanboy of his since he invented DevOps days in like 2009 or whatever. 00:55:40.320 --> 00:55:44.079 So you guys clearly have a really good squad, and like I'm excited to see what you guys are cooking. 00:55:44.239 --> 00:55:48.159 Thanks, and thanks everyone for tuning in, and I hope you join us for the next one. 00:55:48.960 --> 00:55:53.760 The AI Native Dev is brought to you by Tesla, the package manager for skills and context. 00:55:53.840 --> 00:55:56.880 Your hosts are Guy Pajani and me, Simon Maple. 00:55:56.960 --> 00:55:58.639 Our producer is Tom Dowler. 00:55:58.800 --> 00:56:02.079 The AI Native Dev is not just a podcast, it's a community. 00:56:02.159 --> 00:56:05.679 And we host monthly meetups at the Tesla offices in central London. 00:56:05.760 --> 00:56:08.480 Visit Tesla.io forward slash community. 00:56:08.559 --> 00:56:10.880 To learn more, I hope to see you there.