Trascrizione completa di What Do Humans Do Now? Rethinking Software Engineering in the Age of AI Agents

There's some exceptions. 63 00:04:37.955 --> 00:04:43.875 But the task size that I can give it, like that's what's changed. So I can have 64 00:04:44.195 --> 00:04:46.595 12 tasks going now with Fable 65 00:04:47.380 --> 00:04:50.900 and they're each taking multiple hours, sometimes multiple days. 66 00:04:51.300 --> 00:04:53.700 And that's what accelerates 67 00:04:54.020 --> 00:04:58.340 the pace of code generation now is being able to parallelize that 68 00:04:59.060 --> 00:05:03.445 with less oversight, Not zero oversight, but with less oversight. 69 00:05:03.604 --> 00:05:13.044 And so that's what's just completely broken, that old paradigm of we're just going to keep using these human oriented tools and human oriented processes. 70 00:05:13.685 --> 00:05:16.085 And so what comes after that? What comes 71 00:05:16.349 --> 00:05:25.150 now that code is 10 times faster than it was? How do we fit the rest of the software development life cycle around that new 72 00:05:25.550 --> 00:05:26.509 reality? 73 00:05:26.590 --> 00:05:27.229 And 74 00:05:27.870 --> 00:05:31.710 we can kick it off or we can set more stage if you want, but 75 00:05:32.405 --> 00:05:45.044 one of the first things to go, I believe, is the concept of humans spending time reading through diffs and doing code review the way we did, you know, three years ago, even one year ago, 76 00:05:45.604 --> 00:05:48.720 because that's just not a good use of time. And 77 00:05:48.880 --> 00:05:51.120 I can flesh out my 78 00:05:51.440 --> 00:05:54.720 thesis a little bit more, or you can jump in first. 79 00:05:55.120 --> 00:05:56.240 Yeah. So 80 00:05:56.640 --> 00:05:59.760 just as a quick reference for listeners, 81 00:05:59.840 --> 00:06:07.025 you wrote this blog post about the fact that it's time to stop performing code review. I'll add a link in the show notes. And I think that 82 00:06:07.425 --> 00:06:12.465 one of the other interesting elements of this point in time is that, 83 00:06:12.865 --> 00:06:15.745 particularly for people who have been in the industry for 84 00:06:16.120 --> 00:06:17.800 more than two or three years, 85 00:06:18.360 --> 00:06:29.479 they have developed a certain amount of their identity and ego around the code that they write and the way that they approach code and their various competing camps of what constitutes best practice or idiomatic 86 00:06:29.565 --> 00:06:33.405 practices for different languages or frameworks or ecosystems. 87 00:06:34.205 --> 00:06:34.685 And 88 00:06:35.085 --> 00:06:40.925 so one of the points of contention that I've seen in my own experience is people 89 00:06:41.165 --> 00:07:10.595 saying, well, the code that the AI is writing isn't the way that I would have written it, so therefore it's not good. Or it's ignoring this idiom that I have spent a lot of time perfecting where maybe instead of using a bunch of if then blocks, I put it into a dictionary so that it's more of a direct access, or maybe I prefer a more functional style versus imperative style. So there's a lot of these details that, at the end of the day, really don't matter except for the case where maybe they have a performance impact. 90 00:07:11.074 --> 00:07:11.715 And 91 00:07:12.180 --> 00:07:14.660 so that, I think, is another piece where 92 00:07:15.700 --> 00:07:18.260 the human element of engineering 93 00:07:18.340 --> 00:07:19.140 is 94 00:07:19.940 --> 00:07:25.540 coming under fire by the AI because you have to just let go of that fact of, okay. Well, 95 00:07:26.665 --> 00:07:42.185 maybe I didn't even read the code. As long as the code does what I want it to do, that's the thing I care about the most. And, you know, there's the trope that nobody's paying for the software. They're paying for what the software does. And so why do we as engineers spend so much of our time bike shedding or yak shaving about 96 00:07:42.560 --> 00:07:47.680 the way that you write your for loop versus does it do the thing that the customer actually wants? 97 00:07:48.800 --> 00:07:50.560 I think that's 98 00:07:50.560 --> 00:07:57.759 an excellent point and it highlights one of the gaps or the differences in approaches that I've seen 99 00:07:58.035 --> 00:08:02.835 in the talented engineers that I've worked with when it comes to AI, because 100 00:08:03.235 --> 00:08:13.235 I think it's much easier to come to terms with the things that you're talking about for people who have managed or led a team 101 00:08:13.500 --> 00:08:15.260 of human engineers 102 00:08:15.340 --> 00:08:16.780 because it's, 103 00:08:17.660 --> 00:08:28.615 to some degree, like, there's differences for sure. And I'm definitely not saying that managing a team of AI agents is the same as managing a human team because it's not. But it is similar in 104 00:08:28.855 --> 00:08:29.975 this one respect, 105 00:08:30.455 --> 00:08:36.455 which is that if you have a team of human engineers and your code reviews take the form of saying, 106 00:08:36.775 --> 00:08:44.470 I disagree with how you wrote this switch statement instead of putting it in a dictionary and matching on that. Like, 107 00:08:44.790 --> 00:08:47.990 that's not productive use of code review time. 108 00:08:48.310 --> 00:08:48.870 And 109 00:08:51.029 --> 00:08:52.550 it just isn't. And so 110 00:08:53.525 --> 00:08:58.325 I think that people who have been in the position of running those teams 111 00:08:58.725 --> 00:09:00.485 have gone through that 112 00:09:01.045 --> 00:09:02.005 adjustment 113 00:09:02.085 --> 00:09:02.725 of 114 00:09:04.165 --> 00:09:05.765 coming to terms with 115 00:09:06.720 --> 00:09:14.560 whether the code is functionally correct is more important than whether it follows the idioms that are my preferred weapons of choice. 116 00:09:16.400 --> 00:09:17.200 The other 117 00:09:17.760 --> 00:09:24.615 way that I've seen this framed a lot in various blog posts and conversations in the community is that 118 00:09:25.575 --> 00:09:28.215 people

are no longer just individual contributors. 119 00:09:28.215 --> 00:09:34.535 Everybody has to become a manager, but instead of managing people, they're managing agents. And so you really need to 120 00:09:35.210 --> 00:09:42.890 bring your level of abstraction up to the point of the product or the feature and not the lines of code or the modules. 121 00:09:43.290 --> 00:09:43.850 And 122 00:09:44.250 --> 00:09:50.975 I think that one way that I've seen this play out is that teams who do have a strong culture of 123 00:09:51.055 --> 00:09:59.775 doing RFCs and architecture decision records as the initial step along the path of actually getting to that feature delivery 124 00:10:00.175 --> 00:10:13.110 are better situated to take advantage of the current landscape that we're in. Because once you have that RFC or the ADR or the spec that details, this is the thing that we actually want to build, 125 00:10:13.750 --> 00:10:15.430 then the implementation 126 00:10:15.510 --> 00:10:27.725 becomes automatic because the agent has that shared agreed upon context. And I think that the point of review just needs to shift earlier in the process where you need to get more people involved in that 127 00:10:28.445 --> 00:10:33.884 design and debate of what is the thing that we're actually trying to build and why versus how. 128 00:10:34.620 --> 00:10:37.100 And so as long as the 129 00:10:37.420 --> 00:10:38.940 code that lands 130 00:10:39.100 --> 00:10:43.660 can be tied back to and validated against the original design document, 131 00:10:43.660 --> 00:10:45.820 then the human review on the code 132 00:10:46.475 --> 00:10:50.475 doesn't need to happen as you just need to be able to have a way of 133 00:10:51.115 --> 00:10:55.835 doing that validation to show that it aligns with the stated goals. 134 00:10:57.515 --> 00:11:01.510 Two thoughts about that. So first, on the general direction, 135 00:11:01.589 --> 00:11:05.030 yeah, we're all product managers now, right? Or 136 00:11:06.149 --> 00:11:09.110 I think that we're being pushed in the direction 137 00:11:09.430 --> 00:11:15.625 where the value we add as humans is less on how do we implement this 138 00:11:16.185 --> 00:11:16.985 decision 139 00:11:17.385 --> 00:11:18.185 versus 140 00:11:18.425 --> 00:11:22.345 what decisions should we make? What is important to implement next? 141 00:11:22.825 --> 00:11:25.625 And so I think that developers 142 00:11:25.625 --> 00:11:27.785 who worked in an organization where there was 143 00:11:28.780 --> 00:11:29.660 a 144 00:11:29.660 --> 00:11:32.300 very stark line between those two groups 145 00:11:32.540 --> 00:11:38.540 and the engineers just implemented what another team was handing down as the decisions, 146 00:11:39.260 --> 00:11:40.860 I think that's going to be, 147 00:11:43.265 --> 00:11:48.305 the time where that's a useful use of human time is drawing to a close. 148 00:11:49.025 --> 00:11:52.065 It's not completely gone, but it's limited. 149 00:11:52.465 --> 00:11:54.305 And there's 150 00:11:54.305 --> 00:11:59.889 a line of sight to where it's like, yeah, you leave that to the agents. That's just what you do. 151 00:12:00.370 --> 00:12:04.130 So I think that teams that already had 152 00:12:04.290 --> 00:12:08.450 more of a porous boundary between engineer and product manager, 153 00:12:08.529 --> 00:12:11.329 and the product managers are now 154 00:12:12.445 --> 00:12:16.685 implementing proof of concepts with their own agents and the engineers are participating 155 00:12:16.685 --> 00:12:18.125 in design discussions, 156 00:12:18.605 --> 00:12:23.085 I think that's a much healthier place, and they're going to adapt a lot better. 157 00:12:24.940 --> 00:12:25.660 So 158 00:12:25.820 --> 00:12:28.460 spending a bit more time on the 159 00:12:29.020 --> 00:12:29.900 review, 160 00:12:30.380 --> 00:12:34.620 validation, and delivery phase of the software life cycle, 161 00:12:35.100 --> 00:12:37.020 there has been a lot of 162 00:12:38.265 --> 00:12:50.425 stake put in the role of the human in that where, oh, you need to have a human looking at the code before it can go to production because the humans are the ones that know whether it's going to work or not or have that broader context, 163 00:12:51.340 --> 00:12:53.740 which even before AI 164 00:12:53.820 --> 00:13:00.700 increased the volume of code wasn't always true. Because if you have somebody do a code review who hasn't worked in that code base before, 165 00:13:00.940 --> 00:13:04.300 that's not going to be very helpful unless the purpose is purely for 166 00:13:04.855 --> 00:13:07.895 educating them on what that code is doing and how. 167 00:13:08.695 --> 00:13:22.180 But even if you have somebody who is expert in that particular code base, they're liable to miss things. Or if they are the expert in the code base, they're likely already busy with their own implementation and don't want to have to spend those extra cycles on the code review. 168 00:13:23.540 --> 00:13:25.620 But there's still the element of 169 00:13:25.939 --> 00:13:40.765 some sort of validation and guardrails need to happen so that we can have confidence in the code as it is written before we put it into production because we don't wanna be dealing with constant firefighting due to buggy and broken code. And so I'm curious how you're 170 00:13:40.845 --> 00:13:47.630 seeing teams address that angle of what are those checks and guardrails that we need to put in place so that we can 171 00:13:48.510 --> 00:13:51.710 confidently remove humans from that phase of the life cycle. 172 00:13:52.990 --> 00:13:53.710 So 173 00:13:54.350 --> 00:13:55.630 this is where 174 00:13:56.110 --> 00:13:58.110 that article that I wrote that you mentioned 175 00:13:59.995 --> 00:14:01.515 becomes controversial 176 00:14:01.755 --> 00:14:02.715 because 177 00:14:03.275 --> 00:14:04.315 for reasons 178 00:14:04.875 --> 00:14:07.115 that I don't completely understand, 179

00:14:07.435 --> 00:14:09.595 when I say like I did just now, 180 00:14:10.475 --> 00:14:14.955 year from now or two years from now, humans just won't be writing code by hand anymore, 181 00:14:15.600 --> 00:14:17.520 Most people look at the 182 00:14:18.320 --> 00:14:32.015 curve that we've been writing for the last two years and say, Yeah, I can see how if I draw that line out a little bit more, that's probably what's going to happen. But then when I say, And you shouldn't be reviewing it either, and then they're like, Oh, like just heads exploding. 183 00:14:32.175 --> 00:14:34.735 People feel very, very strongly that, 184 00:14:35.535 --> 00:14:40.575 like you said, code should not be pushed to production without a human reading it first. 185 00:14:40.895 --> 00:14:41.535 And 186 00:14:42.095 --> 00:14:45.935 part of the research that I did for that article was 187 00:14:46.589 --> 00:14:49.630 looking at, if you actually study 188 00:14:49.630 --> 00:14:53.709 this and measure it, how many defects 189 00:14:54.509 --> 00:14:56.670 does human code review find? 190 00:14:57.709 --> 00:15:01.389 And the answer is typically about 30% of the defects. 191 00:15:02.055 --> 00:15:02.695 And 192 00:15:03.015 --> 00:15:06.535 if it's a more complicated problem domain like security, 193 00:15:06.775 --> 00:15:08.295 it's more like 20%. 194 00:15:08.695 --> 00:15:14.535 So humans aren't really good at doing code review from a standpoint of, 195 00:15:15.570 --> 00:15:16.210 this, 196 00:15:16.770 --> 00:15:18.370 am I finding the defects? 197 00:15:19.490 --> 00:15:28.464 And so I think the reason why people have that reaction of, I couldn't even dream of pushing this without a human review, 198 00:15:28.625 --> 00:15:36.225 is that we all have memories of the times where we did find a defect. And so this is part of like, our human ape brains 199 00:15:36.785 --> 00:15:38.945 are not evolved to be 200 00:15:39.264 --> 00:15:40.464 statistically 201 00:15:40.704 --> 00:15:42.520 literate. Like, really 202 00:15:42.920 --> 00:15:44.280 good at remembering 203 00:15:45.880 --> 00:15:47.720 exciting things that happened 204 00:15:47.880 --> 00:15:49.400 once or twice, 205 00:15:49.720 --> 00:15:51.720 but really bad at estimating, 206 00:15:51.720 --> 00:15:53.800 like, was this a good use of time 207 00:15:53.975 --> 00:15:59.895 to find those defects given all the other times that I did review and didn't find defects? 208 00:16:00.215 --> 00:16:01.575 So my 209 00:16:02.695 --> 00:16:04.775 standpoint is not 210 00:16:04.935 --> 00:16:12.800 that code review doesn't add value. I'm not saying that because, like I said, I can remember times I found bugs as much as anyone can. 211 00:16:13.120 --> 00:16:15.120 But my standpoint is that 212 00:16:15.680 --> 00:16:31.834 the time invested to find those defects is no longer worth it and should be spent. There are higher value ways to spend that time now. Even if I'm saying I need to spend 30% of my time or 50%

or 80% of my time doing 213 00:16:32.235 --> 00:16:33.755 quality assurance, 214 00:16:33.755 --> 00:16:39.675 or in the literal sense of the word that I'm assuring the quality of the code, that it does what it's supposed to do, 215 00:16:40.380 --> 00:16:44.380 Whatever number of time you think you should be spending on that 216 00:16:44.700 --> 00:16:45.900 class of problem, 217 00:16:46.460 --> 00:16:49.740 there's better options than reading the code by hand. 218 00:16:51.100 --> 00:16:53.260 Yeah, just anecdotally, 219 00:16:53.580 --> 00:16:54.140 I'll 220 00:16:54.504 --> 00:16:56.824 call out that I've actually got 221 00:16:57.144 --> 00:16:59.704 at least one production system. 222 00:16:59.785 --> 00:17:06.825 It's not customer facing, but it is team facing that I have written entirely by AI and haven't read 223 00:17:07.360 --> 00:17:08.959 possibly any of the code, 224 00:17:09.200 --> 00:17:12.799 and it's been very useful and helpful. It's actually a 225 00:17:12.960 --> 00:17:20.399 project that manages the team memories and task and project graphs for coding agents so that it's a shared 226 00:17:21.365 --> 00:17:35.605 flat graph that everybody can access and read and write to for being able to break down the context into chunks so that you don't have to have one session that runs for three days because you don't want to lose all the stuff that you put in at the beginning 227 00:17:36.040 --> 00:17:39.880 and also be able to have the memory be 228 00:17:40.280 --> 00:17:47.320 compatible across different agent harnesses rather than just a random markdown file somewhere on your disk that you hope doesn't disappear sometime. 229 00:17:47.720 --> 00:17:58.405 And I've bypassed a lot of the human review element of it as well because it works. I use it. I make changes. I make sure that the changes do what I want them to do, and then I ship it. 230 00:17:58.885 --> 00:18:00.005 Exactly. 231 00:18:00.085 --> 00:18:02.840 Yeah. And I think that's kind of like the 232 00:18:03.720 --> 00:18:06.840 easy case, right? And I don't mean that disparagingly, 233 00:18:06.840 --> 00:18:10.359 but in the sense that if I'm building an internal tool 234 00:18:10.600 --> 00:18:13.480 for a small audience where 235 00:18:13.835 --> 00:18:17.195 the world doesn't end if there's a bug, 236 00:18:17.595 --> 00:18:19.914 then yeah, just ship it and 237 00:18:20.475 --> 00:18:23.115 use it and fix problems when you find them. 238 00:18:23.675 --> 00:18:28.155 Then the more interesting question is, how do you do 239 00:18:29.320 --> 00:18:31.080 that situation 240 00:18:31.080 --> 00:18:37.400 where the world does end when you ship a bug? Or like there's potentially very bad consequences 241 00:18:38.600 --> 00:18:42.040 if you ship buggy code. And then how do we do that now? 242 00:18:43.785 --> 00:18:47.705 And I think that a healthy way to start that discussion 243 00:18:47.785 --> 00:18:49.705 is from what I just said that 244 00:18:49.945 --> 00:18:54.265 actually you weren't catching all of the bugs by reading the code before anyway. 245 00:18:54.905 --> 00:18:57.625 And therefore, what does that imply for 246 00:18:57.960 --> 00:18:59.639 what we should be doing 247 00:19:00.200 --> 00:19:03.639 with that time that we're no longer spending reading the code? 248 00:19:05.000 --> 00:19:09.960 And I think that one of the ways that we can spend that time is by 249 00:19:10.039 --> 00:19:15.245 getting the agent to help with building those quality gates and validation 250 00:19:15.245 --> 00:19:19.245 loops so that you can catch those errors early and often, 251 00:19:19.565 --> 00:19:22.125 preferably even before it gets to a PR, 252 00:19:22.285 --> 00:19:23.644 where I think maybe that 253 00:19:23.885 --> 00:19:43.625 ties us into some of the work that you're doing, where I know that the Bifrost tool is a linter and MCP server that gives a more comprehensive view of the code structure so the agent can have that insight rather than having to rely on a thousand GREP calls to find a particular symbol. I'd 254 00:19:43.625 --> 00:19:50.825 be interested in hearing some of what you're investing time in on that angle and also some of the other ways that you're seeing teams 255 00:19:50.985 --> 00:19:52.104 use the 256 00:19:52.345 --> 00:19:54.585 accelerated pace of delivery to 257 00:19:55.100 --> 00:19:57.500 put more of that work into the 258 00:19:58.380 --> 00:20:00.940 CI systems and guardrails and 259 00:20:01.660 --> 00:20:06.060 validation cycles that we should have been building all along, but deprioritized 260 00:20:06.060 --> 00:20:09.500 because it took too much time off of the actual product that we were trying to ship? 261 00:20:11.174 --> 00:20:12.215 Yeah. So, 262 00:20:13.975 --> 00:20:18.374 as you said, I'm building a tool called Bifrost that 263 00:20:18.695 --> 00:20:19.335 is... 264 00:20:19.895 --> 00:20:23.095 The way I think of it is it's basically Coverity, 265 00:20:23.470 --> 00:20:27.310 which is a static analysis tool that's been around since, 266 00:20:27.550 --> 00:20:29.710 man, I want to say like 1996. 267 00:20:29.710 --> 00:20:31.230 Like, it's pretty old, 268 00:20:31.550 --> 00:20:33.390 but it's 269 00:20:34.030 --> 00:20:38.350 it's really powerful and it's really slow and it's really difficult to extend. 270 00:20:39.095 --> 00:20:45.815 But what it can do is find things like a use after free, or it can find certain classes of concurrency 271 00:20:45.815 --> 00:20:46.775 deadlocks 272 00:20:46.775 --> 00:20:48.455 by analyzing 273 00:20:48.455 --> 00:20:51.015 the flow of data through your program. 274 00:20:51.495 --> 00:20:53.895 And so I want to bring that 275 00:20:54.270 --> 00:20:55.789 caliber of 276 00:20:57.070 --> 00:20:58.030 deterministic 277 00:20:58.030 --> 00:20:59.070 analysis 278 00:20:59.310 --> 00:21:01.150 into agent harnesses, 279 00:21:01.150 --> 00:21:14.755 where it's not like, let's push this and let CI crank on it for a

couple hours and see what it comes back with. But rather, the agent makes a tool call that says, run the use after free checks, and it gets back an answer in milliseconds 280 00:21:15.315 --> 00:21:17.315 instead of minutes or hours. 281 00:21:17.635 --> 00:21:18.995 And so to 282 00:21:20.150 --> 00:21:21.669 do that, to make 283 00:21:22.150 --> 00:21:27.109 Bifrost itself, which, so we're tackling 13 languages at once. And I 284 00:21:27.590 --> 00:21:34.230 I think, you know, there's other tools out there, but I think we're probably the first one to try to do so many languages all at once. 285 00:21:35.125 --> 00:21:36.644 And obviously, 286 00:21:36.645 --> 00:21:38.244 AI makes that possible 287 00:21:38.485 --> 00:21:40.005 with the word parallelization 288 00:21:40.005 --> 00:21:41.044 that I mentioned, 289 00:21:41.285 --> 00:21:42.804 but guaranteeing 290 00:21:42.885 --> 00:21:46.725 correctness is actually still really hard. There's so many corner cases 291 00:21:46.885 --> 00:21:51.250 when you have so many languages that you're dealing with. And so the 292 00:21:51.250 --> 00:21:59.890 big technique or the big idea that I orient my thinking around is I gave a talk not long ago where I talked about 293 00:22:00.610 --> 00:22:03.250 you should be trying to build an oracle 294 00:22:03.965 --> 00:22:07.965 for your agents, and you should use agents to build that oracle. 295 00:22:08.365 --> 00:22:15.485 And of course, an oracle is a term from AI engineering where we're talking about something that knows the right answer 296 00:22:16.039 --> 00:22:19.159 for a certain class of problem. And it's usually because 297 00:22:19.240 --> 00:22:20.520 we've constructed 298 00:22:20.520 --> 00:22:24.039 the problem in such a way that we know the answer 299 00:22:24.440 --> 00:22:31.000 upfront or we can derive the answer much more easily than we can in the fully general problem space. 300 00:22:31.795 --> 00:22:35.155 And so categories of tools that the 301 00:22:35.155 --> 00:22:36.835 QA engineering 302 00:22:36.835 --> 00:22:40.355 space knows about include things like fuzz testing, 303 00:22:40.835 --> 00:22:43.475 includes things like property checkers. 304 00:22:44.115 --> 00:22:44.835 And so 305 00:22:45.610 --> 00:22:47.610 we've built for Bifrost 306 00:22:47.610 --> 00:22:48.889 all of those things. 307 00:22:50.330 --> 00:22:51.849 And we use that to 308 00:22:52.250 --> 00:22:52.969 let 309 00:22:53.289 --> 00:22:54.249 the agents 310 00:22:54.409 --> 00:22:55.369 triage 311 00:22:55.529 --> 00:23:01.445 the bugs they find. So that's the other thing that agents unlock, by the way, is that a lot of these techniques, 312 00:23:01.525 --> 00:23:03.445 like fuzz testing isn't new and 313 00:23:04.245 --> 00:23:05.125 mutation 314 00:23:05.125 --> 00:23:06.485 testing isn't new. 315 00:23:06.805 --> 00:23:10.965 And all of these techniques that we use, none of these are new. I'm not that creative. 316 00:23:11.285 --> 00:23:13.285 But what's new is that 317 00:23:14.480 --> 00:23:15.280 before 318 00:23:16.240 --> 00:23:17.040 AI, 319 00:23:17.440 --> 00:23:18.400 if you had 320 00:23:19.840 --> 00:23:23.600 a sonar cube or a find bugs or a co verity 321 00:23:23.760 --> 00:23:27.040 or any of these static analysis checking 322 00:23:27.040 --> 00:23:27.600 reports, 323 00:23:28.385 --> 00:23:31.425 like they generate tons of false positives. 324 00:23:31.905 --> 00:23:32.465 And 325 00:23:32.705 --> 00:23:36.545 that's just the nature of the problem space. 326 00:23:37.025 --> 00:23:38.785 And so most teams 327 00:23:39.185 --> 00:23:40.145 didn't bother 328 00:23:40.399 --> 00:23:42.799 unless they were in an extremely 329 00:23:43.600 --> 00:23:45.519 sensitive space where 330 00:23:45.600 --> 00:23:47.999 releasing bugs is just catastrophic, 331 00:23:48.240 --> 00:23:49.839 in which case, like the 332 00:23:50.399 --> 00:23:51.279 investment 333 00:23:51.360 --> 00:23:55.440 of time to use these tools would pay off. But for most 334 00:23:56.085 --> 00:24:00.164 tools, and even at my last company, even at Data Stacks where we're building 335 00:24:00.405 --> 00:24:03.364 Apache Cassandra used by companies like 336 00:24:03.605 --> 00:24:14.159 Apple and Netflix and all of these other companies to store data that they cared very deeply about, Even there, we only made minimal use of this kind of static analysis because like 337 00:24:14.720 --> 00:24:18.799 I said, there's just so much false positives. But now you can just tell 338 00:24:19.760 --> 00:24:24.475 Opus or Soul or your model of choice, go through this report 339 00:24:24.635 --> 00:24:25.355 and 340 00:24:25.515 --> 00:24:30.554 throw out the false positives and create tickets for the ones that we need to fix. That 341 00:24:30.635 --> 00:24:32.715 unlocks so many of these 342 00:24:33.275 --> 00:24:35.355 things that software 343 00:24:35.355 --> 00:24:46.340 engineering researchers have come up with over the last twenty years and nobody could use because we didn't have the time to implement them or the effort didn't pay off. But now 344 00:24:47.220 --> 00:24:50.419 code isn't the only thing that's free anymore or very close to free. 345 00:24:51.060 --> 00:24:51.620 Reviewing 346 00:24:52.275 --> 00:24:55.155 the outputs of these tools is also free now. 347 00:24:56.675 --> 00:24:58.595 And so now that 348 00:24:58.915 --> 00:25:01.715 so much of the work can be automated, 349 00:25:03.475 --> 00:25:12.770 there's always the problem of running around with a hammer and trying to hit everything that might be a nail. And so I'm wondering what are some of the ways that you are seeing the 350 00:25:13.410 --> 00:25:20.050 best return on investment for human time spent and how... What are some of the useful heuristics around 351 00:25:20.895 --> 00:25:30.015 what are the places that the AI is going to be most useful and most effective versus the time where I need to use my own human judgment and reasoning, 352

00:25:30.095 --> 00:25:41.500 which also potentially segues us into the conversation of how do we help people build up that knowledge and reasoning if they're early in their career and haven't gone through all of the painful 353 00:25:41.500 --> 00:25:45.420 learning paths of people who have been in the industry prior to AI. 354 00:25:46.815 --> 00:25:48.335 Those are two 355 00:25:49.135 --> 00:25:52.174 very different topics, but both of them interesting. 356 00:25:52.255 --> 00:25:53.854 So on the first one, 357 00:25:54.174 --> 00:25:57.374 how do you decide where to spend your time? I think... 358 00:25:57.775 --> 00:26:00.654 So for me, the best use of time has been, 359 00:26:01.135 --> 00:26:02.950 as I mentioned, building 360 00:26:03.110 --> 00:26:04.469 more oracles 361 00:26:04.470 --> 00:26:08.149 and building the harnesses to drive those. 362 00:26:10.710 --> 00:26:15.349 Like I said, I'm not a super creative guy. So I just go to 363 00:26:15.669 --> 00:26:17.110 Chad GPT or Claude, 364 00:26:17.625 --> 00:26:19.144 and I say, the 365 00:26:19.945 --> 00:26:24.985 way I would start with a new project is I would say, Hey, what are some of the most interesting 366 00:26:25.065 --> 00:26:36.399 research approaches from the last twenty years of software development or software engineering research that I could apply to this project and order them by your expected cost benefit 367 00:26:36.559 --> 00:26:37.440 and then 368 00:26:38.000 --> 00:26:40.320 start figuring out how to build them from there. 369 00:26:41.120 --> 00:26:44.799 The other thing that's a good use of time is building better tools. 370 00:26:45.544 --> 00:26:47.465 So all 371 00:26:47.544 --> 00:26:56.825 of those things that I'm talking about, we have custom harnesses for Bifrost to drive those. It's not just like type into Codecs, Hey, do some differential 372 00:26:56.905 --> 00:27:00.184 analysis for me, but rather, We're going to build 373 00:27:00.570 --> 00:27:09.849 a system that can do that as a nightly job and then give me a report and feed that through the agent that's going to triage it for 374 00:27:10.010 --> 00:27:11.929 actual defects and so forth. 375 00:27:12.490 --> 00:27:16.385 And that can be at the micro level where, 376 00:27:16.945 --> 00:27:20.065 you know, since Bifrost is a tool for agents to use, 377 00:27:21.025 --> 00:27:24.065 a lot of what we do involves looking through the traces 378 00:27:24.145 --> 00:27:29.585 of a model that's trying to use the Bifrost tools and 379 00:27:30.500 --> 00:27:37.379 figuring out where the tools were less helpful than expected. So one of the very low level things we built is something to parse 380 00:27:37.460 --> 00:27:38.579 Codex's 381 00:27:38.659 --> 00:27:39.779 trajectories 382 00:27:39.779 --> 00:27:44.259 for the Bifrost tool calls and pull out the appropriate context so that 383 00:27:44.684 --> 00:27:48.044 Astra doesn't have to reinvent that wheel 384 00:27:48.205 --> 00:27:50.764 every time

it's And tackling one of these 385 00:27:53.245 --> 00:27:54.124 at 386 00:27:54.284 --> 00:27:56.924 a higher level, it could include things like Mjolnir, 387 00:27:57.529 --> 00:27:58.169 where 388 00:27:58.330 --> 00:27:59.210 I've 389 00:27:59.210 --> 00:28:01.210 created kind of a meta harness 390 00:28:01.690 --> 00:28:06.489 or a harness harness so that I can use my Codex and my 391 00:28:07.130 --> 00:28:08.090 Anthropic 392 00:28:08.250 --> 00:28:10.490 and my Kimi and my GLN 393 00:28:10.654 --> 00:28:11.534 subscriptions 394 00:28:11.534 --> 00:28:15.934 all in a single pane of glass and keep track of what the different agents are doing. 395 00:28:17.375 --> 00:28:19.374 The other thing that you mentioned was 396 00:28:19.534 --> 00:28:26.255 about how do you upskill as a junior engineer? So if you want to take the conversation in that direction, we could do that. 397 00:28:27.240 --> 00:28:31.400 Yeah. I think that that is an interesting angle as well because 398 00:28:31.400 --> 00:28:33.159 everybody's asking the question. 399 00:28:34.040 --> 00:28:38.280 Various people have attempted an answer, but I haven't seen a... 400 00:28:39.525 --> 00:28:41.605 I haven't seen any sort of 401 00:28:42.725 --> 00:28:50.645 standard developed as far as what are the actual ways that we do that. Not that we did a great job of it prior to AI, but now there are 402 00:28:50.965 --> 00:28:58.019 potentially fewer opportunities or depending how you look at it, more opportunity because the junior engineer can get more done, 403 00:28:58.340 --> 00:29:06.019 but they're also holding a bigger foot gun. So there's potentially a larger blast radius of poor understanding of the problem. 404 00:29:06.419 --> 00:29:07.620 And I think that 405 00:29:08.174 --> 00:29:10.014 one of the ways that 406 00:29:10.495 --> 00:29:17.614 humans can spend their time and tying back to what you were saying about mining the agent traces for how the... 407 00:29:18.095 --> 00:29:22.095 How it's issuing the tool calls for Bifrost or any other tool 408 00:29:22.679 --> 00:29:23.879 is that 409 00:29:24.120 --> 00:29:27.159 in order to prevent a lot of wasted 410 00:29:27.399 --> 00:29:34.119 compute and tokens on retreading the same ground over 50 different agent sessions, as well as 411 00:29:34.519 --> 00:29:37.080 giving junior engineers a place to 412 00:29:37.325 --> 00:29:42.605 look to understand what are the best practices or how do we tackle these types of problems 413 00:29:43.165 --> 00:29:46.845 is through the introduction of different skills 414 00:29:47.005 --> 00:29:47.965 documents 415 00:29:47.965 --> 00:29:53.085 that helps the agent do the work, but also acts as an artifact that people can look at to understand 416 00:29:53.539 --> 00:29:57.059 what do we... What what is the approach for doing this, 417 00:29:57.460 --> 00:30:14.144 and why did we do it this way so that acts as a a shared repository of knowledge if you take the time to actually look at it.

And that's actually another area that I and my team have been spending time is every time I have to keep doing something a certain way, I say, okay. Great. Let's go ahead and 418 00:30:14.465 --> 00:30:17.424 read back through all the sessions where I did this particular 419 00:30:17.664 --> 00:30:18.624 workflow, 420 00:30:18.705 --> 00:30:25.140 codify that into a skill so that I can just trigger it when I wanna do that thing again. So one example of that is actually 421 00:30:25.460 --> 00:30:34.420 for code review because I'll open a pull request from my agent. I'll get a different agent to do reviews on that. Lately, it's been Copilot, 422 00:30:34.420 --> 00:30:45.634 but whatever other agent. And then I have a skill that says address PR feedback that will just go through, find all the comments, and pull them in, perform the necessary... Evaluate, perform the necessary changes, 423 00:30:45.875 --> 00:30:47.315 and then close those responses. 424 00:30:48.370 --> 00:30:59.409 And so that gives me a way to go back and forth on create the code, get some feedback on it, address some of the problems without having to involve a human until after that's already been done. If I say, 425 00:30:59.809 --> 00:31:08.585 this actually is a large enough change that I wanna have somebody else look to make sure that my general approach and design is correct before I actually merge and ship this. 426 00:31:09.865 --> 00:31:10.904 Yeah. Someone... 427 00:31:11.544 --> 00:31:20.210 So so for for a couple years there, and I think we've broadly moved past this point, but for a couple of years, people were saying, Well, the AI is like an intern. 428 00:31:20.690 --> 00:31:21.169 And 429 00:31:21.890 --> 00:31:29.970 it's reasonably smart, but it makes a lot of mistakes and it kind of doesn't know what it doesn't know. So you got to supervise it closely. And that was their mental model 430 00:31:30.825 --> 00:31:32.184 of the AI agent. 431 00:31:32.585 --> 00:31:33.144 And 432 00:31:33.544 --> 00:31:36.984 now the agents are smarter and better at writing code, 433 00:31:37.385 --> 00:31:40.664 but they still don't have the big picture. 434 00:31:40.904 --> 00:31:50.349 So someone said, and I don't remember who, that a better model now is it's not an intern so much as it's a great new hire. 435 00:31:51.070 --> 00:31:55.149 And every time you talk to him, it's his first day at the company. 436 00:31:55.965 --> 00:31:57.164 And so 437 00:31:57.405 --> 00:31:58.684 what 438 00:31:58.684 --> 00:32:06.124 that pushes us towards is some of the things that you're talking about that were kind of best practices already in human oriented teams, 439 00:32:06.684 --> 00:32:08.044 at least teams that 440 00:32:08.445 --> 00:32:10.649 hired new people regularly, 441 00:32:10.730 --> 00:32:14.809 that you need to have this knowledge base and this, 442 00:32:14.970 --> 00:32:17.289 you know, a set of runbooks and 443 00:32:17.929 --> 00:32:21.049 things that let those new hires

orient themselves 444 00:32:21.274 --> 00:32:27.195 to what's going on. And so what are the decisions we've made that we don't need to relitigate every day? 445 00:32:27.595 --> 00:32:28.154 And 446 00:32:29.595 --> 00:32:35.434 what's our big hairy goal, right? What are we trying to build here? And why? And what problems are we solving? 447 00:32:37.429 --> 00:32:38.149 And 448 00:32:39.110 --> 00:32:42.149 so the difference is, of course, that 449 00:32:44.070 --> 00:32:45.429 maintaining that 450 00:32:45.830 --> 00:32:49.509 was always super, super difficult in the before times. 451 00:32:49.910 --> 00:32:52.549 And so most teams 452 00:32:52.115 --> 00:32:54.275 didn't 453 00:32:54.275 --> 00:32:59.075 do as good a job as they could have. And now I can just say, Okay, Fable, 454 00:32:59.235 --> 00:33:02.995 document what we did and the dead ends we ran into, 455 00:33:03.155 --> 00:33:03.955 and put 456 00:33:06.435 --> 00:33:07.955 that in our set of 457 00:33:08.490 --> 00:33:09.769 plan documents 458 00:33:09.770 --> 00:33:12.330 so that we don't have to re derive that next time. 459 00:33:12.890 --> 00:33:14.330 And we 460 00:33:14.650 --> 00:33:20.649 can talk about how I don't think text files I come from a database background. So I think there's better solutions than a 461 00:33:20.890 --> 00:33:34.044 directory full of Markdown files. But that's kind of where we are right now, is the Markdown file level. And it works pretty well. Eventually, you do outgrow it, but it does work pretty well, surprisingly well even, 462 00:33:34.525 --> 00:33:36.765 for what it is. And then 463 00:33:38.440 --> 00:33:40.519 as a very tactical piece, 464 00:33:41.400 --> 00:33:44.359 one of the things that works really well for us is 465 00:33:44.679 --> 00:33:46.599 OpenAI wrote an article 466 00:33:47.480 --> 00:33:52.600 almost a year ago now. I think it was in November of 'twenty five, but it 467 00:33:52.515 --> 00:33:53.955 was late 'twenty five. 468 00:33:54.275 --> 00:33:59.154 And they wrote an article about using something called exec plans 469 00:33:59.315 --> 00:34:01.875 to drive agents for longer 470 00:34:01.875 --> 00:34:02.755 tasks. 471 00:34:02.995 --> 00:34:03.794 And the 472 00:34:04.275 --> 00:34:07.155 framing of the article was, here's a structure 473 00:34:07.870 --> 00:34:12.510 for how to create plan documents and have your agent create the plan documents 474 00:34:12.510 --> 00:34:15.230 that it can follow that will keep it on task 475 00:34:15.630 --> 00:34:20.925 over a task that takes hours or days. And remember, this is back like 476 00:34:20.925 --> 00:34:22.605 Codex 5.1, 477 00:34:22.605 --> 00:34:28.765 I think was state of the art because 5.2 came out in December and this was definitely before that. 478 00:34:29.005 --> 00:34:31.565 And the Opus was around 4.5, 479 00:34:31.565 --> 00:34:34.125 I think. And both of these models 480 00:34:34.190 --> 00:34:35.870 were the start 481 00:34:36.270 --> 00:34:36.910 of 482

00:34:37.230 --> 00:34:46.750 the new agentic age that we're in now, but they both got distracted really easily. And they both were a little bit lazy where if you gave them a big task, they'd do the first part and 483 00:34:47.070 --> 00:34:47.550 then stop. 484 00:34:48.444 --> 00:34:52.445 And if you just said, well, slash goal do the big task, then they'd 485 00:34:52.765 --> 00:35:05.930 go all over the place and compaction wasn't as good and so forth. So OpenAI said, Hey, use this exec plan idea to keep them on track across compactions in this longer running task and so forth. And we mostly don't need 486 00:35:06.410 --> 00:35:07.770 plan documents 487 00:35:07.930 --> 00:35:09.610 for that purpose anymore. 488 00:35:10.330 --> 00:35:13.290 Or rather, doing a slash plan in 489 00:35:13.610 --> 00:35:15.370 the harness is sufficient. 490 00:35:15.984 --> 00:35:20.065 But we're still doing exact plan documents because of 491 00:35:20.224 --> 00:35:22.785 this category of problem where 492 00:35:23.025 --> 00:35:27.265 two weeks later or two months later, why did we do things this way? 493 00:35:27.585 --> 00:35:38.440 And the exact plan captures that in a way that seems to fit how the agents want to work a lot better than trying to put it into a comment in the code 494 00:35:38.759 --> 00:35:47.605 because for a bunch of reasons, the grep and set approach to going through code means that they have holes in their knowledge and they'll easily miss that kind of comment. 495 00:35:47.765 --> 00:35:51.925 But if you have a document that says, Here's where all of our plans 496 00:35:51.925 --> 00:35:52.645 live, 497 00:35:53.045 --> 00:35:57.605 and go look at them if you need to understand why we did something, that works a lot better. 498 00:35:58.710 --> 00:36:10.630 Yeah. And similarly for architecture decision records, this is the design and why we chose to do it this way that the humans can also look at, and that's something that predates AI workflows as well of 499 00:36:11.015 --> 00:36:31.260 humans who are joining the company afresh, and they're working in that thorny corner of the code and think, I know exactly how to solve this. And then there's... There... I've definitely seen tropes of people saying, okay. Every time you think you're going to solve this, increment this counter of how many times people have tried to do that before going back to whatever it was before. 500 00:36:31.740 --> 00:36:36.460 And on that point of... Yeah. Yeah. On that point of databases 501 00:36:36.460 --> 00:36:36.940 being 502 00:36:37.415 --> 00:36:45.335 a better structure for some of this style of data, that... That's also why I built the tool that I'm using for my shared agent 503 00:36:45.335 --> 00:36:49.415 task and memory context of... It's actually using a storage layer called OmniGraph 504 00:36:49.790 --> 00:36:52.830 and the backing store that uses Lance as the, 505 00:36:53.230 --> 00:36:58.030 storage format, but builds a graph on top of that. So you can actually have the connectivity 506 00:36:58.030 --> 00:37:09.105 between projects and tasks and memories and have the way to be able to go from one to the other and also being able to have memories that get superseded so that as you find out more things, 507 00:37:09.505 --> 00:37:15.265 you can have the most recent, but you still have the way to backtrack to see what did we think we do at this point in time. 508 00:37:15.585 --> 00:37:16.865 And so that's a 509 00:37:17.190 --> 00:37:19.750 interesting angle as well. Oh, so you 510 00:37:20.070 --> 00:37:24.710 check this out. A couple friends of mine started a company called Vectorize, 511 00:37:25.109 --> 00:37:26.150 and their 512 00:37:26.470 --> 00:37:27.270 flagship 513 00:37:27.270 --> 00:37:29.109 product is called Hindsight. 514 00:37:29.645 --> 00:37:31.005 And it's basically 515 00:37:32.125 --> 00:37:34.525 graph and embeddings based 516 00:37:35.325 --> 00:37:36.685 memory for agents. 517 00:37:37.005 --> 00:37:37.725 So, 518 00:37:38.045 --> 00:37:40.365 yeah, it is on GitHub. 519 00:37:40.685 --> 00:37:43.244 Yeah, I think you should probably check it out and see. 520 00:37:44.170 --> 00:37:55.130 At least there's probably some interesting ideas that might be useful cross pollination, and it might even be where you're like, Oh, I don't need to maintain my own anymore. I can use someone else's who does this full time. 521 00:37:56.825 --> 00:37:59.065 Yeah, I'll definitely take a look at that as well. 522 00:38:00.505 --> 00:38:05.385 And that's the other interesting aspect of the time that we're in, is there is a massive proliferation 523 00:38:05.385 --> 00:38:07.224 of people who are solving 524 00:38:07.224 --> 00:38:11.704 adjacent problems or maybe even the same problem in slightly tangential 525 00:38:11.730 --> 00:38:12.930 ways where 526 00:38:13.090 --> 00:38:19.010 we're having an explosion of what are the different concepts and primitives that we actually need for 527 00:38:19.170 --> 00:38:21.490 software engineering agents or 528 00:38:21.490 --> 00:38:24.130 agents for particular workflows. 529 00:38:24.609 --> 00:38:27.170 So it's the let a thousand flowers bloom 530 00:38:27.455 --> 00:38:28.335 mentality. 531 00:38:28.335 --> 00:38:32.735 It'll be interesting to see where we are two years from now once the 532 00:38:32.975 --> 00:38:38.895 market starts to coalesce around some of the primitives that we all agree are useful. 533 00:38:39.295 --> 00:38:47.910 These are the two or three different right ways to do it and how that collapses some of the tools and products that are being launched right now. 534 00:38:48.710 --> 00:38:53.430 Yeah, I think that's true. And I think that's also something that we collectively 535 00:38:53.990 --> 00:38:55.990 need to be a little bit 536 00:38:57.494 --> 00:38:59.175 intentional about. Because 537 00:38:59.494 --> 00:39:00.375 for me, 538 00:39:00.775 --> 00:39:02.855 and I think for a lot of engineers, 539

00:39:03.895 --> 00:39:04.535 whenever 540 00:39:05.414 --> 00:39:07.895 I see somebody selling something 541 00:39:08.055 --> 00:39:12.960 where I'm like, Oh, I understand how to build that. It's a very, 542 00:39:13.120 --> 00:39:18.080 very... It's not a large step to go from, I understand how to build that, to, I should build that. 543 00:39:18.320 --> 00:39:21.680 Especially if it means that I don't have to pay someone 544 00:39:21.840 --> 00:39:23.840 for something that I could do myself. 545 00:39:24.160 --> 00:39:24.720 But 546 00:39:25.805 --> 00:39:27.645 the good part about 547 00:39:27.885 --> 00:39:31.085 AI code is that building my own is 548 00:39:32.285 --> 00:39:33.964 easier than it's ever been. 549 00:39:34.765 --> 00:39:35.805 But I 550 00:39:36.365 --> 00:39:37.964 think that that means that 551 00:39:38.300 --> 00:39:43.900 there's a trap of not seeing like the rest of the iceberg under the surface 552 00:39:44.060 --> 00:39:46.460 that, you know, building the prototype, 553 00:39:46.460 --> 00:39:47.820 yeah, that's a weekend, 554 00:39:48.060 --> 00:39:51.020 but maintaining it and 555 00:39:52.425 --> 00:39:55.865 handling feature requests from my team, that's 556 00:39:55.865 --> 00:39:57.545 a serious time sink 557 00:39:57.705 --> 00:40:00.345 over the next six months, even 558 00:40:00.665 --> 00:40:04.345 if AI is doing most of the actual code authoring. 559 00:40:04.700 --> 00:40:05.660 So I 560 00:40:06.220 --> 00:40:07.260 think that, 561 00:40:08.140 --> 00:40:09.980 and I certainly do this myself, 562 00:40:10.140 --> 00:40:13.260 and I hope I haven't fallen into that trap with Mjolnir, 563 00:40:13.660 --> 00:40:17.820 that building your own is seductive, but not always the right call. 564 00:40:18.725 --> 00:40:35.365 It's the age of the content management system or static site generator where everybody says, Oh, that's an easy problem. I can build my own. And then they get 80% of the way there and said, Oh, I'm almost done. Now I'll spend the rest of my time trying to build that other 20%. 565 00:40:35.040 --> 00:40:37.120 Yeah, that's a really good analogy. Yeah. 566 00:40:37.920 --> 00:40:39.600 That's 567 00:40:39.600 --> 00:40:44.320 definitely another interesting aspect of... Have I built a static site generator? Yes, I have. 568 00:40:46.095 --> 00:40:53.375 And so that's another interesting aspect of where we are in terms of how do you decide what are the appropriate 569 00:40:53.855 --> 00:40:54.895 ways and 570 00:40:55.455 --> 00:40:58.095 places to spend your limited human attention. 571 00:40:58.809 --> 00:40:59.609 Because, 572 00:40:59.770 --> 00:41:04.650 as you said, it is very seductive to say, oh, I know how to build that. I'll go build it for myself. But now 573 00:41:04.970 --> 00:41:08.250 more of my time is getting sunk into this offshoot 574 00:41:08.490 --> 00:41:24.375 toy project that is maybe helpful, but is distracting from what are the actual concrete broader goals that I'm getting paid to achieve. I mean, if you're doing it for fun in your hobby time, sure. But, again, you can only have so many hobbies before they take 575 00:41:24.375 --> 00:41:26.855 over an increasing amount of your life. And 576 00:41:27.849 --> 00:41:32.330 I I think another aspect of this problem going beyond just 577 00:41:32.490 --> 00:41:36.250 agenting software engineering into the broader agent ecosystem 578 00:41:36.250 --> 00:41:36.890 is 579 00:41:37.210 --> 00:41:42.810 what is the role of humans when agents can do more and more of the daily tasks of 580 00:41:43.185 --> 00:41:45.345 dealing with my emails or 581 00:41:45.665 --> 00:41:46.465 managing 582 00:41:46.465 --> 00:41:47.185 the 583 00:41:47.825 --> 00:41:49.745 data ingest workflows 584 00:41:49.745 --> 00:41:52.865 as data continues to break or 585 00:41:53.185 --> 00:41:53.905 building 586 00:41:54.065 --> 00:42:00.870 and iterating on the product based on what are the requests that I'm getting from my users. And that 587 00:42:01.030 --> 00:42:10.869 brings back that question of human judgment that open source maintainers have had to deal with for years now. Just because somebody can request a feature doesn't mean that it's a feature that I want to own. 588 00:42:11.755 --> 00:42:13.675 No, that's 100 589 00:42:13.675 --> 00:42:14.715 true. And 590 00:42:15.035 --> 00:42:17.995 again, I don't remember who said this first and 591 00:42:18.075 --> 00:42:19.755 possibly lots of people, 592 00:42:20.075 --> 00:42:25.675 but the most important job of an open source maintainer is saying no. And that 593 00:42:26.155 --> 00:42:26.635 was true 594 00:42:27.960 --> 00:42:31.400 five years ago, but it's significantly more true now. 595 00:42:32.119 --> 00:42:32.680 And 596 00:42:33.000 --> 00:42:34.520 to the point that, 597 00:42:34.840 --> 00:42:35.800 by the way, 598 00:42:36.840 --> 00:42:38.600 I definitely appreciate 599 00:42:39.240 --> 00:42:39.960 everyone 600 00:42:40.040 --> 00:42:42.280 whose 601 00:42:41.105 --> 00:42:43.585 libraries I'm using on GitHub 602 00:42:43.585 --> 00:42:46.705 who have PRs open. And just 603 00:42:47.825 --> 00:42:53.025 yesterday, I upstreamed PRs to two different libraries that Brock is using, 604 00:42:54.065 --> 00:42:55.505 and they were both accepted, 605 00:42:56.559 --> 00:42:59.200 or rather the sets of PRs were both accepted. 606 00:42:59.839 --> 00:43:06.080 So, I appreciate being able to contribute like that. But at the same time, I don't think that that's 607 00:43:07.920 --> 00:43:08.400 scalable 608 00:43:09.155 --> 00:43:13.555 as the barrier towards, Oh, I have a problem. Let me throw up a PR. 609 00:43:13.875 --> 00:43:15.395 Because a lot of people 610 00:43:17.395 --> 00:43:19.795 do throw up PRs without actually 611 00:43:21.170 --> 00:43:24.690 doing a whole lot of the kind of review that we're talking about. 612 00:43:25.090 --> 00:43:33.170 And so, it's very easy for those open source maintainers to get snowed under

by trying to give feedback 613 00:43:33.545 --> 00:43:43.545 on those PRs that were written by agents with very little human oversight and sometimes by people who aren't professional developers. So on the one hand, I'm 614 00:43:43.545 --> 00:43:46.105 not here to gatekeep, and I think it's great that 615 00:43:46.830 --> 00:43:49.710 AIs are opening up creating 616 00:43:50.110 --> 00:43:53.230 tools to make lives better in digital 617 00:43:53.790 --> 00:43:54.750 worlds 618 00:43:54.830 --> 00:44:03.915 for people who aren't professional developers. I think that's great. But at the same time, when the number of people who can open a PR to your product 619 00:44:04.155 --> 00:44:04.955 is 620 00:44:05.595 --> 00:44:08.635 literally everyone in the world and not just the professionals, 621 00:44:08.635 --> 00:44:11.835 it does mean that maybe there needs to be another barrier 622 00:44:12.339 --> 00:44:12.980 to 623 00:44:13.380 --> 00:44:15.380 reduce that maintainer 624 00:44:15.380 --> 00:44:16.180 burnout. 625 00:44:16.180 --> 00:44:31.205 And so what we're doing with Bifrost, we haven't done this with any of our other repos, but with our flagship Bifrost repo, we have turned off PRs. And so the issues are open. If you have a problem or a feature you want to request, 626 00:44:31.365 --> 00:44:35.205 open an issue. But it's just faster for me to 627 00:44:35.365 --> 00:44:37.205 drive that in the right direction 628 00:44:38.109 --> 00:44:41.630 with my own agents, with my own harness, with my own QA 629 00:44:41.710 --> 00:44:42.590 system, 630 00:44:42.910 --> 00:44:51.070 than it is to try to do a dance of, Hey, can you fix this? That can potentially take several days with contributors. 631 00:44:51.070 --> 00:44:53.310 So I think that's unfortunate 632 00:44:53.205 --> 00:44:54.005 because 633 00:44:54.005 --> 00:44:55.925 I think that that 634 00:44:56.405 --> 00:44:57.445 open source 635 00:44:57.445 --> 00:44:58.325 ecosystem 636 00:44:58.325 --> 00:45:08.165 and community building and training the next generation and all of that were good outcomes from the way we used to do it. But there is historical precedent, 637 00:45:08.970 --> 00:45:14.730 which is that back in the '80s when Richard Stallman started the Free Software Foundation, 638 00:45:14.890 --> 00:45:29.255 it was not a GitHub style, Hey, everyone, bring your contributions and let's build this together. He believed very strongly and still believes very strongly in the importance of having the source available so that you can make changes and have the freedom 639 00:45:29.495 --> 00:45:30.455 to do so. 640 00:45:30.855 --> 00:45:31.415 But 641 00:45:31.735 --> 00:45:33.015 the FSF 642 00:45:33.015 --> 00:45:37.015 in the 80s did not accept patches from external contributors. 643 00:45:37.860 --> 00:45:41.540 It was much more of a shared source, is what we'd call it today, 644 00:45:41.780 --> 00:45:43.300 than open source. 645 00:45:43.780 --> 00:45:45.220 And I 646 00:45:45.380 --> 00:45:47.540 kind of think that's the direction we're heading in. 647 00:45:49.140 --> 00:46:08.970 Yeah, I've definitely heard about a number of other open source projects that have followed that same model of you can open an issue, and then I will send that issue to my agent to do the code change because of the fact that, as you said, the PR itself requires a lot more time investment and communication overhead. 648 00:46:12.010 --> 00:46:15.770 Think that one way that, as an industry, 649 00:46:15.930 --> 00:46:16.570 we 650 00:46:17.184 --> 00:46:20.145 should be thinking about the ways that we develop 651 00:46:20.145 --> 00:46:21.905 our workflows around 652 00:46:21.984 --> 00:46:25.265 AI being the primary actor 653 00:46:25.505 --> 00:46:26.224 is 654 00:46:26.385 --> 00:46:27.105 to 655 00:46:27.664 --> 00:46:29.664 codify and enforce 656 00:46:29.664 --> 00:47:09.960 more of that human judgment requirement where rather than giving the agent an objective and say, k. Go build this thing, force it to ask you why or what is the actual thing that we're trying to achieve, be... Which that's a human problem as well, where I can go and tell one of my employees, go build this thing, and they'll happily go off and build the thing that I said because I am perceived as the expert. But maybe I don't actually know what I'm talking about in this regard. And so having more of that pushback and requiring the human to be more involved in the end to end delivery of what are we building? Why are we building it? What are the things that we need to think about before we put this thing out into the world? And then also maybe to your point of 657 00:47:10.625 --> 00:47:12.865 there are 50 different versions of 658 00:47:13.025 --> 00:47:14.145 problem x, 659 00:47:14.625 --> 00:47:19.425 agentic memory, coding harnesses. Those are some of the ones for this era. 660 00:47:19.744 --> 00:47:21.825 Go and do some research to see, 661 00:47:22.700 --> 00:47:29.260 do any of those solve 90% of the problem that I think I'm solving, and maybe I should just use those instead of building another one. 662 00:47:30.540 --> 00:47:31.180 Yeah. 663 00:47:32.140 --> 00:47:35.900 Yeah, I do wanna come back to to what you said earlier about 664 00:47:36.295 --> 00:47:40.695 upskilling the next generation. We talked a little bit about onboarding 665 00:47:40.935 --> 00:47:45.175 the agents kind of in the context of onboarding new humans, 666 00:47:45.495 --> 00:47:48.135 but that's 667 00:47:48.135 --> 00:47:50.855 a bit of a different problem than how do you go from 668 00:47:51.710 --> 00:47:55.869 a new college graduate to being able to do what you do 669 00:47:56.190 --> 00:47:56.990 in 670 00:47:57.150 --> 00:48:00.109 terms of being able to supervise agents effectively. 671 00:48:00.589 --> 00:48:01.150 And 672 00:48:02.589 --> 00:48:07.615 I don't really have a good solution to that. And I'm kind of hoping that you do because, 673 00:48:08.335 -->

00:48:17.055 or that at least collectively we can come up with ways to do that. The best that I've found so far, or the best that I've been able to 674 00:48:17.375 --> 00:48:19.610 come up with so far is, 675 00:48:19.770 --> 00:48:22.410 so my 18 year old right now is 676 00:48:23.130 --> 00:48:25.450 studying computer science and he wants 677 00:48:26.090 --> 00:48:27.290 to be an engineer. 678 00:48:27.610 --> 00:48:28.250 And, 679 00:48:28.410 --> 00:48:32.570 you know, the rule of thumb with him is, you know, you can use agents 680 00:48:33.375 --> 00:48:34.015 to 681 00:48:34.255 --> 00:48:35.455 do anything 682 00:48:35.455 --> 00:48:38.415 that's routine that you understand how to do, 683 00:48:38.815 --> 00:48:40.095 but don't 684 00:48:40.095 --> 00:48:48.175 use it for anything. Don't throw an open problem that you don't understand at Claude and say, solve this for me, because then you're just cheating your future self 685 00:48:48.520 --> 00:48:50.360 of that understanding. 686 00:48:50.360 --> 00:48:56.280 And there's been, you know, I think we have a little bit of historical 687 00:48:56.600 --> 00:49:00.440 precedent there where we've taught kids to do arithmetic 688 00:49:00.755 --> 00:49:05.715 in their heads or on paper, even though calculators have existed for 689 00:49:06.515 --> 00:49:07.955 lots of years and 690 00:49:08.355 --> 00:49:11.795 slide rules were a thing even before that. So there's 691 00:49:11.795 --> 00:49:12.915 some value 692 00:49:13.130 --> 00:49:16.170 and we've created some patterns for, 693 00:49:16.330 --> 00:49:24.410 here's how you do something by hand and here's why it's valuable to do something by hand, even though we've already figured out how to automate it. But 694 00:49:25.385 --> 00:49:30.185 this pushes that a lot farther down. It's like even deep into 695 00:49:30.425 --> 00:49:36.185 graduate school, you'll still be doing things by hand that we already can automate with agents, 696 00:49:36.505 --> 00:49:42.500 which is, at the very least, it's a mental shift for our educational system. 697 00:49:43.380 --> 00:49:57.045 And maybe one way to think about this in the context of software engineering teams, particularly for people who are earlier in their career or who are working in an unfamiliar area and trying to upskill is the idea of training wheels where 698 00:49:57.205 --> 00:50:22.655 the harness and the agent will do the work, but maybe it will only execute on it after you've been able to provide a satisfactory description of the problem space and what you've thought through. And maybe as you get more adept at a particular area, it adapts and says, okay, well, you have already proven your expertise in this area. So I'm going to go ahead and do what you asked me to do because you've demonstrated ability 699 00:50:22.655 --> 00:50:33.295 similar to the way that we demonstrate ability in our work and get promotions in that regard. And so having the harness push back more and not just blindly accept every instruction that is thrown at it. 700 00:50:33.855 --> 00:50:35.375 Yeah. And then another 701 00:50:35.720 --> 00:50:38.839 analogy that gives me a little bit of hope here 702 00:50:39.079 --> 00:50:41.400 is the chess space, 703 00:50:41.880 --> 00:50:44.680 where it's literally been thirty years 704 00:50:45.000 --> 00:50:51.240 since a human was the best chess player in the world. Deep Blue beat Gary Kasparov in 1997, 705 00:50:51.945 --> 00:50:54.185 and that was a rematch. He the 706 00:50:54.345 --> 00:50:57.145 first Deep Blue in 'ninety six, if I remember correctly, 707 00:50:57.225 --> 00:51:00.665 but then they went back and revised their algorithms and 708 00:51:01.465 --> 00:51:04.505 expanded their hardware, and then they won the rematch. 709 00:51:05.305 --> 00:51:06.985 And, you know, today, of course, 710 00:51:07.760 --> 00:51:08.880 you don't 711 00:51:09.200 --> 00:51:11.680 need a big RS6000 712 00:51:12.000 --> 00:51:12.800 supercomputer 713 00:51:12.800 --> 00:51:13.760 powering it. 714 00:51:14.800 --> 00:51:17.920 My watch could be all but the best humans at chess. 715 00:51:20.655 --> 00:51:21.375 But, 716 00:51:21.535 --> 00:51:22.575 and yet, 717 00:51:23.375 --> 00:51:24.255 with that, 718 00:51:24.575 --> 00:51:28.975 chess streamers are one of the most popular forms of entertainment on Twitch. 719 00:51:29.215 --> 00:51:29.695 And 720 00:51:31.455 --> 00:51:32.655 they're getting more popular. 721 00:51:33.180 --> 00:51:34.619 So there's 722 00:51:34.619 --> 00:51:38.700 space for, even though machines are better at this than humans, 723 00:51:38.940 --> 00:51:42.380 it's still useful and still interesting for humans to 724 00:51:42.780 --> 00:51:44.460 compete and try to improve 725 00:51:45.035 --> 00:51:45.675 in 726 00:51:45.995 --> 00:51:47.275 that problem space. 727 00:51:47.675 --> 00:51:48.555 And of course, 728 00:51:48.955 --> 00:51:49.755 in the 729 00:51:49.995 --> 00:51:51.595 less intellectual domain, 730 00:51:52.075 --> 00:51:54.635 we've had machines that are better at 731 00:51:55.115 --> 00:51:56.955 any kind of physical 732 00:51:58.310 --> 00:52:03.030 exertion than humans for two hundred years, and we still go to the gym to 733 00:52:03.350 --> 00:52:04.950 get more athletic. So 734 00:52:06.790 --> 00:52:16.535 I'm not in the camp of, Well, AIs are better than humans at everything, so we might as well give up. I think that there is some value to doing things as a human, 735 00:52:16.695 --> 00:52:19.495 even if the machine is better at it than I am. 736 00:52:19.815 --> 00:52:20.775 But yeah, 737 00:52:21.015 --> 00:52:26.295 the world is not going to be the same. It's not going to go back to being the same. So it's going to be exciting. 738 00:52:27.730 --> 00:52:29.170 And also, 739 00:52:29.490 --> 00:52:34.050 there is always a role for the human to play because despite all of the terminology 740 00:52:34.050 --> 00:52:37.810 that we use and all of the hype that has

grown up around the ecosystem, 741 00:52:38.370 --> 00:52:43.785 these large language models or these various deep learning models do not have any agency 742 00:52:43.945 --> 00:52:47.305 of their own. They are a tool that we point in the direction. 743 00:52:47.465 --> 00:52:53.305 And so you still need to have that human judgment of what is the direction I want this thing to go in and understand, 744 00:52:54.369 --> 00:53:00.290 one, what are the capabilities and why am I going in this direction, but also have the sufficient 745 00:53:00.290 --> 00:53:03.650 knowledge to understand whether or not you have achieved the stated goal. 746 00:53:04.770 --> 00:53:08.450 See, that's something that I'm not sure about. That's certainly true today, 747 00:53:08.964 --> 00:53:12.165 and it's certainly true next year. Is it true in ten years? 748 00:53:12.565 --> 00:53:15.605 Do we never get to a point where I can tell 749 00:53:15.845 --> 00:53:17.125 Claude 14, 750 00:53:17.765 --> 00:53:18.325 know, 751 00:53:20.724 --> 00:53:21.925 a loop that 752 00:53:22.329 --> 00:53:26.170 figures out the next thing to build. Here's my goal. 753 00:53:26.170 --> 00:53:29.450 Here's what I'm trying to build. Now just build the next thing 754 00:53:29.770 --> 00:53:32.490 that gets me closer to that goal in a loop. 755 00:53:32.730 --> 00:53:45.865 Do we never get there? I'm less sure about that. But even in that example, you're still the one stating the goal. The AI isn't going to be the one to tell you your goal. I mean, maybe sometimes it does, but you as a human still have agency that 756 00:53:46.265 --> 00:53:47.465 the model itself lacks. 757 00:53:48.069 --> 00:53:49.510 Okay, that's 758 00:53:49.589 --> 00:53:52.790 fair. But that is saying that either 759 00:53:53.030 --> 00:53:55.990 the CEO is the only one who's not replaceable, 760 00:53:56.150 --> 00:53:59.270 or we're all mini CEOs of our own agents. 761 00:54:00.414 --> 00:54:01.214 Yeah, I 762 00:54:02.894 --> 00:54:25.200 don't know. I don't know. I would say that that's a fair approximation of the world as it is right now, where you have people who... Large companies where you have a CEO and there people who are happy to work under them in their particular role. And then you have people who chafe at being told what to do by anyone, and they go and run their own small businesses focused on whatever it is that they want to do in the world. 763 00:54:26.385 --> 00:54:29.265 That's that's a that's a good analogy. 764 00:54:29.345 --> 00:54:29.905 Yes. 765 00:54:30.145 --> 00:54:30.945 Alright. 766 00:54:30.945 --> 00:54:33.265 Now that we've played that out, I guess, 767 00:54:33.825 --> 00:54:44.890 circling back around to what you're building, I'm curious, what are some of the most interesting or innovative or unexpected ways that you've seen the tool chain that you're currently focused on applied? So Bifrost, 768 00:54:44.890 --> 00:54:47.609 Mjolnir, and I think you're also building something called Anvil. 769 00:54:48.170 --> 00:54:53.609 Yeah, so Anvil is we've de emphasized Anvil. So Mjolnir 770 00:54:53.690 --> 00:54:56.089 is, like I said, it's a meta harness. 771 00:54:56.744 --> 00:55:00.345 And what that means in this context is 772 00:55:00.664 --> 00:55:01.385 we're 773 00:55:01.625 --> 00:55:07.385 driving codecs and clod code and kimicode and grok build and 774 00:55:07.545 --> 00:55:08.585 musecode. 775 00:55:08.744 --> 00:55:12.820 We're driving those directly from that text interface 776 00:55:12.820 --> 00:55:14.980 that Mjolnir shows the user. 777 00:55:15.619 --> 00:55:18.660 And we're driving those over a protocol called ACP. 778 00:55:18.900 --> 00:55:23.220 And ACP was created so that you could put Cloud Code into 779 00:55:23.765 --> 00:55:27.205 your IntelliJ or into your Versus Code and so forth, 780 00:55:27.525 --> 00:55:30.965 but it's still Cloud Code under the hood. It's not a separate harness. 781 00:55:31.285 --> 00:55:37.685 And so Anvil was something where we said, well, we've got these interesting ideas for the harness itself, so we'll have our own 782 00:55:38.500 --> 00:55:39.700 ACP server, 783 00:55:39.860 --> 00:55:42.740 which is what the harness's role is. But 784 00:55:43.300 --> 00:55:46.020 as the models have gotten more sophisticated 785 00:55:46.180 --> 00:55:50.580 and they're doing things like, you know, Claude code will invalidate 786 00:55:50.580 --> 00:55:51.300 individual 787 00:55:51.895 --> 00:55:52.775 chunks 788 00:55:52.935 --> 00:55:56.375 of the inference server's key value cache 789 00:55:57.015 --> 00:55:59.415 based on what's going on in your session. 790 00:55:59.735 --> 00:56:02.135 And I'm like, well, that's a level of 791 00:56:02.455 --> 00:56:09.490 integration that I don't think that I can outcompete the big labs at. So Anvil has been deemphasized. 792 00:56:10.609 --> 00:56:13.650 Do think we are adding value with Mjolnir over 793 00:56:14.290 --> 00:56:17.090 the Codex and the Clog code and so forth layer. 794 00:56:17.650 --> 00:56:26.085 And to give an example of what you asked about, what's something that's surprising or that I'm proud of, we just added a feature to Mjolnir 795 00:56:26.244 --> 00:56:27.045 where 796 00:56:27.125 --> 00:56:28.245 it will... 797 00:56:28.485 --> 00:56:30.965 So Mjolnir has a concept of workspaces 798 00:56:30.965 --> 00:56:32.165 and sessions 799 00:56:32.165 --> 00:56:39.230 belong to a workspace, and you can navigate between the sessions and see what's going on. But we added a virtual workspace 800 00:56:39.630 --> 00:56:41.310 for sub agents. 801 00:56:41.630 --> 00:56:44.350 And so you can have your 802 00:56:44.750 --> 00:56:47.790 codex sub agents and click on 803 00:56:48.670 --> 00:56:55.335 the six sub agents label, and it opens up a virtual workspace, and you can see what's going on in there. And those sub 804 00:56:55.575 --> 00:56:58.935 agents don't have to be from the same artist. So you 805 00:56:59.975 -->

00:57:08.970 can have Fable driving a Luna sub agent, which is a very economical way to use Fable because Luna is really, really inexpensive. 806 00:57:09.369 --> 00:57:11.770 Or DeepSeek Flash. I 807 00:57:12.010 --> 00:57:16.410 had Fable driving six DeepSeek Flash sub agents yesterday 808 00:57:16.490 --> 00:57:17.770 working on a problem 809 00:57:18.155 --> 00:57:18.955 because 810 00:57:19.595 --> 00:57:20.395 it's 811 00:57:20.395 --> 00:57:23.915 part of the subscription optimization game where 812 00:57:24.555 --> 00:57:33.755 my Fable quota is much more valuable than the other half of the subscription that only lets me use Opus and Sonnet in Anthropic's case. 813 00:57:35.730 --> 00:57:38.050 So on the meta harness side, 814 00:57:38.770 --> 00:57:44.370 that's what I'm excited about is kind of unlocking the value of my different subscriptions playing together. 815 00:57:44.690 --> 00:57:45.970 And then in 816 00:57:46.210 --> 00:57:47.330 the Bifrost side, 817 00:57:47.994 --> 00:57:53.195 this is going to get really deep into the technical weeds, but we've built 818 00:57:53.435 --> 00:57:55.515 the Bifrost database 819 00:57:55.835 --> 00:57:58.715 on top of Git, basically. 820 00:57:59.035 --> 00:58:00.715 The primary key 821 00:58:01.000 --> 00:58:05.240 in Bifrost database is the Git blob ID, 822 00:58:05.400 --> 00:58:08.040 so the hash of the contents 823 00:58:08.120 --> 00:58:10.600 that's in the Git index. 824 00:58:10.920 --> 00:58:13.080 And so what that unlocks 825 00:58:13.080 --> 00:58:19.995 is that if I have 12 agents working in 12 work trees and making changes simultaneously, 826 00:58:20.395 --> 00:58:23.195 I only have to re index changes 827 00:58:23.195 --> 00:58:24.395 that are 828 00:58:24.475 --> 00:58:26.155 new, that have never existed 829 00:58:27.035 --> 00:58:33.810 in any work tree, and I don't have to re index anything that has ever existed anywhere else. So creating a new work tree 830 00:58:34.370 --> 00:58:37.090 and indexing that in Bifrost is instant 831 00:58:37.970 --> 00:58:41.170 because all of those blobs have already existed. 832 00:58:41.810 --> 00:58:42.690 And then 833 00:58:43.434 --> 00:58:51.915 what that unlocks now is that's what unlocks the speed difference of, I can ask Bifrost during the coding session, 834 00:58:53.115 --> 00:58:53.675 committed 835 00:58:53.994 --> 00:58:56.315 have any concurrency crimes 836 00:58:56.510 --> 00:58:59.069 that the Bifrost policy engine catches 837 00:58:59.069 --> 00:58:59.870 rather 838 00:59:00.109 --> 00:59:00.670 than 839 00:59:00.910 --> 00:59:02.830 an hours long CI session, 840 00:59:02.990 --> 00:59:05.710 because I just have to re index those pieces 841 00:59:06.030 --> 00:59:09.869 that have actively had edits since the last time it got indexed. 842 00:59:10.595 --> 00:59:12.915 And as you have been 843 00:59:13.075 --> 00:59:15.234 working in this space and 844 00:59:15.954 --> 00:59:16.994 trying to 845 00:59:17.954 --> 00:59:30.410 discern what are the most effective ways for humans to be involved in this agent owned software delivery life cycle? What are some of the most interesting or unexpected or challenging lessons that you've learned in the process? 846 00:59:31.530 --> 00:59:33.210 I guess I'm going to 847 00:59:33.850 --> 00:59:35.050 contradict my 848 00:59:35.210 --> 00:59:36.570 talking points a little bit. 849 00:59:39.015 --> 00:59:40.295 Because one of the most 850 00:59:40.775 --> 00:59:41.655 frustrating 851 00:59:41.655 --> 00:59:44.295 and challenging parts is that, 852 00:59:44.934 --> 00:59:52.934 while I think everything we've said about how agents are so good at writing code now that we don't need to review their work is 853 00:59:53.420 --> 00:59:54.940 correct on average. 854 00:59:55.660 --> 00:59:57.820 There's always corner cases 855 00:59:57.980 --> 00:59:58.700 where 856 00:59:59.660 --> 01:00:02.060 our running joke on our Slack is, 857 01:00:02.380 --> 01:00:08.700 I guess I'm not obsolete yet, because you constantly find these places where 858 01:00:09.954 --> 01:00:11.155 Fable decided 859 01:00:11.155 --> 01:00:11.875 that 860 01:00:12.195 --> 01:00:13.795 what I really needed 861 01:00:14.035 --> 01:00:16.035 was to clone. 862 01:00:16.195 --> 01:00:18.355 So Mjolnir's written in Rust, 863 01:00:18.515 --> 01:00:22.435 and so it decided that what I really needed to do was to clone the entire 864 01:00:23.079 --> 01:00:24.119 conversation 865 01:00:24.440 --> 01:00:28.360 every time I appended a new message to that conversation. 866 01:00:28.440 --> 01:00:29.880 So it's like, well, why 867 01:00:31.240 --> 01:00:32.920 is my Mjolnir process 868 01:00:33.000 --> 01:00:34.600 spending all its time 869 01:00:35.000 --> 01:00:37.000 in memory remapping? 870 01:00:37.000 --> 01:00:38.280 Oh, it's because 871 01:00:38.655 --> 01:00:39.295 your 872 01:00:39.535 --> 01:00:40.335 allocations 873 01:00:40.415 --> 01:00:44.895 are through the roof. It's like, well, should I really need to tell Fable 874 01:00:45.055 --> 01:00:46.735 in 2026 875 01:00:46.815 --> 01:00:47.615 that 876 01:00:48.095 --> 01:00:52.175 is not a good memory allocation pattern? Like, I feel like I shouldn't, 877 01:00:52.335 --> 01:00:54.880 and yet in this case I did. So 878 01:00:55.280 --> 01:00:57.760 yeah, the edge is still jagged, 879 01:00:58.000 --> 01:00:59.360 but I do think that 880 01:00:59.920 --> 01:01:02.480 it is up and to the right over time. 881 01:01:03.360 --> 01:01:08.080 At some point, not today, at some point, we won't have to think about that anymore. 882 01:01:09.325 --> 01:01:11.165 Yeah. The models are definitely 883 01:01:11.245 --> 01:01:13.565 very adept at being confidently wrong. 884 01:01:14.205 --> 01:01:16.525 Yes. That is still true. Yes. 885 01:01:17.885 --> 01:01:18.925 And so 886 01:01:19.085 --> 01:01:20.445 as people are 887 01:01:21.010 --> 01:01:23.810 remapping their overall workflow 888 01:01:23.890 --> 01:01:28.530 and rethinking

the role of humans in the production of software, 889 01:01:29.090 --> 01:01:34.850 what are some of the things you have planned for future looking tools or 890 01:01:35.025 --> 01:01:38.305 thought pieces or experiments to help 891 01:01:38.545 --> 01:01:41.665 us as an industry move in a 892 01:01:42.145 --> 01:01:43.505 beneficial direction 893 01:01:43.585 --> 01:01:45.744 for some definition of beneficial? 894 01:01:46.385 --> 01:01:47.265 Well, 895 01:01:47.265 --> 01:01:49.424 I guess just at a really tactical level, 896 01:01:50.589 --> 01:01:52.910 one of the things that I'm frustrated 897 01:01:52.910 --> 01:01:53.790 about, again, 898 01:01:54.190 --> 01:01:56.670 is the state of model benchmarking. 899 01:01:57.390 --> 01:01:59.390 And I say again because 900 01:01:59.390 --> 01:02:02.430 in 2025, 901 01:02:03.355 --> 01:02:10.315 the state of the art for benchmarking coding models was SWE bench, the original SWE bench, which consists 902 01:02:10.875 --> 01:02:11.595 of 903 01:02:11.755 --> 01:02:13.275 100% Python 904 01:02:13.435 --> 01:02:15.035 and like 49% 905 01:02:15.035 --> 01:02:17.355 of it, I don't remember the exact number, is Django. 906 01:02:17.770 --> 01:02:18.490 So 907 01:02:18.730 --> 01:02:21.610 a high sweepage code was basically just measuring 908 01:02:22.010 --> 01:02:35.765 how well... Oh, and the problems they give the models were all based on GitHub tickets from like a ten year time span. So a high sweep in score was basically how well has your model memorized Django 909 01:02:35.845 --> 01:02:40.485 and a little bit scikit learn and a little bit a couple other Python packages. 910 01:02:41.045 --> 01:02:42.725 And so 911 01:02:43.220 --> 01:02:46.500 I wrote something called Brockbench to do more realistic 912 01:02:46.500 --> 01:02:47.780 coding challenges. 913 01:02:47.940 --> 01:02:53.140 And then at the beginning of this year, we kind of got a next generation of coding benchmarks, 914 01:02:53.940 --> 01:02:55.140 especially DeepSUI 915 01:02:55.140 --> 01:02:56.820 and CursorBench 916 01:02:57.059 --> 01:02:58.020 and 917 01:02:58.914 --> 01:02:59.875 FrontierSUI, 918 01:02:59.875 --> 01:03:02.115 I think is another one. There's 919 01:03:02.115 --> 01:03:03.555 two or three that are reasonably 920 01:03:03.954 --> 01:03:04.755 good now. 921 01:03:05.154 --> 01:03:07.714 And so I said, okay, don't have to do this anymore. 922 01:03:08.355 --> 01:03:09.315 But lately, 923 01:03:09.474 --> 01:03:11.234 with the last couple months of releases, 924 01:03:11.820 --> 01:03:14.060 DeepSUI is saturated. 925 01:03:14.060 --> 01:03:14.940 Everything's 926 01:03:14.940 --> 01:03:16.940 peaking at about 80% and 927 01:03:17.260 --> 01:03:21.820 it's telling me things like Opus five is better than Fable 5.1. 928 01:03:21.820 --> 01:03:23.980 And I know that's not true. That 929 01:03:24.140 --> 01:03:28.055 is just not true. Whatever you're measuring there is 930 01:03:28.535 --> 01:03:29.655 less useful 931 01:03:29.815 --> 01:03:34.615 than it was six months ago. And I don't know if it's because the agents or the 932 01:03:34.775 --> 01:03:37.494 labs are training on the benchmark or it's 933 01:03:37.494 --> 01:03:38.455 less useful. 934 01:03:38.694 --> 01:03:45.349 And so what I've been working on over the last couple weeks has been, I'm taking live 935 01:03:45.349 --> 01:03:46.790 Bifrost tickets 936 01:03:47.190 --> 01:03:47.910 and 937 01:03:47.990 --> 01:03:50.470 I'm doing a bake off between 938 01:03:50.710 --> 01:03:51.589 two 939 01:03:51.990 --> 01:03:53.589 or three models at a time. 940 01:03:54.150 --> 01:03:58.630 And I'm having Fable and Astra judge 941 01:03:57.875 --> 01:03:58.994 the results. 942 01:03:59.315 --> 01:04:03.395 So I'm gonna have basically a series of pair rankings across 943 01:04:03.474 --> 01:04:05.155 a couple 100 tasks 944 01:04:05.234 --> 01:04:05.955 for 945 01:04:07.234 --> 01:04:08.515 DeepSeek Flash, 946 01:04:08.915 --> 01:04:18.119 for GLM five point three and five point three Flash, for MuseSpark, for Grok Build. Basically, all the models that you might want to use when you're like, Oh, help, 947 01:04:18.200 --> 01:04:18.840 I'm 948 01:04:19.640 --> 01:04:20.520 out of my 949 01:04:21.160 --> 01:04:22.680 CLOUD subscription tokens. 950 01:04:23.125 --> 01:04:31.205 What can I use now that's not going to break the bank and is still going to help me get things done? That's basically the question that I'm trying to answer. 951 01:04:31.445 --> 01:04:32.085 So I 952 01:04:32.405 --> 01:04:33.685 should have the first 953 01:04:34.005 --> 01:04:36.965 results by the end of this week, next week at the latest. 954 01:04:37.869 --> 01:04:57.825 And are there any other aspects of the work that you're doing at Brock AI and the various tool chains that you're investing in, or just this overall question of where and how best to spend our human attention in this world of AI generated everything that we didn't discuss yet that you'd like to cover before we close out the show? 955 01:04:59.025 --> 01:05:00.385 Well, my 956 01:05:00.385 --> 01:05:05.744 goal with Brock AI is to get to that ideal point that you articulated earlier of 957 01:05:06.080 --> 01:05:09.680 we're spending our time as humans deciding what to build 958 01:05:10.080 --> 01:05:15.360 and less on checking, hey, did the model actually build what I told it to do? 959 01:05:15.680 --> 01:05:17.760 And that's what we're trying to build with Brock. 960 01:05:19.045 --> 01:05:35.260 All right. Well, for anybody who wants to get in touch with you and follow along with the work that you and your team are doing, I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get your perspective on what you see as being the biggest gaps in the tooling, technology, or human training that's available for building AI systems today? 961 01:05:36.220 --> 01:05:38.940 Biggest gap? I think the biggest gap is 962 01:05:39.819 --> 01:05:41.579 agent oriented collaboration 963 01:05:41.579 --> 01:05:45.900 tools. So we've got GitHub that's oriented towards humans. 964 01:05:46.140 --> 01:05:48.460 We've got Linear that's 965 01:05:48.905 --> 01:05:55.705 also oriented towards humans and maybe doing a faster job of adapting to agents than GitHub is. 966 01:05:55.945 --> 01:05:59.225 But what I would like is a world where 967 01:05:59.385 --> 01:06:00.825 when I file a ticket, 968 01:06:02.380 --> 01:06:15.580 something in the system can say, Oh, this is related to code that agent three forty five wrote last week. Let me revive that session on the appropriate harness and tell it, Hey, you caused a regression. 969 01:06:15.915 --> 01:06:19.515 So it's got all the context there. It doesn't need to rediscover it. 970 01:06:19.675 --> 01:06:23.435 And just wire those pieces of the life cycle together. 971 01:06:24.155 --> 01:06:25.035 If somebody 972 01:06:25.115 --> 01:06:27.355 had that, I would be a customer tomorrow. 973 01:06:28.809 --> 01:06:36.250 All right. Well, thank you very much for taking the time today to join me and share your thoughts and opinions and expertise 974 01:06:36.250 --> 01:06:38.970 on this fast moving space of 975 01:06:39.130 --> 01:06:45.345 what do humans do now. So, thank you for all the time and energy you're putting into that, and I hope you enjoy the rest of your day. 976 01:06:45.825 --> 01:06:47.345 Thanks, Tobias. It's been a pleasure. 977 01:06:55.100 --> 01:06:59.580 Thank you for listening, and don't forget to check out our other shows. Podcast.net 978 01:06:59.580 --> 01:07:08.460 covers the Python language, its community, and the innovative ways it is being used. And the AI engineering podcast is your guide to the fast moving world of building AI systems. 979 01:07:09.325 --> 01:07:19.325 Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a project from the show, then tell us about it. Email hosts@dataengineeringpodcast.com 980 01:07:19.325 --> 01:07:25.565 with your story. Just to help other people find the show, please leave a review on Apple podcasts and tell your friends and coworkers.