What Do Humans Do Now? Rethinking Software Engineering in the Age of AI Agents

WEBVTT
NOTE
Transcription provided by Podhome.fm
Created: 10/15/2026 23:44:12
Duration: 4055.205
Channels: 1
1
00:00:11.440 --> 00:00:15.440
Hello, and welcome to the Data Engineering Podcast, the show about modern data management.
2
00:00:16.155 --> 00:00:25.755
Your host is Tobias Maci, and today I'm interviewing Jonathan Ellis about how to make the best use of human time when agents are building the software. So, Jonathan, can you start by introducing yourself?
3
00:00:26.555 --> 00:00:30.490
Absolutely. I'm Jonathan Ellis. I started DataStax
4
00:00:30.490 --> 00:00:33.530
fifteen years ago to commercialize Apache Cassandra.
5
00:00:33.690 --> 00:00:37.130
Spent a long time in the space of distributed databases
6
00:00:37.130 --> 00:00:38.730
and the JVM.
7
00:00:39.129 --> 00:00:40.970
And now I'm
8
00:00:40.970 --> 00:00:43.850
running a small AI startup called Brock AI,
9
00:00:44.625 --> 00:00:54.945
where I'm doing neither of those things and writing native code in Rust through agents. And do you remember how you first got started working in the ML and AI space?
10
00:00:55.984 --> 00:00:56.704
Basically,
11
00:00:57.745 --> 00:00:58.945
well, so I guess
12
00:00:59.570 --> 00:01:05.890
to back way, way up, I've always been attracted to the idea that as engineers,
13
00:01:05.970 --> 00:01:12.370
we should be comfortable building our own tools and automating our own jobs and making ourselves more productive,
14
00:01:12.945 --> 00:01:16.065
which is a philosophy that goes all the way back to
15
00:01:17.025 --> 00:01:20.625
Richard Stallman and Emacs and the people before him.
16
00:01:21.025 --> 00:01:21.585
So
17
00:01:21.985 --> 00:01:27.025
one of the coolest things for me is how AI unlocks that, whereas
18
00:01:27.360 --> 00:01:32.160
five years ago, I would have said, Oh man, I'm spending my time in IntelliJ,
19
00:01:32.160 --> 00:01:32.800
and
20
00:01:33.120 --> 00:01:38.800
it's, what is it, 5,000,000 lines of code? It's a ridiculously large project, very intimidating
21
00:01:38.880 --> 00:01:40.560
to make contributions to.
22
00:01:41.315 --> 00:01:44.915
And now I can say, hey, I really wish this feature existed
23
00:01:45.155 --> 00:01:47.395
in IntelliJ. Go and
24
00:01:47.955 --> 00:01:51.075
code dive it, figure it out, come back. Usually,
25
00:01:51.235 --> 00:02:00.790
I think, and this comes back to our topic today, I don't think the right approach is to say, just go build this, but rather come back to me and give me the top three options
26
00:02:00.950 --> 00:02:09.350
and let's discuss the pros and cons and go from there. But any tool that I use day to day now, I can enhance
27
00:02:09.675 --> 00:02:11.195
extend using AI
28
00:02:11.435 --> 00:02:13.915
in a way that wouldn't have made sense
29
00:02:14.235 --> 00:02:17.115
from a time investment perspective before
30
00:02:17.115 --> 00:02:18.075
the advent
31
00:02:18.235 --> 00:02:19.435
of GenAI.
32
00:02:20.235 --> 00:02:20.715
So
33
00:02:21.195 --> 00:02:28.910
Brock is basically an extension of that where I said, Hey, there's a bunch of tools that I think should exist and nobody's building them, so I'm going to go build them.
34
00:02:29.709 --> 00:02:31.870
Yeah, it's definitely pretty remarkable,
35
00:02:31.870 --> 00:02:32.430
the
36
00:02:32.829 --> 00:02:40.035
unlock that has happened because of the speed at which you can go from idea to delivery or at least prototype.
37
00:02:40.355 --> 00:02:40.755
And
38
00:02:41.155 --> 00:02:41.875
the
39
00:02:42.435 --> 00:02:44.275
ways that that manifests
40
00:02:44.275 --> 00:02:50.115
in terms of an organizational context where you're working in a professional engineering team can have
41
00:02:50.435 --> 00:02:58.740
a lot of knock on effects in terms of what people are doing and how they spend their time, particularly if the engineering team predates
42
00:02:58.819 --> 00:03:02.180
the past two years. So people who have built up a
43
00:03:02.420 --> 00:03:04.819
certain set of workflows and expectations
44
00:03:05.555 --> 00:03:10.275
largely around code review, which is part of what we're here to talk about. And so
45
00:03:10.355 --> 00:03:26.240
because of the fact that writing the code itself is no longer the bottleneck that shifts the bottlenecks everywhere else, And I'm wondering how you're seeing that impact the ways that engineering teams think about their job and the work that they do.
46
00:03:27.360 --> 00:03:29.840
Yeah, and I think this is a place where
47
00:03:30.875 --> 00:03:34.955
collectively as an industry, we have more questions than answers.
48
00:03:35.114 --> 00:03:35.595
And,
49
00:03:36.474 --> 00:03:38.795
you know, I certainly have opinions,
50
00:03:38.875 --> 00:03:41.355
and I think we're starting to stumble towards
51
00:03:41.515 --> 00:03:42.795
some of those answers.
52
00:03:43.034 --> 00:03:45.915
But really, I think what happened and how we got here is,
53
00:03:46.610 --> 00:03:47.570
at first,
54
00:03:47.970 --> 00:03:51.490
AI was accelerating the act of creating code
55
00:03:52.370 --> 00:03:55.410
incrementally more and more with every new model release,
56
00:03:55.650 --> 00:03:57.890
but everyone was still just kind of
57
00:03:58.450 --> 00:04:05.645
slotting that in to the existing teams and processes. And it's like, Oh, great. Jonathan's faster at writing code than he used to be.
58
00:04:06.685 --> 00:04:12.205
That's great, and we don't need to change anything else, except maybe he tackles, he goes through the backlog faster.
59
00:04:12.525 --> 00:04:19.860
But now we're starting to see, we've gone from, Oh, it makes code 50% faster, or it makes it 100%
60
00:04:19.860 --> 00:04:24.660
faster or 200% faster. Now it really is 10x faster now,
61
00:04:24.980 --> 00:04:27.380
and it's still getting even faster still
62
00:04:28.660 --> 00:04:37.635
because as the new models come out now, the number of tokens per second they're generating hasn't gotten faster for the most part. There's some exceptions.
63
00:04:37.955 --> 00:04:43.875
But the task size that I can give it, like that's what's changed. So I can have
64
00:04:44.195 --> 00:04:46.595
12 tasks going now with Fable
65
00:04:47.380 --> 00:04:50.900
and they're each taking multiple hours, sometimes multiple days.
66
00:04:51.300 --> 00:04:53.700
And that's what accelerates
67
00:04:54.020 --> 00:04:58.340
the pace of code generation now is being able to parallelize that
68
00:04:59.060 --> 00:05:03.445
with less oversight, Not zero oversight, but with less oversight.
69
00:05:03.604 --> 00:05:13.044
And so that's what's just completely broken, that old paradigm of we're just going to keep using these human oriented tools and human oriented processes.
70
00:05:13.685 --> 00:05:16.085
And so what comes after that? What comes
71
00:05:16.349 --> 00:05:25.150
now that code is 10 times faster than it was? How do we fit the rest of the software development life cycle around that new
72
00:05:25.550 --> 00:05:26.509
reality?
73
00:05:26.590 --> 00:05:27.229
And
74
00:05:27.870 --> 00:05:31.710
we can kick it off or we can set more stage if you want, but
75
00:05:32.405 --> 00:05:45.044
one of the first things to go, I believe, is the concept of humans spending time reading through diffs and doing code review the way we did, you know, three years ago, even one year ago,
76
00:05:45.604 --> 00:05:48.720
because that's just not a good use of time. And
77
00:05:48.880 --> 00:05:51.120
I can flesh out my
78
00:05:51.440 --> 00:05:54.720
thesis a little bit more, or you can jump in first.
79
00:05:55.120 --> 00:05:56.240
Yeah. So
80
00:05:56.640 --> 00:05:59.760
just as a quick reference for listeners,
81
00:05:59.840 --> 00:06:07.025
you wrote this blog post about the fact that it's time to stop performing code review. I'll add a link in the show notes. And I think that
82
00:06:07.425 --> 00:06:12.465
one of the other interesting elements of this point in time is that,
83
00:06:12.865 --> 00:06:15.745
particularly for people who have been in the industry for
84
00:06:16.120 --> 00:06:17.800
more than two or three years,
85
00:06:18.360 --> 00:06:29.479
they have developed a certain amount of their identity and ego around the code that they write and the way that they approach code and their various competing camps of what constitutes best practice or idiomatic
86
00:06:29.565 --> 00:06:33.405
practices for different languages or frameworks or ecosystems.
87
00:06:34.205 --> 00:06:34.685
And
88
00:06:35.085 --> 00:06:40.925
so one of the points of contention that I've seen in my own experience is people
89
00:06:41.165 --> 00:07:10.595
saying, well, the code that the AI is writing isn't the way that I would have written it, so therefore it's not good. Or it's ignoring this idiom that I have spent a lot of time perfecting where maybe instead of using a bunch of if then blocks, I put it into a dictionary so that it's more of a direct access, or maybe I prefer a more functional style versus imperative style. So there's a lot of these details that, at the end of the day, really don't matter except for the case where maybe they have a performance impact.
90
00:07:11.074 --> 00:07:11.715
And
91
00:07:12.180 --> 00:07:14.660
so that, I think, is another piece where
92
00:07:15.700 --> 00:07:18.260
the human element of engineering
93
00:07:18.340 --> 00:07:19.140
is
94
00:07:19.940 --> 00:07:25.540
coming under fire by the AI because you have to just let go of that fact of, okay. Well,
95
00:07:26.665 --> 00:07:42.185
maybe I didn't even read the code. As long as the code does what I want it to do, that's the thing I care about the most. And, you know, there's the trope that nobody's paying for the software. They're paying for what the software does. And so why do we as engineers spend so much of our time bike shedding or yak shaving about
96
00:07:42.560 --> 00:07:47.680
the way that you write your for loop versus does it do the thing that the customer actually wants?
97
00:07:48.800 --> 00:07:50.560
I think that's
98
00:07:50.560 --> 00:07:57.759
an excellent point and it highlights one of the gaps or the differences in approaches that I've seen
99
00:07:58.035 --> 00:08:02.835
in the talented engineers that I've worked with when it comes to AI, because
100
00:08:03.235 --> 00:08:13.235
I think it's much easier to come to terms with the things that you're talking about for people who have managed or led a team
101
00:08:13.500 --> 00:08:15.260
of human engineers
102
00:08:15.340 --> 00:08:16.780
because it's,
103
00:08:17.660 --> 00:08:28.615
to some degree, like, there's differences for sure. And I'm definitely not saying that managing a team of AI agents is the same as managing a human team because it's not. But it is similar in
104
00:08:28.855 --> 00:08:29.975
this one respect,
105
00:08:30.455 --> 00:08:36.455
which is that if you have a team of human engineers and your code reviews take the form of saying,
106
00:08:36.775 --> 00:08:44.470
I disagree with how you wrote this switch statement instead of putting it in a dictionary and matching on that. Like,
107
00:08:44.790 --> 00:08:47.990
that's not productive use of code review time.
108
00:08:48.310 --> 00:08:48.870
And
109
00:08:51.029 --> 00:08:52.550
it just isn't. And so
110
00:08:53.525 --> 00:08:58.325
I think that people who have been in the position of running those teams
111
00:08:58.725 --> 00:09:00.485
have gone through that
112
00:09:01.045 --> 00:09:02.005
adjustment
113
00:09:02.085 --> 00:09:02.725
of
114
00:09:04.165 --> 00:09:05.765
coming to terms with
115
00:09:06.720 --> 00:09:14.560
whether the code is functionally correct is more important than whether it follows the idioms that are my preferred weapons of choice.
116
00:09:16.400 --> 00:09:17.200
The other
117
00:09:17.760 --> 00:09:24.615
way that I've seen this framed a lot in various blog posts and conversations in the community is that
118
00:09:25.575 --> 00:09:28.215
people are no longer just individual contributors.
119
00:09:28.215 --> 00:09:34.535
Everybody has to become a manager, but instead of managing people, they're managing agents. And so you really need to
120
00:09:35.210 --> 00:09:42.890
bring your level of abstraction up to the point of the product or the feature and not the lines of code or the modules.
121
00:09:43.290 --> 00:09:43.850
And
122
00:09:44.250 --> 00:09:50.975
I think that one way that I've seen this play out is that teams who do have a strong culture of
123
00:09:51.055 --> 00:09:59.775
doing RFCs and architecture decision records as the initial step along the path of actually getting to that feature delivery
124
00:10:00.175 --> 00:10:13.110
are better situated to take advantage of the current landscape that we're in. Because once you have that RFC or the ADR or the spec that details, this is the thing that we actually want to build,
125
00:10:13.750 --> 00:10:15.430
then the implementation
126
00:10:15.510 --> 00:10:27.725
becomes automatic because the agent has that shared agreed upon context. And I think that the point of review just needs to shift earlier in the process where you need to get more people involved in that
127
00:10:28.445 --> 00:10:33.884
design and debate of what is the thing that we're actually trying to build and why versus how.
128
00:10:34.620 --> 00:10:37.100
And so as long as the
129
00:10:37.420 --> 00:10:38.940
code that lands
130
00:10:39.100 --> 00:10:43.660
can be tied back to and validated against the original design document,
131
00:10:43.660 --> 00:10:45.820
then the human review on the code
132
00:10:46.475 --> 00:10:50.475
doesn't need to happen as you just need to be able to have a way of
133
00:10:51.115 --> 00:10:55.835
doing that validation to show that it aligns with the stated goals.
134
00:10:57.515 --> 00:11:01.510
Two thoughts about that. So first, on the general direction,
135
00:11:01.589 --> 00:11:05.030
yeah, we're all product managers now, right? Or
136
00:11:06.149 --> 00:11:09.110
I think that we're being pushed in the direction
137
00:11:09.430 --> 00:11:15.625
where the value we add as humans is less on how do we implement this
138
00:11:16.185 --> 00:11:16.985
decision
139
00:11:17.385 --> 00:11:18.185
versus
140
00:11:18.425 --> 00:11:22.345
what decisions should we make? What is important to implement next?
141
00:11:22.825 --> 00:11:25.625
And so I think that developers
142
00:11:25.625 --> 00:11:27.785
who worked in an organization where there was
143
00:11:28.780 --> 00:11:29.660
a
144
00:11:29.660 --> 00:11:32.300
very stark line between those two groups
145
00:11:32.540 --> 00:11:38.540
and the engineers just implemented what another team was handing down as the decisions,
146
00:11:39.260 --> 00:11:40.860
I think that's going to be,
147
00:11:43.265 --> 00:11:48.305
the time where that's a useful use of human time is drawing to a close.
148
00:11:49.025 --> 00:11:52.065
It's not completely gone, but it's limited.
149
00:11:52.465 --> 00:11:54.305
And there's
150
00:11:54.305 --> 00:11:59.889
a line of sight to where it's like, yeah, you leave that to the agents. That's just what you do.
151
00:12:00.370 --> 00:12:04.130
So I think that teams that already had
152
00:12:04.290 --> 00:12:08.450
more of a porous boundary between engineer and product manager,
153
00:12:08.529 --> 00:12:11.329
and the product managers are now
154
00:12:12.445 --> 00:12:16.685
implementing proof of concepts with their own agents and the engineers are participating
155
00:12:16.685 --> 00:12:18.125
in design discussions,
156
00:12:18.605 --> 00:12:23.085
I think that's a much healthier place, and they're going to adapt a lot better.
157
00:12:24.940 --> 00:12:25.660
So
158
00:12:25.820 --> 00:12:28.460
spending a bit more time on the
159
00:12:29.020 --> 00:12:29.900
review,
160
00:12:30.380 --> 00:12:34.620
validation, and delivery phase of the software life cycle,
161
00:12:35.100 --> 00:12:37.020
there has been a lot of
162
00:12:38.265 --> 00:12:50.425
stake put in the role of the human in that where, oh, you need to have a human looking at the code before it can go to production because the humans are the ones that know whether it's going to work or not or have that broader context,
163
00:12:51.340 --> 00:12:53.740
which even before AI
164
00:12:53.820 --> 00:13:00.700
increased the volume of code wasn't always true. Because if you have somebody do a code review who hasn't worked in that code base before,
165
00:13:00.940 --> 00:13:04.300
that's not going to be very helpful unless the purpose is purely for
166
00:13:04.855 --> 00:13:07.895
educating them on what that code is doing and how.
167
00:13:08.695 --> 00:13:22.180
But even if you have somebody who is expert in that particular code base, they're liable to miss things. Or if they are the expert in the code base, they're likely already busy with their own implementation and don't want to have to spend those extra cycles on the code review.
168
00:13:23.540 --> 00:13:25.620
But there's still the element of
169
00:13:25.939 --> 00:13:40.765
some sort of validation and guardrails need to happen so that we can have confidence in the code as it is written before we put it into production because we don't wanna be dealing with constant firefighting due to buggy and broken code. And so I'm curious how you're
170
00:13:40.845 --> 00:13:47.630
seeing teams address that angle of what are those checks and guardrails that we need to put in place so that we can
171
00:13:48.510 --> 00:13:51.710
confidently remove humans from that phase of the life cycle.
172
00:13:52.990 --> 00:13:53.710
So
173
00:13:54.350 --> 00:13:55.630
this is where
174
00:13:56.110 --> 00:13:58.110
that article that I wrote that you mentioned
175
00:13:59.995 --> 00:14:01.515
becomes controversial
176
00:14:01.755 --> 00:14:02.715
because
177
00:14:03.275 --> 00:14:04.315
for reasons
178
00:14:04.875 --> 00:14:07.115
that I don't completely understand,
179
00:14:07.435 --> 00:14:09.595
when I say like I did just now,
180
00:14:10.475 --> 00:14:14.955
year from now or two years from now, humans just won't be writing code by hand anymore,
181
00:14:15.600 --> 00:14:17.520
Most people look at the
182
00:14:18.320 --> 00:14:32.015
curve that we've been writing for the last two years and say, Yeah, I can see how if I draw that line out a little bit more, that's probably what's going to happen. But then when I say, And you shouldn't be reviewing it either, and then they're like, Oh, like just heads exploding.
183
00:14:32.175 --> 00:14:34.735
People feel very, very strongly that,
184
00:14:35.535 --> 00:14:40.575
like you said, code should not be pushed to production without a human reading it first.
185
00:14:40.895 --> 00:14:41.535
And
186
00:14:42.095 --> 00:14:45.935
part of the research that I did for that article was
187
00:14:46.589 --> 00:14:49.630
looking at, if you actually study
188
00:14:49.630 --> 00:14:53.709
this and measure it, how many defects
189
00:14:54.509 --> 00:14:56.670
does human code review find?
190
00:14:57.709 --> 00:15:01.389
And the answer is typically about 30% of the defects.
191
00:15:02.055 --> 00:15:02.695
And
192
00:15:03.015 --> 00:15:06.535
if it's a more complicated problem domain like security,
193
00:15:06.775 --> 00:15:08.295
it's more like 20%.
194
00:15:08.695 --> 00:15:14.535
So humans aren't really good at doing code review from a standpoint of,
195
00:15:15.570 --> 00:15:16.210
this,
196
00:15:16.770 --> 00:15:18.370
am I finding the defects?
197
00:15:19.490 --> 00:15:28.464
And so I think the reason why people have that reaction of, I couldn't even dream of pushing this without a human review,
198
00:15:28.625 --> 00:15:36.225
is that we all have memories of the times where we did find a defect. And so this is part of like, our human ape brains
199
00:15:36.785 --> 00:15:38.945
are not evolved to be
200
00:15:39.264 --> 00:15:40.464
statistically
201
00:15:40.704 --> 00:15:42.520
literate. Like, really
202
00:15:42.920 --> 00:15:44.280
good at remembering
203
00:15:45.880 --> 00:15:47.720
exciting things that happened
204
00:15:47.880 --> 00:15:49.400
once or twice,
205
00:15:49.720 --> 00:15:51.720
but really bad at estimating,
206
00:15:51.720 --> 00:15:53.800
like, was this a good use of time
207
00:15:53.975 --> 00:15:59.895
to find those defects given all the other times that I did review and didn't find defects?
208
00:16:00.215 --> 00:16:01.575
So my
209
00:16:02.695 --> 00:16:04.775
standpoint is not
210
00:16:04.935 --> 00:16:12.800
that code review doesn't add value. I'm not saying that because, like I said, I can remember times I found bugs as much as anyone can.
211
00:16:13.120 --> 00:16:15.120
But my standpoint is that
212
00:16:15.680 --> 00:16:31.834
the time invested to find those defects is no longer worth it and should be spent. There are higher value ways to spend that time now. Even if I'm saying I need to spend 30% of my time or 50% or 80% of my time doing
213
00:16:32.235 --> 00:16:33.755
quality assurance,
214
00:16:33.755 --> 00:16:39.675
or in the literal sense of the word that I'm assuring the quality of the code, that it does what it's supposed to do,
215
00:16:40.380 --> 00:16:44.380
Whatever number of time you think you should be spending on that
216
00:16:44.700 --> 00:16:45.900
class of problem,
217
00:16:46.460 --> 00:16:49.740
there's better options than reading the code by hand.
218
00:16:51.100 --> 00:16:53.260
Yeah, just anecdotally,
219
00:16:53.580 --> 00:16:54.140
I'll
220
00:16:54.504 --> 00:16:56.824
call out that I've actually got
221
00:16:57.144 --> 00:16:59.704
at least one production system.
222
00:16:59.785 --> 00:17:06.825
It's not customer facing, but it is team facing that I have written entirely by AI and haven't read
223
00:17:07.360 --> 00:17:08.959
possibly any of the code,
224
00:17:09.200 --> 00:17:12.799
and it's been very useful and helpful. It's actually a
225
00:17:12.960 --> 00:17:20.399
project that manages the team memories and task and project graphs for coding agents so that it's a shared
226
00:17:21.365 --> 00:17:35.605
flat graph that everybody can access and read and write to for being able to break down the context into chunks so that you don't have to have one session that runs for three days because you don't want to lose all the stuff that you put in at the beginning
227
00:17:36.040 --> 00:17:39.880
and also be able to have the memory be
228
00:17:40.280 --> 00:17:47.320
compatible across different agent harnesses rather than just a random markdown file somewhere on your disk that you hope doesn't disappear sometime.
229
00:17:47.720 --> 00:17:58.405
And I've bypassed a lot of the human review element of it as well because it works. I use it. I make changes. I make sure that the changes do what I want them to do, and then I ship it.
230
00:17:58.885 --> 00:18:00.005
Exactly.
231
00:18:00.085 --> 00:18:02.840
Yeah. And I think that's kind of like the
232
00:18:03.720 --> 00:18:06.840
easy case, right? And I don't mean that disparagingly,
233
00:18:06.840 --> 00:18:10.359
but in the sense that if I'm building an internal tool
234
00:18:10.600 --> 00:18:13.480
for a small audience where
235
00:18:13.835 --> 00:18:17.195
the world doesn't end if there's a bug,
236
00:18:17.595 --> 00:18:19.914
then yeah, just ship it and
237
00:18:20.475 --> 00:18:23.115
use it and fix problems when you find them.
238
00:18:23.675 --> 00:18:28.155
Then the more interesting question is, how do you do
239
00:18:29.320 --> 00:18:31.080
that situation
240
00:18:31.080 --> 00:18:37.400
where the world does end when you ship a bug? Or like there's potentially very bad consequences
241
00:18:38.600 --> 00:18:42.040
if you ship buggy code. And then how do we do that now?
242
00:18:43.785 --> 00:18:47.705
And I think that a healthy way to start that discussion
243
00:18:47.785 --> 00:18:49.705
is from what I just said that
244
00:18:49.945 --> 00:18:54.265
actually you weren't catching all of the bugs by reading the code before anyway.
245
00:18:54.905 --> 00:18:57.625
And therefore, what does that imply for
246
00:18:57.960 --> 00:18:59.639
what we should be doing
247
00:19:00.200 --> 00:19:03.639
with that time that we're no longer spending reading the code?
248
00:19:05.000 --> 00:19:09.960
And I think that one of the ways that we can spend that time is by
249
00:19:10.039 --> 00:19:15.245
getting the agent to help with building those quality gates and validation
250
00:19:15.245 --> 00:19:19.245
loops so that you can catch those errors early and often,
251
00:19:19.565 --> 00:19:22.125
preferably even before it gets to a PR,
252
00:19:22.285 --> 00:19:23.644
where I think maybe that
253
00:19:23.885 --> 00:19:43.625
ties us into some of the work that you're doing, where I know that the Bifrost tool is a linter and MCP server that gives a more comprehensive view of the code structure so the agent can have that insight rather than having to rely on a thousand GREP calls to find a particular symbol. I'd
254
00:19:43.625 --> 00:19:50.825
be interested in hearing some of what you're investing time in on that angle and also some of the other ways that you're seeing teams
255
00:19:50.985 --> 00:19:52.104
use the
256
00:19:52.345 --> 00:19:54.585
accelerated pace of delivery to
257
00:19:55.100 --> 00:19:57.500
put more of that work into the
258
00:19:58.380 --> 00:20:00.940
CI systems and guardrails and
259
00:20:01.660 --> 00:20:06.060
validation cycles that we should have been building all along, but deprioritized
260
00:20:06.060 --> 00:20:09.500
because it took too much time off of the actual product that we were trying to ship?
261
00:20:11.174 --> 00:20:12.215
Yeah. So,
262
00:20:13.975 --> 00:20:18.374
as you said, I'm building a tool called Bifrost that
263
00:20:18.695 --> 00:20:19.335
is...
264
00:20:19.895 --> 00:20:23.095
The way I think of it is it's basically Coverity,
265
00:20:23.470 --> 00:20:27.310
which is a static analysis tool that's been around since,
266
00:20:27.550 --> 00:20:29.710
man, I want to say like 1996.
267
00:20:29.710 --> 00:20:31.230
Like, it's pretty old,
268
00:20:31.550 --> 00:20:33.390
but it's
269
00:20:34.030 --> 00:20:38.350
it's really powerful and it's really slow and it's really difficult to extend.
270
00:20:39.095 --> 00:20:45.815
But what it can do is find things like a use after free, or it can find certain classes of concurrency
271
00:20:45.815 --> 00:20:46.775
deadlocks
272
00:20:46.775 --> 00:20:48.455
by analyzing
273
00:20:48.455 --> 00:20:51.015
the flow of data through your program.
274
00:20:51.495 --> 00:20:53.895
And so I want to bring that
275
00:20:54.270 --> 00:20:55.789
caliber of
276
00:20:57.070 --> 00:20:58.030
deterministic
277
00:20:58.030 --> 00:20:59.070
analysis
278
00:20:59.310 --> 00:21:01.150
into agent harnesses,
279
00:21:01.150 --> 00:21:14.755
where it's not like, let's push this and let CI crank on it for a couple hours and see what it comes back with. But rather, the agent makes a tool call that says, run the use after free checks, and it gets back an answer in milliseconds
280
00:21:15.315 --> 00:21:17.315
instead of minutes or hours.
281
00:21:17.635 --> 00:21:18.995
And so to
282
00:21:20.150 --> 00:21:21.669
do that, to make
283
00:21:22.150 --> 00:21:27.109
Bifrost itself, which, so we're tackling 13 languages at once. And I
284
00:21:27.590 --> 00:21:34.230
I think, you know, there's other tools out there, but I think we're probably the first one to try to do so many languages all at once.
285
00:21:35.125 --> 00:21:36.644
And obviously,
286
00:21:36.645 --> 00:21:38.244
AI makes that possible
287
00:21:38.485 --> 00:21:40.005
with the word parallelization
288
00:21:40.005 --> 00:21:41.044
that I mentioned,
289
00:21:41.285 --> 00:21:42.804
but guaranteeing
290
00:21:42.885 --> 00:21:46.725
correctness is actually still really hard. There's so many corner cases
291
00:21:46.885 --> 00:21:51.250
when you have so many languages that you're dealing with. And so the
292
00:21:51.250 --> 00:21:59.890
big technique or the big idea that I orient my thinking around is I gave a talk not long ago where I talked about
293
00:22:00.610 --> 00:22:03.250
you should be trying to build an oracle
294
00:22:03.965 --> 00:22:07.965
for your agents, and you should use agents to build that oracle.
295
00:22:08.365 --> 00:22:15.485
And of course, an oracle is a term from AI engineering where we're talking about something that knows the right answer
296
00:22:16.039 --> 00:22:19.159
for a certain class of problem. And it's usually because
297
00:22:19.240 --> 00:22:20.520
we've constructed
298
00:22:20.520 --> 00:22:24.039
the problem in such a way that we know the answer
299
00:22:24.440 --> 00:22:31.000
upfront or we can derive the answer much more easily than we can in the fully general problem space.
300
00:22:31.795 --> 00:22:35.155
And so categories of tools that the
301
00:22:35.155 --> 00:22:36.835
QA engineering
302
00:22:36.835 --> 00:22:40.355
space knows about include things like fuzz testing,
303
00:22:40.835 --> 00:22:43.475
includes things like property checkers.
304
00:22:44.115 --> 00:22:44.835
And so
305
00:22:45.610 --> 00:22:47.610
we've built for Bifrost
306
00:22:47.610 --> 00:22:48.889
all of those things.
307
00:22:50.330 --> 00:22:51.849
And we use that to
308
00:22:52.250 --> 00:22:52.969
let
309
00:22:53.289 --> 00:22:54.249
the agents
310
00:22:54.409 --> 00:22:55.369
triage
311
00:22:55.529 --> 00:23:01.445
the bugs they find. So that's the other thing that agents unlock, by the way, is that a lot of these techniques,
312
00:23:01.525 --> 00:23:03.445
like fuzz testing isn't new and
313
00:23:04.245 --> 00:23:05.125
mutation
314
00:23:05.125 --> 00:23:06.485
testing isn't new.
315
00:23:06.805 --> 00:23:10.965
And all of these techniques that we use, none of these are new. I'm not that creative.
316
00:23:11.285 --> 00:23:13.285
But what's new is that
317
00:23:14.480 --> 00:23:15.280
before
318
00:23:16.240 --> 00:23:17.040
AI,
319
00:23:17.440 --> 00:23:18.400
if you had
320
00:23:19.840 --> 00:23:23.600
a sonar cube or a find bugs or a co verity
321
00:23:23.760 --> 00:23:27.040
or any of these static analysis checking
322
00:23:27.040 --> 00:23:27.600
reports,
323
00:23:28.385 --> 00:23:31.425
like they generate tons of false positives.
324
00:23:31.905 --> 00:23:32.465
And
325
00:23:32.705 --> 00:23:36.545
that's just the nature of the problem space.
326
00:23:37.025 --> 00:23:38.785
And so most teams
327
00:23:39.185 --> 00:23:40.145
didn't bother
328
00:23:40.399 --> 00:23:42.799
unless they were in an extremely
329
00:23:43.600 --> 00:23:45.519
sensitive space where
330
00:23:45.600 --> 00:23:47.999
releasing bugs is just catastrophic,
331
00:23:48.240 --> 00:23:49.839
in which case, like the
332
00:23:50.399 --> 00:23:51.279
investment
333
00:23:51.360 --> 00:23:55.440
of time to use these tools would pay off. But for most
334
00:23:56.085 --> 00:24:00.164
tools, and even at my last company, even at Data Stacks where we're building
335
00:24:00.405 --> 00:24:03.364
Apache Cassandra used by companies like
336
00:24:03.605 --> 00:24:14.159
Apple and Netflix and all of these other companies to store data that they cared very deeply about, Even there, we only made minimal use of this kind of static analysis because like
337
00:24:14.720 --> 00:24:18.799
I said, there's just so much false positives. But now you can just tell
338
00:24:19.760 --> 00:24:24.475
Opus or Soul or your model of choice, go through this report
339
00:24:24.635 --> 00:24:25.355
and
340
00:24:25.515 --> 00:24:30.554
throw out the false positives and create tickets for the ones that we need to fix. That
341
00:24:30.635 --> 00:24:32.715
unlocks so many of these
342
00:24:33.275 --> 00:24:35.355
things that software
343
00:24:35.355 --> 00:24:46.340
engineering researchers have come up with over the last twenty years and nobody could use because we didn't have the time to implement them or the effort didn't pay off. But now
344
00:24:47.220 --> 00:24:50.419
code isn't the only thing that's free anymore or very close to free.
345
00:24:51.060 --> 00:24:51.620
Reviewing
346
00:24:52.275 --> 00:24:55.155
the outputs of these tools is also free now.
347
00:24:56.675 --> 00:24:58.595
And so now that
348
00:24:58.915 --> 00:25:01.715
so much of the work can be automated,
349
00:25:03.475 --> 00:25:12.770
there's always the problem of running around with a hammer and trying to hit everything that might be a nail. And so I'm wondering what are some of the ways that you are seeing the
350
00:25:13.410 --> 00:25:20.050
best return on investment for human time spent and how... What are some of the useful heuristics around
351
00:25:20.895 --> 00:25:30.015
what are the places that the AI is going to be most useful and most effective versus the time where I need to use my own human judgment and reasoning,
352
00:25:30.095 --> 00:25:41.500
which also potentially segues us into the conversation of how do we help people build up that knowledge and reasoning if they're early in their career and haven't gone through all of the painful
353
00:25:41.500 --> 00:25:45.420
learning paths of people who have been in the industry prior to AI.
354
00:25:46.815 --> 00:25:48.335
Those are two
355
00:25:49.135 --> 00:25:52.174
very different topics, but both of them interesting.
356
00:25:52.255 --> 00:25:53.854
So on the first one,
357
00:25:54.174 --> 00:25:57.374
how do you decide where to spend your time? I think...
358
00:25:57.775 --> 00:26:00.654
So for me, the best use of time has been,
359
00:26:01.135 --> 00:26:02.950
as I mentioned, building
360
00:26:03.110 --> 00:26:04.469
more oracles
361
00:26:04.470 --> 00:26:08.149
and building the harnesses to drive those.
362
00:26:10.710 --> 00:26:15.349
Like I said, I'm not a super creative guy. So I just go to
363
00:26:15.669 --> 00:26:17.110
Chad GPT or Claude,
364
00:26:17.625 --> 00:26:19.144
and I say, the
365
00:26:19.945 --> 00:26:24.985
way I would start with a new project is I would say, Hey, what are some of the most interesting
366
00:26:25.065 --> 00:26:36.399
research approaches from the last twenty years of software development or software engineering research that I could apply to this project and order them by your expected cost benefit
367
00:26:36.559 --> 00:26:37.440
and then
368
00:26:38.000 --> 00:26:40.320
start figuring out how to build them from there.
369
00:26:41.120 --> 00:26:44.799
The other thing that's a good use of time is building better tools.
370
00:26:45.544 --> 00:26:47.465
So all
371
00:26:47.544 --> 00:26:56.825
of those things that I'm talking about, we have custom harnesses for Bifrost to drive those. It's not just like type into Codecs, Hey, do some differential
372
00:26:56.905 --> 00:27:00.184
analysis for me, but rather, We're going to build
373
00:27:00.570 --> 00:27:09.849
a system that can do that as a nightly job and then give me a report and feed that through the agent that's going to triage it for
374
00:27:10.010 --> 00:27:11.929
actual defects and so forth.
375
00:27:12.490 --> 00:27:16.385
And that can be at the micro level where,
376
00:27:16.945 --> 00:27:20.065
you know, since Bifrost is a tool for agents to use,
377
00:27:21.025 --> 00:27:24.065
a lot of what we do involves looking through the traces
378
00:27:24.145 --> 00:27:29.585
of a model that's trying to use the Bifrost tools and
379
00:27:30.500 --> 00:27:37.379
figuring out where the tools were less helpful than expected. So one of the very low level things we built is something to parse
380
00:27:37.460 --> 00:27:38.579
Codex's
381
00:27:38.659 --> 00:27:39.779
trajectories
382
00:27:39.779 --> 00:27:44.259
for the Bifrost tool calls and pull out the appropriate context so that
383
00:27:44.684 --> 00:27:48.044
Astra doesn't have to reinvent that wheel
384
00:27:48.205 --> 00:27:50.764
every time it's And tackling one of these
385
00:27:53.245 --> 00:27:54.124
at
386
00:27:54.284 --> 00:27:56.924
a higher level, it could include things like Mjolnir,
387
00:27:57.529 --> 00:27:58.169
where
388
00:27:58.330 --> 00:27:59.210
I've
389
00:27:59.210 --> 00:28:01.210
created kind of a meta harness
390
00:28:01.690 --> 00:28:06.489
or a harness harness so that I can use my Codex and my
391
00:28:07.130 --> 00:28:08.090
Anthropic
392
00:28:08.250 --> 00:28:10.490
and my Kimi and my GLN
393
00:28:10.654 --> 00:28:11.534
subscriptions
394
00:28:11.534 --> 00:28:15.934
all in a single pane of glass and keep track of what the different agents are doing.
395
00:28:17.375 --> 00:28:19.374
The other thing that you mentioned was
396
00:28:19.534 --> 00:28:26.255
about how do you upskill as a junior engineer? So if you want to take the conversation in that direction, we could do that.
397
00:28:27.240 --> 00:28:31.400
Yeah. I think that that is an interesting angle as well because
398
00:28:31.400 --> 00:28:33.159
everybody's asking the question.
399
00:28:34.040 --> 00:28:38.280
Various people have attempted an answer, but I haven't seen a...
400
00:28:39.525 --> 00:28:41.605
I haven't seen any sort of
401
00:28:42.725 --> 00:28:50.645
standard developed as far as what are the actual ways that we do that. Not that we did a great job of it prior to AI, but now there are
402
00:28:50.965 --> 00:28:58.019
potentially fewer opportunities or depending how you look at it, more opportunity because the junior engineer can get more done,
403
00:28:58.340 --> 00:29:06.019
but they're also holding a bigger foot gun. So there's potentially a larger blast radius of poor understanding of the problem.
404
00:29:06.419 --> 00:29:07.620
And I think that
405
00:29:08.174 --> 00:29:10.014
one of the ways that
406
00:29:10.495 --> 00:29:17.614
humans can spend their time and tying back to what you were saying about mining the agent traces for how the...
407
00:29:18.095 --> 00:29:22.095
How it's issuing the tool calls for Bifrost or any other tool
408
00:29:22.679 --> 00:29:23.879
is that
409
00:29:24.120 --> 00:29:27.159
in order to prevent a lot of wasted
410
00:29:27.399 --> 00:29:34.119
compute and tokens on retreading the same ground over 50 different agent sessions, as well as
411
00:29:34.519 --> 00:29:37.080
giving junior engineers a place to
412
00:29:37.325 --> 00:29:42.605
look to understand what are the best practices or how do we tackle these types of problems
413
00:29:43.165 --> 00:29:46.845
is through the introduction of different skills
414
00:29:47.005 --> 00:29:47.965
documents
415
00:29:47.965 --> 00:29:53.085
that helps the agent do the work, but also acts as an artifact that people can look at to understand
416
00:29:53.539 --> 00:29:57.059
what do we... What what is the approach for doing this,
417
00:29:57.460 --> 00:30:14.144
and why did we do it this way so that acts as a a shared repository of knowledge if you take the time to actually look at it. And that's actually another area that I and my team have been spending time is every time I have to keep doing something a certain way, I say, okay. Great. Let's go ahead and
418
00:30:14.465 --> 00:30:17.424
read back through all the sessions where I did this particular
419
00:30:17.664 --> 00:30:18.624
workflow,
420
00:30:18.705 --> 00:30:25.140
codify that into a skill so that I can just trigger it when I wanna do that thing again. So one example of that is actually
421
00:30:25.460 --> 00:30:34.420
for code review because I'll open a pull request from my agent. I'll get a different agent to do reviews on that. Lately, it's been Copilot,
422
00:30:34.420 --> 00:30:45.634
but whatever other agent. And then I have a skill that says address PR feedback that will just go through, find all the comments, and pull them in, perform the necessary... Evaluate, perform the necessary changes,
423
00:30:45.875 --> 00:30:47.315
and then close those responses.
424
00:30:48.370 --> 00:30:59.409
And so that gives me a way to go back and forth on create the code, get some feedback on it, address some of the problems without having to involve a human until after that's already been done. If I say,
425
00:30:59.809 --> 00:31:08.585
this actually is a large enough change that I wanna have somebody else look to make sure that my general approach and design is correct before I actually merge and ship this.
426
00:31:09.865 --> 00:31:10.904
Yeah. Someone...
427
00:31:11.544 --> 00:31:20.210
So so for for a couple years there, and I think we've broadly moved past this point, but for a couple of years, people were saying, Well, the AI is like an intern.
428
00:31:20.690 --> 00:31:21.169
And
429
00:31:21.890 --> 00:31:29.970
it's reasonably smart, but it makes a lot of mistakes and it kind of doesn't know what it doesn't know. So you got to supervise it closely. And that was their mental model
430
00:31:30.825 --> 00:31:32.184
of the AI agent.
431
00:31:32.585 --> 00:31:33.144
And
432
00:31:33.544 --> 00:31:36.984
now the agents are smarter and better at writing code,
433
00:31:37.385 --> 00:31:40.664
but they still don't have the big picture.
434
00:31:40.904 --> 00:31:50.349
So someone said, and I don't remember who, that a better model now is it's not an intern so much as it's a great new hire.
435
00:31:51.070 --> 00:31:55.149
And every time you talk to him, it's his first day at the company.
436
00:31:55.965 --> 00:31:57.164
And so
437
00:31:57.405 --> 00:31:58.684
what
438
00:31:58.684 --> 00:32:06.124
that pushes us towards is some of the things that you're talking about that were kind of best practices already in human oriented teams,
439
00:32:06.684 --> 00:32:08.044
at least teams that
440
00:32:08.445 --> 00:32:10.649
hired new people regularly,
441
00:32:10.730 --> 00:32:14.809
that you need to have this knowledge base and this,
442
00:32:14.970 --> 00:32:17.289
you know, a set of runbooks and
443
00:32:17.929 --> 00:32:21.049
things that let those new hires orient themselves
444
00:32:21.274 --> 00:32:27.195
to what's going on. And so what are the decisions we've made that we don't need to relitigate every day?
445
00:32:27.595 --> 00:32:28.154
And
446
00:32:29.595 --> 00:32:35.434
what's our big hairy goal, right? What are we trying to build here? And why? And what problems are we solving?
447
00:32:37.429 --> 00:32:38.149
And
448
00:32:39.110 --> 00:32:42.149
so the difference is, of course, that
449
00:32:44.070 --> 00:32:45.429
maintaining that
450
00:32:45.830 --> 00:32:49.509
was always super, super difficult in the before times.
451
00:32:49.910 --> 00:32:52.549
And so most teams
452
00:32:52.115 --> 00:32:54.275
didn't
453
00:32:54.275 --> 00:32:59.075
do as good a job as they could have. And now I can just say, Okay, Fable,
454
00:32:59.235 --> 00:33:02.995
document what we did and the dead ends we ran into,
455
00:33:03.155 --> 00:33:03.955
and put
456
00:33:06.435 --> 00:33:07.955
that in our set of
457
00:33:08.490 --> 00:33:09.769
plan documents
458
00:33:09.770 --> 00:33:12.330
so that we don't have to re derive that next time.
459
00:33:12.890 --> 00:33:14.330
And we
460
00:33:14.650 --> 00:33:20.649
can talk about how I don't think text files I come from a database background. So I think there's better solutions than a
461
00:33:20.890 --> 00:33:34.044
directory full of Markdown files. But that's kind of where we are right now, is the Markdown file level. And it works pretty well. Eventually, you do outgrow it, but it does work pretty well, surprisingly well even,
462
00:33:34.525 --> 00:33:36.765
for what it is. And then
463
00:33:38.440 --> 00:33:40.519
as a very tactical piece,
464
00:33:41.400 --> 00:33:44.359
one of the things that works really well for us is
465
00:33:44.679 --> 00:33:46.599
OpenAI wrote an article
466
00:33:47.480 --> 00:33:52.600
almost a year ago now. I think it was in November of 'twenty five, but it
467
00:33:52.515 --> 00:33:53.955
was late 'twenty five.
468
00:33:54.275 --> 00:33:59.154
And they wrote an article about using something called exec plans
469
00:33:59.315 --> 00:34:01.875
to drive agents for longer
470
00:34:01.875 --> 00:34:02.755
tasks.
471
00:34:02.995 --> 00:34:03.794
And the
472
00:34:04.275 --> 00:34:07.155
framing of the article was, here's a structure
473
00:34:07.870 --> 00:34:12.510
for how to create plan documents and have your agent create the plan documents
474
00:34:12.510 --> 00:34:15.230
that it can follow that will keep it on task
475
00:34:15.630 --> 00:34:20.925
over a task that takes hours or days. And remember, this is back like
476
00:34:20.925 --> 00:34:22.605
Codex 5.1,
477
00:34:22.605 --> 00:34:28.765
I think was state of the art because 5.2 came out in December and this was definitely before that.
478
00:34:29.005 --> 00:34:31.565
And the Opus was around 4.5,
479
00:34:31.565 --> 00:34:34.125
I think. And both of these models
480
00:34:34.190 --> 00:34:35.870
were the start
481
00:34:36.270 --> 00:34:36.910
of
482
00:34:37.230 --> 00:34:46.750
the new agentic age that we're in now, but they both got distracted really easily. And they both were a little bit lazy where if you gave them a big task, they'd do the first part and
483
00:34:47.070 --> 00:34:47.550
then stop.
484
00:34:48.444 --> 00:34:52.445
And if you just said, well, slash goal do the big task, then they'd
485
00:34:52.765 --> 00:35:05.930
go all over the place and compaction wasn't as good and so forth. So OpenAI said, Hey, use this exec plan idea to keep them on track across compactions in this longer running task and so forth. And we mostly don't need
486
00:35:06.410 --> 00:35:07.770
plan documents
487
00:35:07.930 --> 00:35:09.610
for that purpose anymore.
488
00:35:10.330 --> 00:35:13.290
Or rather, doing a slash plan in
489
00:35:13.610 --> 00:35:15.370
the harness is sufficient.
490
00:35:15.984 --> 00:35:20.065
But we're still doing exact plan documents because of
491
00:35:20.224 --> 00:35:22.785
this category of problem where
492
00:35:23.025 --> 00:35:27.265
two weeks later or two months later, why did we do things this way?
493
00:35:27.585 --> 00:35:38.440
And the exact plan captures that in a way that seems to fit how the agents want to work a lot better than trying to put it into a comment in the code
494
00:35:38.759 --> 00:35:47.605
because for a bunch of reasons, the grep and set approach to going through code means that they have holes in their knowledge and they'll easily miss that kind of comment.
495
00:35:47.765 --> 00:35:51.925
But if you have a document that says, Here's where all of our plans
496
00:35:51.925 --> 00:35:52.645
live,
497
00:35:53.045 --> 00:35:57.605
and go look at them if you need to understand why we did something, that works a lot better.
498
00:35:58.710 --> 00:36:10.630
Yeah. And similarly for architecture decision records, this is the design and why we chose to do it this way that the humans can also look at, and that's something that predates AI workflows as well of
499
00:36:11.015 --> 00:36:31.260
humans who are joining the company afresh, and they're working in that thorny corner of the code and think, I know exactly how to solve this. And then there's... There... I've definitely seen tropes of people saying, okay. Every time you think you're going to solve this, increment this counter of how many times people have tried to do that before going back to whatever it was before.
500
00:36:31.740 --> 00:36:36.460
And on that point of... Yeah. Yeah. On that point of databases
501
00:36:36.460 --> 00:36:36.940
being
502
00:36:37.415 --> 00:36:45.335
a better structure for some of this style of data, that... That's also why I built the tool that I'm using for my shared agent
503
00:36:45.335 --> 00:36:49.415
task and memory context of... It's actually using a storage layer called OmniGraph
504
00:36:49.790 --> 00:36:52.830
and the backing store that uses Lance as the,
505
00:36:53.230 --> 00:36:58.030
storage format, but builds a graph on top of that. So you can actually have the connectivity
506
00:36:58.030 --> 00:37:09.105
between projects and tasks and memories and have the way to be able to go from one to the other and also being able to have memories that get superseded so that as you find out more things,
507
00:37:09.505 --> 00:37:15.265
you can have the most recent, but you still have the way to backtrack to see what did we think we do at this point in time.
508
00:37:15.585 --> 00:37:16.865
And so that's a
509
00:37:17.190 --> 00:37:19.750
interesting angle as well. Oh, so you
510
00:37:20.070 --> 00:37:24.710
check this out. A couple friends of mine started a company called Vectorize,
511
00:37:25.109 --> 00:37:26.150
and their
512
00:37:26.470 --> 00:37:27.270
flagship
513
00:37:27.270 --> 00:37:29.109
product is called Hindsight.
514
00:37:29.645 --> 00:37:31.005
And it's basically
515
00:37:32.125 --> 00:37:34.525
graph and embeddings based
516
00:37:35.325 --> 00:37:36.685
memory for agents.
517
00:37:37.005 --> 00:37:37.725
So,
518
00:37:38.045 --> 00:37:40.365
yeah, it is on GitHub.
519
00:37:40.685 --> 00:37:43.244
Yeah, I think you should probably check it out and see.
520
00:37:44.170 --> 00:37:55.130
At least there's probably some interesting ideas that might be useful cross pollination, and it might even be where you're like, Oh, I don't need to maintain my own anymore. I can use someone else's who does this full time.
521
00:37:56.825 --> 00:37:59.065
Yeah, I'll definitely take a look at that as well.
522
00:38:00.505 --> 00:38:05.385
And that's the other interesting aspect of the time that we're in, is there is a massive proliferation
523
00:38:05.385 --> 00:38:07.224
of people who are solving
524
00:38:07.224 --> 00:38:11.704
adjacent problems or maybe even the same problem in slightly tangential
525
00:38:11.730 --> 00:38:12.930
ways where
526
00:38:13.090 --> 00:38:19.010
we're having an explosion of what are the different concepts and primitives that we actually need for
527
00:38:19.170 --> 00:38:21.490
software engineering agents or
528
00:38:21.490 --> 00:38:24.130
agents for particular workflows.
529
00:38:24.609 --> 00:38:27.170
So it's the let a thousand flowers bloom
530
00:38:27.455 --> 00:38:28.335
mentality.
531
00:38:28.335 --> 00:38:32.735
It'll be interesting to see where we are two years from now once the
532
00:38:32.975 --> 00:38:38.895
market starts to coalesce around some of the primitives that we all agree are useful.
533
00:38:39.295 --> 00:38:47.910
These are the two or three different right ways to do it and how that collapses some of the tools and products that are being launched right now.
534
00:38:48.710 --> 00:38:53.430
Yeah, I think that's true. And I think that's also something that we collectively
535
00:38:53.990 --> 00:38:55.990
need to be a little bit
536
00:38:57.494 --> 00:38:59.175
intentional about. Because
537
00:38:59.494 --> 00:39:00.375
for me,
538
00:39:00.775 --> 00:39:02.855
and I think for a lot of engineers,
539
00:39:03.895 --> 00:39:04.535
whenever
540
00:39:05.414 --> 00:39:07.895
I see somebody selling something
541
00:39:08.055 --> 00:39:12.960
where I'm like, Oh, I understand how to build that. It's a very,
542
00:39:13.120 --> 00:39:18.080
very... It's not a large step to go from, I understand how to build that, to, I should build that.
543
00:39:18.320 --> 00:39:21.680
Especially if it means that I don't have to pay someone
544
00:39:21.840 --> 00:39:23.840
for something that I could do myself.
545
00:39:24.160 --> 00:39:24.720
But
546
00:39:25.805 --> 00:39:27.645
the good part about
547
00:39:27.885 --> 00:39:31.085
AI code is that building my own is
548
00:39:32.285 --> 00:39:33.964
easier than it's ever been.
549
00:39:34.765 --> 00:39:35.805
But I
550
00:39:36.365 --> 00:39:37.964
think that that means that
551
00:39:38.300 --> 00:39:43.900
there's a trap of not seeing like the rest of the iceberg under the surface
552
00:39:44.060 --> 00:39:46.460
that, you know, building the prototype,
553
00:39:46.460 --> 00:39:47.820
yeah, that's a weekend,
554
00:39:48.060 --> 00:39:51.020
but maintaining it and
555
00:39:52.425 --> 00:39:55.865
handling feature requests from my team, that's
556
00:39:55.865 --> 00:39:57.545
a serious time sink
557
00:39:57.705 --> 00:40:00.345
over the next six months, even
558
00:40:00.665 --> 00:40:04.345
if AI is doing most of the actual code authoring.
559
00:40:04.700 --> 00:40:05.660
So I
560
00:40:06.220 --> 00:40:07.260
think that,
561
00:40:08.140 --> 00:40:09.980
and I certainly do this myself,
562
00:40:10.140 --> 00:40:13.260
and I hope I haven't fallen into that trap with Mjolnir,
563
00:40:13.660 --> 00:40:17.820
that building your own is seductive, but not always the right call.
564
00:40:18.725 --> 00:40:35.365
It's the age of the content management system or static site generator where everybody says, Oh, that's an easy problem. I can build my own. And then they get 80% of the way there and said, Oh, I'm almost done. Now I'll spend the rest of my time trying to build that other 20%.
565
00:40:35.040 --> 00:40:37.120
Yeah, that's a really good analogy. Yeah.
566
00:40:37.920 --> 00:40:39.600
That's
567
00:40:39.600 --> 00:40:44.320
definitely another interesting aspect of... Have I built a static site generator? Yes, I have.
568
00:40:46.095 --> 00:40:53.375
And so that's another interesting aspect of where we are in terms of how do you decide what are the appropriate
569
00:40:53.855 --> 00:40:54.895
ways and
570
00:40:55.455 --> 00:40:58.095
places to spend your limited human attention.
571
00:40:58.809 --> 00:40:59.609
Because,
572
00:40:59.770 --> 00:41:04.650
as you said, it is very seductive to say, oh, I know how to build that. I'll go build it for myself. But now
573
00:41:04.970 --> 00:41:08.250
more of my time is getting sunk into this offshoot
574
00:41:08.490 --> 00:41:24.375
toy project that is maybe helpful, but is distracting from what are the actual concrete broader goals that I'm getting paid to achieve. I mean, if you're doing it for fun in your hobby time, sure. But, again, you can only have so many hobbies before they take
575
00:41:24.375 --> 00:41:26.855
over an increasing amount of your life. And
576
00:41:27.849 --> 00:41:32.330
I I think another aspect of this problem going beyond just
577
00:41:32.490 --> 00:41:36.250
agenting software engineering into the broader agent ecosystem
578
00:41:36.250 --> 00:41:36.890
is
579
00:41:37.210 --> 00:41:42.810
what is the role of humans when agents can do more and more of the daily tasks of
580
00:41:43.185 --> 00:41:45.345
dealing with my emails or
581
00:41:45.665 --> 00:41:46.465
managing
582
00:41:46.465 --> 00:41:47.185
the
583
00:41:47.825 --> 00:41:49.745
data ingest workflows
584
00:41:49.745 --> 00:41:52.865
as data continues to break or
585
00:41:53.185 --> 00:41:53.905
building
586
00:41:54.065 --> 00:42:00.870
and iterating on the product based on what are the requests that I'm getting from my users. And that
587
00:42:01.030 --> 00:42:10.869
brings back that question of human judgment that open source maintainers have had to deal with for years now. Just because somebody can request a feature doesn't mean that it's a feature that I want to own.
588
00:42:11.755 --> 00:42:13.675
No, that's 100
589
00:42:13.675 --> 00:42:14.715
true. And
590
00:42:15.035 --> 00:42:17.995
again, I don't remember who said this first and
591
00:42:18.075 --> 00:42:19.755
possibly lots of people,
592
00:42:20.075 --> 00:42:25.675
but the most important job of an open source maintainer is saying no. And that
593
00:42:26.155 --> 00:42:26.635
was true
594
00:42:27.960 --> 00:42:31.400
five years ago, but it's significantly more true now.
595
00:42:32.119 --> 00:42:32.680
And
596
00:42:33.000 --> 00:42:34.520
to the point that,
597
00:42:34.840 --> 00:42:35.800
by the way,
598
00:42:36.840 --> 00:42:38.600
I definitely appreciate
599
00:42:39.240 --> 00:42:39.960
everyone
600
00:42:40.040 --> 00:42:42.280
whose
601
00:42:41.105 --> 00:42:43.585
libraries I'm using on GitHub
602
00:42:43.585 --> 00:42:46.705
who have PRs open. And just
603
00:42:47.825 --> 00:42:53.025
yesterday, I upstreamed PRs to two different libraries that Brock is using,
604
00:42:54.065 --> 00:42:55.505
and they were both accepted,
605
00:42:56.559 --> 00:42:59.200
or rather the sets of PRs were both accepted.
606
00:42:59.839 --> 00:43:06.080
So, I appreciate being able to contribute like that. But at the same time, I don't think that that's
607
00:43:07.920 --> 00:43:08.400
scalable
608
00:43:09.155 --> 00:43:13.555
as the barrier towards, Oh, I have a problem. Let me throw up a PR.
609
00:43:13.875 --> 00:43:15.395
Because a lot of people
610
00:43:17.395 --> 00:43:19.795
do throw up PRs without actually
611
00:43:21.170 --> 00:43:24.690
doing a whole lot of the kind of review that we're talking about.
612
00:43:25.090 --> 00:43:33.170
And so, it's very easy for those open source maintainers to get snowed under by trying to give feedback
613
00:43:33.545 --> 00:43:43.545
on those PRs that were written by agents with very little human oversight and sometimes by people who aren't professional developers. So on the one hand, I'm
614
00:43:43.545 --> 00:43:46.105
not here to gatekeep, and I think it's great that
615
00:43:46.830 --> 00:43:49.710
AIs are opening up creating
616
00:43:50.110 --> 00:43:53.230
tools to make lives better in digital
617
00:43:53.790 --> 00:43:54.750
worlds
618
00:43:54.830 --> 00:44:03.915
for people who aren't professional developers. I think that's great. But at the same time, when the number of people who can open a PR to your product
619
00:44:04.155 --> 00:44:04.955
is
620
00:44:05.595 --> 00:44:08.635
literally everyone in the world and not just the professionals,
621
00:44:08.635 --> 00:44:11.835
it does mean that maybe there needs to be another barrier
622
00:44:12.339 --> 00:44:12.980
to
623
00:44:13.380 --> 00:44:15.380
reduce that maintainer
624
00:44:15.380 --> 00:44:16.180
burnout.
625
00:44:16.180 --> 00:44:31.205
And so what we're doing with Bifrost, we haven't done this with any of our other repos, but with our flagship Bifrost repo, we have turned off PRs. And so the issues are open. If you have a problem or a feature you want to request,
626
00:44:31.365 --> 00:44:35.205
open an issue. But it's just faster for me to
627
00:44:35.365 --> 00:44:37.205
drive that in the right direction
628
00:44:38.109 --> 00:44:41.630
with my own agents, with my own harness, with my own QA
629
00:44:41.710 --> 00:44:42.590
system,
630
00:44:42.910 --> 00:44:51.070
than it is to try to do a dance of, Hey, can you fix this? That can potentially take several days with contributors.
631
00:44:51.070 --> 00:44:53.310
So I think that's unfortunate
632
00:44:53.205 --> 00:44:54.005
because
633
00:44:54.005 --> 00:44:55.925
I think that that
634
00:44:56.405 --> 00:44:57.445
open source
635
00:44:57.445 --> 00:44:58.325
ecosystem
636
00:44:58.325 --> 00:45:08.165
and community building and training the next generation and all of that were good outcomes from the way we used to do it. But there is historical precedent,
637
00:45:08.970 --> 00:45:14.730
which is that back in the '80s when Richard Stallman started the Free Software Foundation,
638
00:45:14.890 --> 00:45:29.255
it was not a GitHub style, Hey, everyone, bring your contributions and let's build this together. He believed very strongly and still believes very strongly in the importance of having the source available so that you can make changes and have the freedom
639
00:45:29.495 --> 00:45:30.455
to do so.
640
00:45:30.855 --> 00:45:31.415
But
641
00:45:31.735 --> 00:45:33.015
the FSF
642
00:45:33.015 --> 00:45:37.015
in the 80s did not accept patches from external contributors.
643
00:45:37.860 --> 00:45:41.540
It was much more of a shared source, is what we'd call it today,
644
00:45:41.780 --> 00:45:43.300
than open source.
645
00:45:43.780 --> 00:45:45.220
And I
646
00:45:45.380 --> 00:45:47.540
kind of think that's the direction we're heading in.
647
00:45:49.140 --> 00:46:08.970
Yeah, I've definitely heard about a number of other open source projects that have followed that same model of you can open an issue, and then I will send that issue to my agent to do the code change because of the fact that, as you said, the PR itself requires a lot more time investment and communication overhead.
648
00:46:12.010 --> 00:46:15.770
Think that one way that, as an industry,
649
00:46:15.930 --> 00:46:16.570
we
650
00:46:17.184 --> 00:46:20.145
should be thinking about the ways that we develop
651
00:46:20.145 --> 00:46:21.905
our workflows around
652
00:46:21.984 --> 00:46:25.265
AI being the primary actor
653
00:46:25.505 --> 00:46:26.224
is
654
00:46:26.385 --> 00:46:27.105
to
655
00:46:27.664 --> 00:46:29.664
codify and enforce
656
00:46:29.664 --> 00:47:09.960
more of that human judgment requirement where rather than giving the agent an objective and say, k. Go build this thing, force it to ask you why or what is the actual thing that we're trying to achieve, be... Which that's a human problem as well, where I can go and tell one of my employees, go build this thing, and they'll happily go off and build the thing that I said because I am perceived as the expert. But maybe I don't actually know what I'm talking about in this regard. And so having more of that pushback and requiring the human to be more involved in the end to end delivery of what are we building? Why are we building it? What are the things that we need to think about before we put this thing out into the world? And then also maybe to your point of
657
00:47:10.625 --> 00:47:12.865
there are 50 different versions of
658
00:47:13.025 --> 00:47:14.145
problem x,
659
00:47:14.625 --> 00:47:19.425
agentic memory, coding harnesses. Those are some of the ones for this era.
660
00:47:19.744 --> 00:47:21.825
Go and do some research to see,
661
00:47:22.700 --> 00:47:29.260
do any of those solve 90% of the problem that I think I'm solving, and maybe I should just use those instead of building another one.
662
00:47:30.540 --> 00:47:31.180
Yeah.
663
00:47:32.140 --> 00:47:35.900
Yeah, I do wanna come back to to what you said earlier about
664
00:47:36.295 --> 00:47:40.695
upskilling the next generation. We talked a little bit about onboarding
665
00:47:40.935 --> 00:47:45.175
the agents kind of in the context of onboarding new humans,
666
00:47:45.495 --> 00:47:48.135
but that's
667
00:47:48.135 --> 00:47:50.855
a bit of a different problem than how do you go from
668
00:47:51.710 --> 00:47:55.869
a new college graduate to being able to do what you do
669
00:47:56.190 --> 00:47:56.990
in
670
00:47:57.150 --> 00:48:00.109
terms of being able to supervise agents effectively.
671
00:48:00.589 --> 00:48:01.150
And
672
00:48:02.589 --> 00:48:07.615
I don't really have a good solution to that. And I'm kind of hoping that you do because,
673
00:48:08.335 --> 00:48:17.055
or that at least collectively we can come up with ways to do that. The best that I've found so far, or the best that I've been able to
674
00:48:17.375 --> 00:48:19.610
come up with so far is,
675
00:48:19.770 --> 00:48:22.410
so my 18 year old right now is
676
00:48:23.130 --> 00:48:25.450
studying computer science and he wants
677
00:48:26.090 --> 00:48:27.290
to be an engineer.
678
00:48:27.610 --> 00:48:28.250
And,
679
00:48:28.410 --> 00:48:32.570
you know, the rule of thumb with him is, you know, you can use agents
680
00:48:33.375 --> 00:48:34.015
to
681
00:48:34.255 --> 00:48:35.455
do anything
682
00:48:35.455 --> 00:48:38.415
that's routine that you understand how to do,
683
00:48:38.815 --> 00:48:40.095
but don't
684
00:48:40.095 --> 00:48:48.175
use it for anything. Don't throw an open problem that you don't understand at Claude and say, solve this for me, because then you're just cheating your future self
685
00:48:48.520 --> 00:48:50.360
of that understanding.
686
00:48:50.360 --> 00:48:56.280
And there's been, you know, I think we have a little bit of historical
687
00:48:56.600 --> 00:49:00.440
precedent there where we've taught kids to do arithmetic
688
00:49:00.755 --> 00:49:05.715
in their heads or on paper, even though calculators have existed for
689
00:49:06.515 --> 00:49:07.955
lots of years and
690
00:49:08.355 --> 00:49:11.795
slide rules were a thing even before that. So there's
691
00:49:11.795 --> 00:49:12.915
some value
692
00:49:13.130 --> 00:49:16.170
and we've created some patterns for,
693
00:49:16.330 --> 00:49:24.410
here's how you do something by hand and here's why it's valuable to do something by hand, even though we've already figured out how to automate it. But
694
00:49:25.385 --> 00:49:30.185
this pushes that a lot farther down. It's like even deep into
695
00:49:30.425 --> 00:49:36.185
graduate school, you'll still be doing things by hand that we already can automate with agents,
696
00:49:36.505 --> 00:49:42.500
which is, at the very least, it's a mental shift for our educational system.
697
00:49:43.380 --> 00:49:57.045
And maybe one way to think about this in the context of software engineering teams, particularly for people who are earlier in their career or who are working in an unfamiliar area and trying to upskill is the idea of training wheels where
698
00:49:57.205 --> 00:50:22.655
the harness and the agent will do the work, but maybe it will only execute on it after you've been able to provide a satisfactory description of the problem space and what you've thought through. And maybe as you get more adept at a particular area, it adapts and says, okay, well, you have already proven your expertise in this area. So I'm going to go ahead and do what you asked me to do because you've demonstrated ability
699
00:50:22.655 --> 00:50:33.295
similar to the way that we demonstrate ability in our work and get promotions in that regard. And so having the harness push back more and not just blindly accept every instruction that is thrown at it.
700
00:50:33.855 --> 00:50:35.375
Yeah. And then another
701
00:50:35.720 --> 00:50:38.839
analogy that gives me a little bit of hope here
702
00:50:39.079 --> 00:50:41.400
is the chess space,
703
00:50:41.880 --> 00:50:44.680
where it's literally been thirty years
704
00:50:45.000 --> 00:50:51.240
since a human was the best chess player in the world. Deep Blue beat Gary Kasparov in 1997,
705
00:50:51.945 --> 00:50:54.185
and that was a rematch. He the
706
00:50:54.345 --> 00:50:57.145
first Deep Blue in 'ninety six, if I remember correctly,
707
00:50:57.225 --> 00:51:00.665
but then they went back and revised their algorithms and
708
00:51:01.465 --> 00:51:04.505
expanded their hardware, and then they won the rematch.
709
00:51:05.305 --> 00:51:06.985
And, you know, today, of course,
710
00:51:07.760 --> 00:51:08.880
you don't
711
00:51:09.200 --> 00:51:11.680
need a big RS6000
712
00:51:12.000 --> 00:51:12.800
supercomputer
713
00:51:12.800 --> 00:51:13.760
powering it.
714
00:51:14.800 --> 00:51:17.920
My watch could be all but the best humans at chess.
715
00:51:20.655 --> 00:51:21.375
But,
716
00:51:21.535 --> 00:51:22.575
and yet,
717
00:51:23.375 --> 00:51:24.255
with that,
718
00:51:24.575 --> 00:51:28.975
chess streamers are one of the most popular forms of entertainment on Twitch.
719
00:51:29.215 --> 00:51:29.695
And
720
00:51:31.455 --> 00:51:32.655
they're getting more popular.
721
00:51:33.180 --> 00:51:34.619
So there's
722
00:51:34.619 --> 00:51:38.700
space for, even though machines are better at this than humans,
723
00:51:38.940 --> 00:51:42.380
it's still useful and still interesting for humans to
724
00:51:42.780 --> 00:51:44.460
compete and try to improve
725
00:51:45.035 --> 00:51:45.675
in
726
00:51:45.995 --> 00:51:47.275
that problem space.
727
00:51:47.675 --> 00:51:48.555
And of course,
728
00:51:48.955 --> 00:51:49.755
in the
729
00:51:49.995 --> 00:51:51.595
less intellectual domain,
730
00:51:52.075 --> 00:51:54.635
we've had machines that are better at
731
00:51:55.115 --> 00:51:56.955
any kind of physical
732
00:51:58.310 --> 00:52:03.030
exertion than humans for two hundred years, and we still go to the gym to
733
00:52:03.350 --> 00:52:04.950
get more athletic. So
734
00:52:06.790 --> 00:52:16.535
I'm not in the camp of, Well, AIs are better than humans at everything, so we might as well give up. I think that there is some value to doing things as a human,
735
00:52:16.695 --> 00:52:19.495
even if the machine is better at it than I am.
736
00:52:19.815 --> 00:52:20.775
But yeah,
737
00:52:21.015 --> 00:52:26.295
the world is not going to be the same. It's not going to go back to being the same. So it's going to be exciting.
738
00:52:27.730 --> 00:52:29.170
And also,
739
00:52:29.490 --> 00:52:34.050
there is always a role for the human to play because despite all of the terminology
740
00:52:34.050 --> 00:52:37.810
that we use and all of the hype that has grown up around the ecosystem,
741
00:52:38.370 --> 00:52:43.785
these large language models or these various deep learning models do not have any agency
742
00:52:43.945 --> 00:52:47.305
of their own. They are a tool that we point in the direction.
743
00:52:47.465 --> 00:52:53.305
And so you still need to have that human judgment of what is the direction I want this thing to go in and understand,
744
00:52:54.369 --> 00:53:00.290
one, what are the capabilities and why am I going in this direction, but also have the sufficient
745
00:53:00.290 --> 00:53:03.650
knowledge to understand whether or not you have achieved the stated goal.
746
00:53:04.770 --> 00:53:08.450
See, that's something that I'm not sure about. That's certainly true today,
747
00:53:08.964 --> 00:53:12.165
and it's certainly true next year. Is it true in ten years?
748
00:53:12.565 --> 00:53:15.605
Do we never get to a point where I can tell
749
00:53:15.845 --> 00:53:17.125
Claude 14,
750
00:53:17.765 --> 00:53:18.325
know,
751
00:53:20.724 --> 00:53:21.925
a loop that
752
00:53:22.329 --> 00:53:26.170
figures out the next thing to build. Here's my goal.
753
00:53:26.170 --> 00:53:29.450
Here's what I'm trying to build. Now just build the next thing
754
00:53:29.770 --> 00:53:32.490
that gets me closer to that goal in a loop.
755
00:53:32.730 --> 00:53:45.865
Do we never get there? I'm less sure about that. But even in that example, you're still the one stating the goal. The AI isn't going to be the one to tell you your goal. I mean, maybe sometimes it does, but you as a human still have agency that
756
00:53:46.265 --> 00:53:47.465
the model itself lacks.
757
00:53:48.069 --> 00:53:49.510
Okay, that's
758
00:53:49.589 --> 00:53:52.790
fair. But that is saying that either
759
00:53:53.030 --> 00:53:55.990
the CEO is the only one who's not replaceable,
760
00:53:56.150 --> 00:53:59.270
or we're all mini CEOs of our own agents.
761
00:54:00.414 --> 00:54:01.214
Yeah, I
762
00:54:02.894 --> 00:54:25.200
don't know. I don't know. I would say that that's a fair approximation of the world as it is right now, where you have people who... Large companies where you have a CEO and there people who are happy to work under them in their particular role. And then you have people who chafe at being told what to do by anyone, and they go and run their own small businesses focused on whatever it is that they want to do in the world.
763
00:54:26.385 --> 00:54:29.265
That's that's a that's a good analogy.
764
00:54:29.345 --> 00:54:29.905
Yes.
765
00:54:30.145 --> 00:54:30.945
Alright.
766
00:54:30.945 --> 00:54:33.265
Now that we've played that out, I guess,
767
00:54:33.825 --> 00:54:44.890
circling back around to what you're building, I'm curious, what are some of the most interesting or innovative or unexpected ways that you've seen the tool chain that you're currently focused on applied? So Bifrost,
768
00:54:44.890 --> 00:54:47.609
Mjolnir, and I think you're also building something called Anvil.
769
00:54:48.170 --> 00:54:53.609
Yeah, so Anvil is we've de emphasized Anvil. So Mjolnir
770
00:54:53.690 --> 00:54:56.089
is, like I said, it's a meta harness.
771
00:54:56.744 --> 00:55:00.345
And what that means in this context is
772
00:55:00.664 --> 00:55:01.385
we're
773
00:55:01.625 --> 00:55:07.385
driving codecs and clod code and kimicode and grok build and
774
00:55:07.545 --> 00:55:08.585
musecode.
775
00:55:08.744 --> 00:55:12.820
We're driving those directly from that text interface
776
00:55:12.820 --> 00:55:14.980
that Mjolnir shows the user.
777
00:55:15.619 --> 00:55:18.660
And we're driving those over a protocol called ACP.
778
00:55:18.900 --> 00:55:23.220
And ACP was created so that you could put Cloud Code into
779
00:55:23.765 --> 00:55:27.205
your IntelliJ or into your Versus Code and so forth,
780
00:55:27.525 --> 00:55:30.965
but it's still Cloud Code under the hood. It's not a separate harness.
781
00:55:31.285 --> 00:55:37.685
And so Anvil was something where we said, well, we've got these interesting ideas for the harness itself, so we'll have our own
782
00:55:38.500 --> 00:55:39.700
ACP server,
783
00:55:39.860 --> 00:55:42.740
which is what the harness's role is. But
784
00:55:43.300 --> 00:55:46.020
as the models have gotten more sophisticated
785
00:55:46.180 --> 00:55:50.580
and they're doing things like, you know, Claude code will invalidate
786
00:55:50.580 --> 00:55:51.300
individual
787
00:55:51.895 --> 00:55:52.775
chunks
788
00:55:52.935 --> 00:55:56.375
of the inference server's key value cache
789
00:55:57.015 --> 00:55:59.415
based on what's going on in your session.
790
00:55:59.735 --> 00:56:02.135
And I'm like, well, that's a level of
791
00:56:02.455 --> 00:56:09.490
integration that I don't think that I can outcompete the big labs at. So Anvil has been deemphasized.
792
00:56:10.609 --> 00:56:13.650
Do think we are adding value with Mjolnir over
793
00:56:14.290 --> 00:56:17.090
the Codex and the Clog code and so forth layer.
794
00:56:17.650 --> 00:56:26.085
And to give an example of what you asked about, what's something that's surprising or that I'm proud of, we just added a feature to Mjolnir
795
00:56:26.244 --> 00:56:27.045
where
796
00:56:27.125 --> 00:56:28.245
it will...
797
00:56:28.485 --> 00:56:30.965
So Mjolnir has a concept of workspaces
798
00:56:30.965 --> 00:56:32.165
and sessions
799
00:56:32.165 --> 00:56:39.230
belong to a workspace, and you can navigate between the sessions and see what's going on. But we added a virtual workspace
800
00:56:39.630 --> 00:56:41.310
for sub agents.
801
00:56:41.630 --> 00:56:44.350
And so you can have your
802
00:56:44.750 --> 00:56:47.790
codex sub agents and click on
803
00:56:48.670 --> 00:56:55.335
the six sub agents label, and it opens up a virtual workspace, and you can see what's going on in there. And those sub
804
00:56:55.575 --> 00:56:58.935
agents don't have to be from the same artist. So you
805
00:56:59.975 --> 00:57:08.970
can have Fable driving a Luna sub agent, which is a very economical way to use Fable because Luna is really, really inexpensive.
806
00:57:09.369 --> 00:57:11.770
Or DeepSeek Flash. I
807
00:57:12.010 --> 00:57:16.410
had Fable driving six DeepSeek Flash sub agents yesterday
808
00:57:16.490 --> 00:57:17.770
working on a problem
809
00:57:18.155 --> 00:57:18.955
because
810
00:57:19.595 --> 00:57:20.395
it's
811
00:57:20.395 --> 00:57:23.915
part of the subscription optimization game where
812
00:57:24.555 --> 00:57:33.755
my Fable quota is much more valuable than the other half of the subscription that only lets me use Opus and Sonnet in Anthropic's case.
813
00:57:35.730 --> 00:57:38.050
So on the meta harness side,
814
00:57:38.770 --> 00:57:44.370
that's what I'm excited about is kind of unlocking the value of my different subscriptions playing together.
815
00:57:44.690 --> 00:57:45.970
And then in
816
00:57:46.210 --> 00:57:47.330
the Bifrost side,
817
00:57:47.994 --> 00:57:53.195
this is going to get really deep into the technical weeds, but we've built
818
00:57:53.435 --> 00:57:55.515
the Bifrost database
819
00:57:55.835 --> 00:57:58.715
on top of Git, basically.
820
00:57:59.035 --> 00:58:00.715
The primary key
821
00:58:01.000 --> 00:58:05.240
in Bifrost database is the Git blob ID,
822
00:58:05.400 --> 00:58:08.040
so the hash of the contents
823
00:58:08.120 --> 00:58:10.600
that's in the Git index.
824
00:58:10.920 --> 00:58:13.080
And so what that unlocks
825
00:58:13.080 --> 00:58:19.995
is that if I have 12 agents working in 12 work trees and making changes simultaneously,
826
00:58:20.395 --> 00:58:23.195
I only have to re index changes
827
00:58:23.195 --> 00:58:24.395
that are
828
00:58:24.475 --> 00:58:26.155
new, that have never existed
829
00:58:27.035 --> 00:58:33.810
in any work tree, and I don't have to re index anything that has ever existed anywhere else. So creating a new work tree
830
00:58:34.370 --> 00:58:37.090
and indexing that in Bifrost is instant
831
00:58:37.970 --> 00:58:41.170
because all of those blobs have already existed.
832
00:58:41.810 --> 00:58:42.690
And then
833
00:58:43.434 --> 00:58:51.915
what that unlocks now is that's what unlocks the speed difference of, I can ask Bifrost during the coding session,
834
00:58:53.115 --> 00:58:53.675
committed
835
00:58:53.994 --> 00:58:56.315
have any concurrency crimes
836
00:58:56.510 --> 00:58:59.069
that the Bifrost policy engine catches
837
00:58:59.069 --> 00:58:59.870
rather
838
00:59:00.109 --> 00:59:00.670
than
839
00:59:00.910 --> 00:59:02.830
an hours long CI session,
840
00:59:02.990 --> 00:59:05.710
because I just have to re index those pieces
841
00:59:06.030 --> 00:59:09.869
that have actively had edits since the last time it got indexed.
842
00:59:10.595 --> 00:59:12.915
And as you have been
843
00:59:13.075 --> 00:59:15.234
working in this space and
844
00:59:15.954 --> 00:59:16.994
trying to
845
00:59:17.954 --> 00:59:30.410
discern what are the most effective ways for humans to be involved in this agent owned software delivery life cycle? What are some of the most interesting or unexpected or challenging lessons that you've learned in the process?
846
00:59:31.530 --> 00:59:33.210
I guess I'm going to
847
00:59:33.850 --> 00:59:35.050
contradict my
848
00:59:35.210 --> 00:59:36.570
talking points a little bit.
849
00:59:39.015 --> 00:59:40.295
Because one of the most
850
00:59:40.775 --> 00:59:41.655
frustrating
851
00:59:41.655 --> 00:59:44.295
and challenging parts is that,
852
00:59:44.934 --> 00:59:52.934
while I think everything we've said about how agents are so good at writing code now that we don't need to review their work is
853
00:59:53.420 --> 00:59:54.940
correct on average.
854
00:59:55.660 --> 00:59:57.820
There's always corner cases
855
00:59:57.980 --> 00:59:58.700
where
856
00:59:59.660 --> 01:00:02.060
our running joke on our Slack is,
857
01:00:02.380 --> 01:00:08.700
I guess I'm not obsolete yet, because you constantly find these places where
858
01:00:09.954 --> 01:00:11.155
Fable decided
859
01:00:11.155 --> 01:00:11.875
that
860
01:00:12.195 --> 01:00:13.795
what I really needed
861
01:00:14.035 --> 01:00:16.035
was to clone.
862
01:00:16.195 --> 01:00:18.355
So Mjolnir's written in Rust,
863
01:00:18.515 --> 01:00:22.435
and so it decided that what I really needed to do was to clone the entire
864
01:00:23.079 --> 01:00:24.119
conversation
865
01:00:24.440 --> 01:00:28.360
every time I appended a new message to that conversation.
866
01:00:28.440 --> 01:00:29.880
So it's like, well, why
867
01:00:31.240 --> 01:00:32.920
is my Mjolnir process
868
01:00:33.000 --> 01:00:34.600
spending all its time
869
01:00:35.000 --> 01:00:37.000
in memory remapping?
870
01:00:37.000 --> 01:00:38.280
Oh, it's because
871
01:00:38.655 --> 01:00:39.295
your
872
01:00:39.535 --> 01:00:40.335
allocations
873
01:00:40.415 --> 01:00:44.895
are through the roof. It's like, well, should I really need to tell Fable
874
01:00:45.055 --> 01:00:46.735
in 2026
875
01:00:46.815 --> 01:00:47.615
that
876
01:00:48.095 --> 01:00:52.175
is not a good memory allocation pattern? Like, I feel like I shouldn't,
877
01:00:52.335 --> 01:00:54.880
and yet in this case I did. So
878
01:00:55.280 --> 01:00:57.760
yeah, the edge is still jagged,
879
01:00:58.000 --> 01:00:59.360
but I do think that
880
01:00:59.920 --> 01:01:02.480
it is up and to the right over time.
881
01:01:03.360 --> 01:01:08.080
At some point, not today, at some point, we won't have to think about that anymore.
882
01:01:09.325 --> 01:01:11.165
Yeah. The models are definitely
883
01:01:11.245 --> 01:01:13.565
very adept at being confidently wrong.
884
01:01:14.205 --> 01:01:16.525
Yes. That is still true. Yes.
885
01:01:17.885 --> 01:01:18.925
And so
886
01:01:19.085 --> 01:01:20.445
as people are
887
01:01:21.010 --> 01:01:23.810
remapping their overall workflow
888
01:01:23.890 --> 01:01:28.530
and rethinking the role of humans in the production of software,
889
01:01:29.090 --> 01:01:34.850
what are some of the things you have planned for future looking tools or
890
01:01:35.025 --> 01:01:38.305
thought pieces or experiments to help
891
01:01:38.545 --> 01:01:41.665
us as an industry move in a
892
01:01:42.145 --> 01:01:43.505
beneficial direction
893
01:01:43.585 --> 01:01:45.744
for some definition of beneficial?
894
01:01:46.385 --> 01:01:47.265
Well,
895
01:01:47.265 --> 01:01:49.424
I guess just at a really tactical level,
896
01:01:50.589 --> 01:01:52.910
one of the things that I'm frustrated
897
01:01:52.910 --> 01:01:53.790
about, again,
898
01:01:54.190 --> 01:01:56.670
is the state of model benchmarking.
899
01:01:57.390 --> 01:01:59.390
And I say again because
900
01:01:59.390 --> 01:02:02.430
in 2025,
901
01:02:03.355 --> 01:02:10.315
the state of the art for benchmarking coding models was SWE bench, the original SWE bench, which consists
902
01:02:10.875 --> 01:02:11.595
of
903
01:02:11.755 --> 01:02:13.275
100% Python
904
01:02:13.435 --> 01:02:15.035
and like 49%
905
01:02:15.035 --> 01:02:17.355
of it, I don't remember the exact number, is Django.
906
01:02:17.770 --> 01:02:18.490
So
907
01:02:18.730 --> 01:02:21.610
a high sweepage code was basically just measuring
908
01:02:22.010 --> 01:02:35.765
how well... Oh, and the problems they give the models were all based on GitHub tickets from like a ten year time span. So a high sweep in score was basically how well has your model memorized Django
909
01:02:35.845 --> 01:02:40.485
and a little bit scikit learn and a little bit a couple other Python packages.
910
01:02:41.045 --> 01:02:42.725
And so
911
01:02:43.220 --> 01:02:46.500
I wrote something called Brockbench to do more realistic
912
01:02:46.500 --> 01:02:47.780
coding challenges.
913
01:02:47.940 --> 01:02:53.140
And then at the beginning of this year, we kind of got a next generation of coding benchmarks,
914
01:02:53.940 --> 01:02:55.140
especially DeepSUI
915
01:02:55.140 --> 01:02:56.820
and CursorBench
916
01:02:57.059 --> 01:02:58.020
and
917
01:02:58.914 --> 01:02:59.875
FrontierSUI,
918
01:02:59.875 --> 01:03:02.115
I think is another one. There's
919
01:03:02.115 --> 01:03:03.555
two or three that are reasonably
920
01:03:03.954 --> 01:03:04.755
good now.
921
01:03:05.154 --> 01:03:07.714
And so I said, okay, don't have to do this anymore.
922
01:03:08.355 --> 01:03:09.315
But lately,
923
01:03:09.474 --> 01:03:11.234
with the last couple months of releases,
924
01:03:11.820 --> 01:03:14.060
DeepSUI is saturated.
925
01:03:14.060 --> 01:03:14.940
Everything's
926
01:03:14.940 --> 01:03:16.940
peaking at about 80% and
927
01:03:17.260 --> 01:03:21.820
it's telling me things like Opus five is better than Fable 5.1.
928
01:03:21.820 --> 01:03:23.980
And I know that's not true. That
929
01:03:24.140 --> 01:03:28.055
is just not true. Whatever you're measuring there is
930
01:03:28.535 --> 01:03:29.655
less useful
931
01:03:29.815 --> 01:03:34.615
than it was six months ago. And I don't know if it's because the agents or the
932
01:03:34.775 --> 01:03:37.494
labs are training on the benchmark or it's
933
01:03:37.494 --> 01:03:38.455
less useful.
934
01:03:38.694 --> 01:03:45.349
And so what I've been working on over the last couple weeks has been, I'm taking live
935
01:03:45.349 --> 01:03:46.790
Bifrost tickets
936
01:03:47.190 --> 01:03:47.910
and
937
01:03:47.990 --> 01:03:50.470
I'm doing a bake off between
938
01:03:50.710 --> 01:03:51.589
two
939
01:03:51.990 --> 01:03:53.589
or three models at a time.
940
01:03:54.150 --> 01:03:58.630
And I'm having Fable and Astra judge
941
01:03:57.875 --> 01:03:58.994
the results.
942
01:03:59.315 --> 01:04:03.395
So I'm gonna have basically a series of pair rankings across
943
01:04:03.474 --> 01:04:05.155
a couple 100 tasks
944
01:04:05.234 --> 01:04:05.955
for
945
01:04:07.234 --> 01:04:08.515
DeepSeek Flash,
946
01:04:08.915 --> 01:04:18.119
for GLM five point three and five point three Flash, for MuseSpark, for Grok Build. Basically, all the models that you might want to use when you're like, Oh, help,
947
01:04:18.200 --> 01:04:18.840
I'm
948
01:04:19.640 --> 01:04:20.520
out of my
949
01:04:21.160 --> 01:04:22.680
CLOUD subscription tokens.
950
01:04:23.125 --> 01:04:31.205
What can I use now that's not going to break the bank and is still going to help me get things done? That's basically the question that I'm trying to answer.
951
01:04:31.445 --> 01:04:32.085
So I
952
01:04:32.405 --> 01:04:33.685
should have the first
953
01:04:34.005 --> 01:04:36.965
results by the end of this week, next week at the latest.
954
01:04:37.869 --> 01:04:57.825
And are there any other aspects of the work that you're doing at Brock AI and the various tool chains that you're investing in, or just this overall question of where and how best to spend our human attention in this world of AI generated everything that we didn't discuss yet that you'd like to cover before we close out the show?
955
01:04:59.025 --> 01:05:00.385
Well, my
956
01:05:00.385 --> 01:05:05.744
goal with Brock AI is to get to that ideal point that you articulated earlier of
957
01:05:06.080 --> 01:05:09.680
we're spending our time as humans deciding what to build
958
01:05:10.080 --> 01:05:15.360
and less on checking, hey, did the model actually build what I told it to do?
959
01:05:15.680 --> 01:05:17.760
And that's what we're trying to build with Brock.
960
01:05:19.045 --> 01:05:35.260
All right. Well, for anybody who wants to get in touch with you and follow along with the work that you and your team are doing, I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get your perspective on what you see as being the biggest gaps in the tooling, technology, or human training that's available for building AI systems today?
961
01:05:36.220 --> 01:05:38.940
Biggest gap? I think the biggest gap is
962
01:05:39.819 --> 01:05:41.579
agent oriented collaboration
963
01:05:41.579 --> 01:05:45.900
tools. So we've got GitHub that's oriented towards humans.
964
01:05:46.140 --> 01:05:48.460
We've got Linear that's
965
01:05:48.905 --> 01:05:55.705
also oriented towards humans and maybe doing a faster job of adapting to agents than GitHub is.
966
01:05:55.945 --> 01:05:59.225
But what I would like is a world where
967
01:05:59.385 --> 01:06:00.825
when I file a ticket,
968
01:06:02.380 --> 01:06:15.580
something in the system can say, Oh, this is related to code that agent three forty five wrote last week. Let me revive that session on the appropriate harness and tell it, Hey, you caused a regression.
969
01:06:15.915 --> 01:06:19.515
So it's got all the context there. It doesn't need to rediscover it.
970
01:06:19.675 --> 01:06:23.435
And just wire those pieces of the life cycle together.
971
01:06:24.155 --> 01:06:25.035
If somebody
972
01:06:25.115 --> 01:06:27.355
had that, I would be a customer tomorrow.
973
01:06:28.809 --> 01:06:36.250
All right. Well, thank you very much for taking the time today to join me and share your thoughts and opinions and expertise
974
01:06:36.250 --> 01:06:38.970
on this fast moving space of
975
01:06:39.130 --> 01:06:45.345
what do humans do now. So, thank you for all the time and energy you're putting into that, and I hope you enjoy the rest of your day.
976
01:06:45.825 --> 01:06:47.345
Thanks, Tobias. It's been a pleasure.
977
01:06:55.100 --> 01:06:59.580
Thank you for listening, and don't forget to check out our other shows. Podcast.net
978
01:06:59.580 --> 01:07:08.460
covers the Python language, its community, and the innovative ways it is being used. And the AI engineering podcast is your guide to the fast moving world of building AI systems.
979
01:07:09.325 --> 01:07:19.325
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a project from the show, then tell us about it. Email hosts@dataengineeringpodcast.com
980
01:07:19.325 --> 01:07:25.565
with your story. Just to help other people find the show, please leave a review on Apple podcasts and tell your friends and coworkers.