TRANSCRIPTIE 🔗
WEBVTT
1
00:00:00.271 --> 00:00:03.413
Today we look at the Dark Software Factory with Justin McCarthy.
2
00:00:03.553 --> 00:00:12.499
Justin spent a decade as CTO of StrongDM building the security plumbing Fortune 500 enterprises run on, where correctness is not optional.
3
00:00:12.639 --> 00:00:13.960
He now runs Diffusion.
4
00:00:13.980 --> 00:00:19.784
Dan Shapiro coined that term Dark Software Factory after watching how Justin's team works.
5
00:00:20.284 --> 00:00:23.105
It's a factory that writes software with nobody on the floor.
6
00:00:23.386 --> 00:00:24.106
The lights are off.
7
00:00:24.326 --> 00:00:38.353
By the end of this episode, you will know how he writes a spec an agent can execute, why he runs 10 reviews across different models instead of asking one model twice, how his team ended up building their own Okta, their own Slack, their own Jira, just to have something to test against.
8
00:00:38.833 --> 00:00:40.874
It is one test for whether any of it's working.
9
00:00:43.423 --> 00:00:43.983
Thank you, Sam.
10
00:00:44.003 --> 00:00:45.904
Thanks for the introduction and thanks for the invite.
11
00:00:46.184 --> 00:00:52.926
You spent a decade as CTO at StrongDM building the plumbing that controls who can touch what inside these enterprises.
12
00:00:53.466 --> 00:01:00.209
And I think that was part of what was so impressive to me and my team when you demoed your agentic framework.
13
00:01:00.549 --> 00:01:11.953
Can you say a little bit about what gave you the confidence to deploy AI generated code that you are not inspecting and looking at personally to enterprises in a security context?
14
00:01:12.971 --> 00:01:19.817
So one of the great things about cybersecurity is just that it is a domain where correctness isn't optional, right?
15
00:01:19.837 --> 00:01:29.406
You could argue that there's the life safety systems, you know, the pacemakers, and then maybe right below that are some of the cybersecurity controls that we rely on to protect all of our important information.
16
00:01:29.946 --> 00:01:35.889
So that domain, it tends to bring about a mindset that is a quite serious mindset.
17
00:01:35.949 --> 00:01:39.791
And let's say it's maybe the opposite of a vibe coding mindset.
18
00:01:39.971 --> 00:01:44.093
You're not trying to, you know, feel in all cases, like feel out what things could be.
19
00:01:44.113 --> 00:01:50.316
You're trying to sort of anneal and prove and toughen and harden a piece that you can have confidence in.
20
00:01:50.357 --> 00:01:56.940
Just nobody wants to be woken up in the middle of the night for any problems relating to your authorization and authentication systems.
21
00:01:57.520 --> 00:02:04.706
So I'll just note that, you know, that was really the backdrop and the background for the initial investigations that myself and our team did.
22
00:02:05.326 --> 00:02:26.323
And in terms of building confidence, this is where the analogy to how we've always built software, I think just, again, I would say, because I guess some substantially recognize this, but it bears repeating, you know, if you are a VP of X at your company, you're not placing an ad directly, you're not, you know, writing the copy directly, you're not writing the code directly.
23
00:02:26.703 --> 00:02:35.789
You're participating in a hierarchy of, you know, maybe non-deterministic agents that are pursuing the goals through multiple possible trajectories.
24
00:02:35.929 --> 00:02:39.551
And then through review cycles, you're building confidence that you're doing things the right way.
25
00:02:40.151 --> 00:02:46.975
Any compliance regime, for example, a cybersecurity or financial compliance regime, they all rely on sampling, right?
26
00:02:47.015 --> 00:02:48.796
They all rely on inspection.
27
00:02:49.297 --> 00:02:54.360
They all rely on a little bit of process and stating what your process is and then following your process.
28
00:02:54.820 --> 00:02:58.805
So I just note that that's, for example, like the SOC 2 compliance process.
29
00:02:59.285 --> 00:03:07.775
It's very familiar that you write down what you're going to do, what routines and ceremonies you're going to do to build confidence, and then you follow those routines.
30
00:03:08.275 --> 00:03:11.959
So I would just say that directly applies to how we manage agents.
31
00:03:12.120 --> 00:03:12.340
Yeah.
32
00:03:12.500 --> 00:03:15.803
you still have to have some form of adversarial review.
33
00:03:15.843 --> 00:03:20.668
You still have to have some form of sampling of what's actually happened in the system.
34
00:03:20.988 --> 00:03:29.977
And then you need to review those samples in a way that can roll up and check some feasibly for whoever is running that inquiry.
35
00:03:30.177 --> 00:03:40.568
So it's really all about the review process, the feedback loop, and the validation that gives you the confidence in deploying the system.
36
00:03:41.108 --> 00:03:41.809
Is that right?
37
00:03:41.869 --> 00:03:51.379
Rather than inspecting and determining through deduction that, hey, if this system is designed this way, then it's going to produce the intended output.
38
00:03:52.102 --> 00:03:58.726
Yeah, and we may end up talking a little bit about formal systems because there are systems that are amenable to formalization, right?
39
00:03:58.766 --> 00:04:00.186
So you can actually have mathematical proofs.
40
00:04:00.246 --> 00:04:06.710
But sort of for everything else, this is just some sort of a hierarchy of inspection and some sort of a hierarchy of review.
41
00:04:06.970 --> 00:04:11.593
But what I'll note is, and this is something, again, folks should recognize this from working with agents.
42
00:04:12.033 --> 00:04:19.697
If you're in one context and you ask the agent if its ideas are good, its ideas in that context are always the best thing since sliced bread.
43
00:04:20.017 --> 00:04:32.804
So the idea that Claude and Codex just thought of, best idea ever, you clear that
Load full transcript
context and they are capable of reflecting with a much more skeptical eye on the transcript of that context.
44
00:04:33.185 --> 00:04:45.952
Better still, you switch to another model which has slightly different weights, slightly different training data, slightly different post-tuning, and you're going to get an even richer review that is potentially going to point at something.
45
00:04:46.412 --> 00:04:54.118
The other thing I'll note here is, and this is for me counterintuitive from, you know, how I would have assumed the world would be going a couple of years ago.
46
00:04:54.579 --> 00:05:03.646
You know, there's something about these models that they don't tend to degrade into noise in the presence of more and more successive and iterative apps.
47
00:05:03.646 --> 00:05:04.987
applications of the models.
48
00:05:05.487 --> 00:05:14.793
Somehow, when you put a buffet of 10 options in front of them, they're quite accurate at choosing among those 10 options, which one is the superior option, right?
49
00:05:14.813 --> 00:05:18.156
Which one is the Pareto optimal option of that portfolio.
50
00:05:18.656 --> 00:05:25.260
And so what that means is maybe your review process is, so first off, definitely not just asking the model, hey, do you have a good idea?
51
00:05:25.801 --> 00:05:26.261
That won't work.
52
00:05:27.202 --> 00:05:28.923
You'll get gaslit, you'll get sick fancy.
53
00:05:29.423 --> 00:05:40.488
But if you ask an ensemble of different models with slightly different, you know, slightly different prompts to all focus on this idea and all write a report, pretty soon you're getting five reports, 10 reports, 12 reports.
54
00:05:40.908 --> 00:05:50.293
And within those reports, a synthesis model can look through and really come up with, you know, the high probability findings that are in there and the low probability findings.
55
00:05:50.973 --> 00:05:51.834
create those piles.
56
00:05:51.954 --> 00:05:56.417
And then for high probability finding, then it's not sufficient to say you might have something.
57
00:05:56.697 --> 00:05:58.098
That's where you go into proving it.
58
00:05:58.118 --> 00:05:58.358
Okay.
59
00:05:58.658 --> 00:06:12.148
And so this is where you're going to convert that finding into some sort of an evidence and some sort of a demonstration of a vulnerability, of a weakness, of evidence in a real production log or a real production trajectory.
60
00:06:12.637 --> 00:06:12.837
Yeah.
61
00:06:12.877 --> 00:06:19.281
So actually, can we zoom out a little bit and can you define for me the dark, like the term dark software factory?
62
00:06:19.321 --> 00:06:29.167
Because what you're talking about, I feel like are one or two components of it, but, but it's really an ensemble of components that work together to remove the human from the loop.
63
00:06:29.667 --> 00:06:30.488
Is that right?
64
00:06:31.049 --> 00:06:31.590
Yeah, sure.
65
00:06:31.650 --> 00:06:39.101
So what I would say, and this isn't exactly my
term, so you did credit Dan Shafiro in the intro, so please ping Dan for this term.
66
00:06:39.141 --> 00:06:43.808
But the notion of software factories in general, dark or otherwise, I would say-
67
00:06:44.629 --> 00:06:47.531
But Dan, I feel like is inspired by your work.
68
00:06:47.571 --> 00:06:53.475
And really what I'm most, you know, what I'm trying to give the audience is a sense of what I saw, right?
69
00:06:53.976 --> 00:07:04.023
And which is so much more than, for example, fanning out a query across multiple models with multiple contexts in order to cherry pick kind of the best output and reassemble the best output.
70
00:07:04.503 --> 00:07:13.830
Right, like that may have been one part of what I saw, but there's, you know, the system was much greater than that part and honestly, the sum of all of the parts.
71
00:07:14.130 --> 00:07:33.038
Yeah, so let me just say, you know, all of this, I think it's pretty simply understood as it's just the sort of furious industrialization of what's possible by applying way more tokens than you would expect and really insisting that you close the loop between your intent for a given product and how it's actually running in production.
72
00:07:33.498 --> 00:07:41.104
How it's actually running in production is an essential piece of amplifying and sort of essentially pathfinding back to that intent.
73
00:07:41.585 --> 00:07:44.167
OK, and so that's the loop that you're ultimately closing.
74
00:07:44.347 --> 00:07:51.453
Unfortunately, it has to close not just through text log files, but it has to close through screenshots and images and Zendesk tickets.
75
00:07:51.693 --> 00:07:51.853
Right.
76
00:07:52.153 --> 00:08:01.561
So there's far more real world capture that you have to do in order to converge these systems to what you want, which is a reliable, performance, secure system.
77
00:08:02.041 --> 00:08:02.301
Okay.
78
00:08:02.741 --> 00:08:14.069
And in terms of that intent expression, there are lots of potential forms, but I'll say like they're just, you know, you could argue it's specification, but it's also a pitch, right?
79
00:08:14.409 --> 00:08:15.650
It's a sole document.
80
00:08:15.770 --> 00:08:18.491
It's a reason for being for this particular software piece.
81
00:08:19.052 --> 00:08:24.275
And then you're using the agents to decompose the hill climbing steps toward that vision.
82
00:08:24.825 --> 00:08:29.668
Okay, so give me just some really visceral examples of this.
83
00:08:30.288 --> 00:08:38.093
Because this, I think, is really the key point, is that closing the feedback loop to production, right?
84
00:08:38.734 --> 00:08:44.797
And I think many engineers watching this today are integrating log files into their agentic engineering in some way, right?
85
00:08:44.957 --> 00:08:48.600
You spend 90% of your tokens on validation.
86
00:08:49.320 --> 00:08:50.481
on closing that loop.
87
00:08:50.621 --> 00:08:52.582
Is that still true, would you say?
88
00:08:52.602 --> 00:08:54.003
Yeah, it's still true.
89
00:08:54.083 --> 00:08:57.426
And in each time interval, the models obviously their capability improves.
90
00:08:57.586 --> 00:09:02.509
And so they can sort of, they can hill climb toward your goal, let's say more efficiently, right?
91
00:09:02.630 --> 00:09:09.214
So it's also true that just like our harnesses can get simpler over time, our validation loops can get simpler over time.
92
00:09:09.675 --> 00:09:13.297
But still, that is where the vast majority of your tokens are going to go.
93
00:09:13.317 --> 00:09:16.800
And I'll just use one example that I really like.
94
00:09:17.340 --> 00:09:36.437
In any application that has, let's say, a user interface, whether it's a TUI or a graphical user interface, I no longer ever ship or touch anything that doesn't have some sort of a, whether it's with post-hog, but more likely with your own internal system, some sort of a way of measuring those intervals where the user might be confused.
95
00:09:37.446 --> 00:09:38.707
They might be lingering too long.
96
00:09:39.388 --> 00:09:42.250
It seems like they should be able to submit the next step and they haven't yet.
97
00:09:42.711 --> 00:09:42.971
Okay.
98
00:09:43.452 --> 00:09:50.398
And really zooming in on that specific detail and that aspect of interacting with software and asking, why aren't they clicking play yet?
99
00:09:50.678 --> 00:09:51.759
Why aren't they clicking go?
100
00:09:51.919 --> 00:09:54.021
Like, why aren't they using this software?
101
00:09:54.181 --> 00:09:56.263
And really making that itself part of the closed loop.
102
00:09:56.583 --> 00:09:56.864
Okay.
103
00:09:56.964 --> 00:09:59.726
So we're going to have an intent that's unfulfilled.
104
00:10:00.567 --> 00:10:20.825
right an expression of a wish that that isn't satisfied in in you the use of our software well that metric stream is is an equal party to the other requirements that you have going into the next cycle the next sprint and so on right so and up to and including screenshot the thing capture the mouse movement right like
105
00:10:21.545 --> 00:10:32.121
That's the level of richness that you ultimately want to provide, remembering that these models are multimodal and remembering that like a single screenshot is worth a thousand words for these models.
106
00:10:32.382 --> 00:10:36.408
OK, so give them a screenshot, make it easy to do that, make it a part of your product.
107
00:10:37.183 --> 00:10:44.009
Can you talk a little bit about DTU and that work at StrongDM as kind of an example of closing this loop?
108
00:10:44.449 --> 00:10:44.990
Yeah, sure.
109
00:10:45.270 --> 00:10:51.596
So I actually noticed this is increasingly becoming a pretty standard practice with other folks that I talk to.
110
00:10:51.816 --> 00:10:56.560
Really, this is about inverting habits around our.
111
00:10:56.940 --> 00:10:58.501
standard of evidence.
112
00:10:58.822 --> 00:10:59.102
Okay.
113
00:10:59.562 --> 00:11:05.587
So we used to maybe be satisfied with the CICD system giving us a green check after a couple of minutes.
114
00:11:06.147 --> 00:11:09.750
And that was one standard of evidence that we were ready to ship to production.
115
00:11:09.910 --> 00:11:12.472
And it was nice to have those concise and fast and so forth.
116
00:11:12.873 --> 00:11:13.813
But you know what?
117
00:11:14.073 --> 00:11:19.538
There's a moment, especially when the agents are participating, where, you know, maybe we want to use it ourselves.
118
00:11:19.598 --> 00:11:20.979
Maybe we want to click around ourselves.
119
00:11:21.039 --> 00:11:24.862
Maybe we want to actually, you know, upload and then get a report back.
120
00:11:25.242 --> 00:11:35.190
So for those interactions that are a little bit bigger, that are intuitive to us, sometimes those involve, you know, interacting with Salesforce.com or interacting with a Google spreadsheet or something like that.
121
00:11:35.611 --> 00:11:42.797
So for those kinds of interactions that would constitute evidence to us that the application was working, they're really, really big end-to-end tests.
122
00:11:43.437 --> 00:11:49.482
And so we could elevate our ambitions and say, what if I could run a thousand of those tests or 10,000 or 100,000 per day?
123
00:11:49.742 --> 00:11:50.522
That sounds good.
124
00:11:51.023 --> 00:11:56.207
But eventually you encounter integrations that are unwilling to let you run at that speed, right?
125
00:11:56.267 --> 00:12:01.170
So Stripe won't let you actually process that many credit cards through the real production pipeline.
126
00:12:01.310 --> 00:12:04.533
But you want to know that the real payments are being processed, right?
127
00:12:05.033 --> 00:12:10.916
So for each of those integrations, this is another just growing out of our own old habits.
128
00:12:11.336 --> 00:12:21.641
You know, for us, it began with the question, you know, could we create something that looked enough like Slack that we could have valid agentic Slack interactions?
129
00:12:21.901 --> 00:12:22.142
Okay.
130
00:12:22.522 --> 00:12:32.947
And then once we had that and it was satisfying, we asked, well, could we create enough, let's say, Okta so that we had a shared identity between Okta and Slack and maybe Jira?
131
00:12:33.347 --> 00:12:33.587
Okay.
132
00:12:33.888 --> 00:12:39.252
And then we could begin to create these synthetic organizations with consistent identity that had like long term history.
133
00:12:39.572 --> 00:12:39.792
Okay.
134
00:12:40.273 --> 00:12:47.218
And you keep compounding that question until ultimately this accumulated into what we ultimately called the digital twin universe.
135
00:12:47.458 --> 00:12:51.402
So from a technical perspective, it was a single Unix process, right?
136
00:12:51.422 --> 00:12:53.563
So you would have one, it was written in Go mostly.
137
00:12:53.944 --> 00:13:03.031
So you would have one Go process, but inside of there, all of the memory that constituted all of the event transitions for all of these virtual entities in this world were,
138
00:13:03.271 --> 00:13:06.595
for a whole suite of integrated applications was all embedded in there.
139
00:13:07.276 --> 00:13:07.456
Right.
140
00:13:07.576 --> 00:13:23.355
So as a result, you could share a Google sheet over Slack, update the Google sheet in, you know, with another virtual user and have both of those events recorded snapshot, replayable, reversible, everything you could possibly want about that, about that virtualized digital twin.
141
00:13:24.070 --> 00:13:31.056
So just to make sure I'm understanding, what I'm hearing you say is you are building a product that integrated into these other third party services.
142
00:13:31.556 --> 00:13:38.061
And to move quickly, you could not close the loop with the third party services themselves without hitting rate limits.
143
00:13:38.582 --> 00:13:46.148
And so you naturally recreated those third party services purely for your agents to interact with and use your product.
144
00:13:46.428 --> 00:13:51.853
And so you almost created an enterprise of agents that are using your product continuously
145
00:13:52.533 --> 00:14:02.917
hammering it, experimenting, integrating, testing the integrations with these different third party services that are not actually the API endpoints are not actually slack.com or whatever.
146
00:14:02.957 --> 00:14:07.719
They're your own internal API endpoints, but they behave exactly as you would expect slack.com to behave.
147
00:14:07.779 --> 00:14:10.861
And you can do this because writing code is so cheap today.
148
00:14:10.921 --> 00:14:13.722
And so what's expensive is the spec and the validation.
149
00:14:14.162 --> 00:14:16.083
And so slack has already been validated.
150
00:14:16.103 --> 00:14:20.545
So you can say, okay, go build me a set of slack APIs because I want to test something
151
00:14:21.065 --> 00:14:22.566
And I want to test how it works against Slack.
152
00:14:23.127 --> 00:14:23.327
Yes.
153
00:14:23.867 --> 00:14:30.753
And I do want to also say, you know, in a lot of cases, it's not even about speed of the rate or rate limits or something like that.
154
00:14:30.953 --> 00:14:36.617
If you go to Okta and say, I want to start 10,000 synthetic companies, they simply won't let you.
155
00:14:37.298 --> 00:14:37.438
Yeah.
156
00:14:37.618 --> 00:14:37.878
Right.
157
00:14:37.958 --> 00:14:42.562
So there are fraud and business strategy reasons why they simply will not let you do that.
158
00:14:43.082 --> 00:14:43.282
Right.
159
00:14:43.622 --> 00:14:44.743
Same thing for Salesforce.
160
00:14:44.843 --> 00:14:48.506
You can't go to Salesforce and say, can I please have 100,000 sandboxes?
161
00:14:49.067 --> 00:14:58.114
But many of these scenarios entail that level of identity complexity because you're simulating real enterprises and companies actually collaborating with each other.
162
00:14:58.774 --> 00:14:58.934
Right.
163
00:14:59.055 --> 00:15:03.298
And so you need that sort of origin of identity to exist in your digital twin universe.
164
00:15:03.700 --> 00:15:15.052
I'm wondering, is there something that we can actually see to really understand this kind of closed loop behavior and how it's different than maybe how a typical engineer might be using goals and cloud code or something like that?
165
00:15:15.773 --> 00:15:16.194
Sure.
166
00:15:16.654 --> 00:15:22.481
So I did prepare one example, and this is something, you know, a couple of folks have come up with problems related to this recently.
167
00:15:22.521 --> 00:15:24.122
So I'll just I'll just set up the problem.
168
00:15:25.223 --> 00:15:39.653
So obviously, even though the models are multimodal, even though they're beginning to have understanding of like what's like video scenes, okay, they actually have very limited reasoning, let's say spatial reasoning, but then also down to the more...
169
00:15:40.113 --> 00:15:46.094
let's say more tight tolerance notion of spatial reasoning, which is like something like 3D printing or CNC.
170
00:15:46.595 --> 00:15:46.835
Okay.
171
00:15:47.255 --> 00:15:55.837
So when you're dealing with a machined piece or an additive manufacturing piece, there are, you know, there are a million different concerns.
172
00:15:56.237 --> 00:16:09.100
And there are, there's a lot of prior art in computer design technology that helps to, for example, that whether you're, you know, your, your piece has enough thickness to conceivably stand, you know, stand on its own.
173
00:16:09.380 --> 00:16:09.580
Right.
174
00:16:09.660 --> 00:16:16.642
So those sort of tools are, you know, they're built into CAD and machine tools to build into slicers in the additive manufacturing universe.
175
00:16:16.782 --> 00:16:19.043
So all those tools are really important building blocks.
176
00:16:19.443 --> 00:16:26.005
But, you know, as a, you know, if you're just getting into manufacturing for the first time, you often just want to blurt out like, hey, make me a robot.
177
00:16:26.705 --> 00:16:26.925
Right.
178
00:16:27.005 --> 00:16:32.589
So, you know, design a robot, but I wanted to have cool floppy ears like my dog.
179
00:16:33.130 --> 00:16:37.713
And so and so so that sort of an utterance, we can make that utterance.
180
00:16:38.313 --> 00:16:44.218
And we all have this expectation that the models are somehow going to be able to help us with that because they help us with everything else.
181
00:16:44.458 --> 00:16:50.722
OK. And I'll just say you hit them all very, very quickly when you ask for a cool dog shaped robot with floppy ears.
182
00:16:50.963 --> 00:16:51.203
OK. Yeah.
183
00:16:51.933 --> 00:16:58.655
Now, the other thing that I think we all feel is increasingly these models do have expertise in a lot of domains.
184
00:16:58.776 --> 00:17:01.316
And increasingly, obviously, they know how to write code.
185
00:17:01.797 --> 00:17:13.621
So what I find is when you encounter this domain where it feels like we're almost there, but we're not quite there, what's often missing is a small, you could call it a prosthetic, or you could call it just sort of like a learning aid.
186
00:17:13.801 --> 00:17:15.642
But something that brings the domain knowledge.
187
00:17:16.542 --> 00:17:20.246
into a projection that the model can understand and work with.
188
00:17:20.746 --> 00:17:25.891
And so a lot of times you hit a limit on what you think is possible with agentic engineering.
189
00:17:26.352 --> 00:17:32.597
And that limit is in fact, just related to the fact that their, you know, their vision is blurry in, you know, some aspect of accounting.
190
00:17:32.878 --> 00:17:37.102
Their vision is blurry in some aspect of, in this case, you know, dealing with STL files.
191
00:17:37.882 --> 00:17:44.068
When you say prosthetic, are you referring to like a constraint on the language model itself?
192
00:17:44.088 --> 00:17:53.596
Or are you referring more to what we were just discussing around the lack of context and let's say the digital twin universe as a way of connecting that context into the...
193
00:17:54.377 --> 00:17:55.518
into the decision loop.
194
00:17:56.118 --> 00:18:03.702
So when we ask for the model to produce anything, you know, we are, whether we know it or not, relying on the weights that have already been baked in, right?
195
00:18:04.162 --> 00:18:10.005
But we really increasingly want to move, if we're, you know, the closer and closer we're getting to more than a vibe-coded web page,
196
00:18:10.825 --> 00:18:23.709
We have to provide a token context where that model is not relying on its memories of 3D printers, but instead it has tools in its hand that are related to proving stuff with 3D printers.
197
00:18:24.809 --> 00:18:26.409
And so we know this is tool calling.
198
00:18:26.569 --> 00:18:30.631
We kind of know this is MCP, but really it's all of the above.
199
00:18:30.891 --> 00:18:35.132
It's a toolkit that just makes everything close at hand for the model.
200
00:18:35.932 --> 00:18:42.257
And it makes it so that it can turn something, which is, you know, maybe it's a video, maybe it's audio, maybe it's an image.
201
00:18:42.617 --> 00:18:44.198
They're really good at looking at images.
202
00:18:44.478 --> 00:18:45.499
So set that aside.
203
00:18:45.899 --> 00:18:47.140
But maybe it's a file format.
204
00:18:47.320 --> 00:18:52.344
I actually, I have a scan of my home that is a LiDAR point cloud.
205
00:18:52.624 --> 00:18:52.904
Okay.
206
00:18:53.284 --> 00:18:57.287
And so that five gigabyte LiDAR point cloud, you can't drop it into cloud code.
207
00:18:57.968 --> 00:18:58.168
Okay.
208
00:18:58.588 --> 00:19:03.672
So you need some mapping functions to convert that into something that they can see and work with.
209
00:19:04.172 --> 00:19:11.340
So one way I do that is whether you want to call it a prosthetic or a sort of, you can even think of it as a mech suit or
something like that.
210
00:19:11.380 --> 00:19:12.742
It's a pure augmentation.
211
00:19:13.042 --> 00:19:22.132
But we're augmenting the strengths of the models and we're making it so that they can decide, hey, I want to map this object into a way that I can see it.
212
00:19:22.312 --> 00:19:25.696
And then write a little Python code to mutate it and send that Python code back to the object.
213
00:19:26.055 --> 00:19:28.297
Okay, so you're framing the problem.
214
00:19:28.677 --> 00:19:31.860
I'm so curious what the solution here, what you're going to demo.
215
00:19:32.561 --> 00:19:37.345
Okay, so something I threw together, and you can see from the commit history, this actually came together just last night.
216
00:19:37.545 --> 00:19:47.213
Really, this is just accumulating a set of, let's say, deterministic tools that are aids to the process of, let's say, seeing in 3D from the model's perspective.
217
00:19:47.433 --> 00:19:51.275
OK, so all of us, we have human intuition about 3D.
218
00:19:51.315 --> 00:19:53.176
We have human intuition about physics.
219
00:19:53.216 --> 00:19:54.737
The models have limited intuition.
220
00:19:55.097 --> 00:19:57.538
So we're just supplementing their intuition.
221
00:19:57.919 --> 00:20:05.423
OK, so CCAD is not just a so you can see in here that there is a there there's an agents MD.
222
00:20:06.063 --> 00:20:08.784
that is essentially explaining how to use this project.
223
00:20:09.344 --> 00:20:19.588
But also it has a collection of deterministic tools that can help perform analyses and different kinds of projection on the model objects.
224
00:20:20.208 --> 00:20:24.290
And I just wanted to throw this together because obviously I knew we were talking today.
225
00:20:24.790 --> 00:20:31.533
And also part of this agentic moment is at every interval and in every direction, just increasing your ambition
226
00:20:32.233 --> 00:20:33.574
you know, at every opportunity.
227
00:20:33.734 --> 00:20:58.146
And so what I wanted to just, you know, call out is that, you know, with really not thinking about this much at all, other than giving it the direction of go toward a toolkit that is going to assist with 3D, you know, manipulation, 3D reasoning, I was able to put together the system that now seems to be helpful in, you know, let's say, simple agentic 3D design and validation tasks.
228
00:20:58.912 --> 00:20:59.292
Got it.
229
00:20:59.332 --> 00:21:04.775
So you're kind of making the point that's not like dark software factories don't need to be that intimidating, right?
230
00:21:04.855 --> 00:21:06.396
Like if you happen to be.
231
00:21:06.416 --> 00:21:07.876
Yeah, they absolutely don't.
232
00:21:07.916 --> 00:21:13.819
Like and so ideally what you're looking at here is you're looking at one node in your software factory.
233
00:21:13.839 --> 00:21:18.282
So let's say you had and actually I have another I have another system open here.
234
00:21:18.322 --> 00:21:21.563
So like a famous CAD
environment is Tinkercad.
235
00:21:21.583 --> 00:21:25.085
So this is like the most basic fun one that a lot of folks enjoy.
236
00:21:25.565 --> 00:21:28.288
So in Tinkercad here, I have a printable micrometer.
237
00:21:28.588 --> 00:21:28.828
Okay.
238
00:21:29.289 --> 00:21:41.300
So if I were on the Tinkercad team right now, I might create something like CCAD to aid the further production of the next version of Tinkercad, right?
239
00:21:41.680 --> 00:21:45.243
Because what we're going to do is we're going to use CCAD to bring...
240
00:21:46.084 --> 00:21:50.868
Claude and Codex closer to, let's say, intuitive rapid speed.
241
00:21:51.449 --> 00:21:57.393
So like, for example, the way they can write out, you know, quick sort, right, they'll just blurt it out, and it'll be correct.
242
00:21:57.854 --> 00:22:05.160
We want to get closer to that speed for a model that doesn't otherwise have, you know, a lot of 3d reasoning built into the weights.
243
00:22:05.460 --> 00:22:12.763
And so something like CCAD is, it's almost like a domain harness almost for 3D, right?
244
00:22:13.024 --> 00:22:18.026
And then so you're coupling that to a coding harness like Codex or Cursor.
245
00:22:18.466 --> 00:22:29.651
And those coding harnesses together with the domain harness are going to allow the agent to make moves that it would, you know, maybe eventually would get there, but it's going to be able to go much straighter at the target.
246
00:22:30.092 --> 00:22:30.392
Got it.
247
00:22:31.012 --> 00:22:36.715
So just to illustrate a little bit of this, I have a couple of objects here in my TakerCAD.
248
00:22:36.735 --> 00:22:42.498
So let's actually download as an object file here.
249
00:22:42.918 --> 00:22:46.860
We're going to download the little automated ball sorter.
250
00:22:47.100 --> 00:22:47.340
Okay.
251
00:22:47.780 --> 00:22:50.962
So in the automated ball sorter, we're now going to go to...
252
00:22:51.782 --> 00:22:53.763
And you'll see that I'm using Codex.
253
00:22:53.863 --> 00:22:57.745
I actually daily drive essentially all of the harnesses.
254
00:22:58.366 --> 00:23:06.210
This is part of my practice is to regularly switch between all of them so that I'm staying fluent and flexible.
255
00:23:06.530 --> 00:23:09.191
All right, we've got the ball sorter.
256
00:23:09.371 --> 00:23:11.393
We're going to go ahead and grab that zip file.
257
00:23:11.853 --> 00:23:21.598
And what you'll notice here is that we now have some local rendering of exactly what we were seeing a moment ago over in Tinkercad, including, for example, this red ball
258
00:23:21.718 --> 00:23:23.879
that sort of offset from that tray there.
259
00:23:24.219 --> 00:23:27.181
Okay, so you just created this CCAD, right?
260
00:23:27.441 --> 00:23:34.224
So what we're seeing here was just kind of one shot, I'm assuming, created by Codex, is that right?
261
00:23:34.424 --> 00:23:41.768
Yeah, and what you can one shot versus what you want out of a software factory are two very different things.
262
00:23:42.348 --> 00:23:45.791
However, they're building blocks toward that software factory, right?
263
00:23:45.991 --> 00:23:46.391
Right.
264
00:23:46.551 --> 00:23:59.581
So this, like this, you know, fully functional CAD viewing application, and as you'll see in a moment, like design vetting application, this is a primitive building block in your, you know, 3D printing oriented software factory.
265
00:24:00.542 --> 00:24:09.309
And the fact that this is essentially one-shotable overnight, like that's another way where we have to be updating our expectations, you know, every week and every month, right?
266
00:24:09.329 --> 00:24:09.509
Right.
267
00:24:09.609 --> 00:24:11.832
This is like what was groundbreaking at the time.
268
00:24:12.353 --> 00:24:18.340
And it wasn't one shot, but one shotting Slack APIs or Okta's APIs because you're building on top of that.
269
00:24:18.381 --> 00:24:22.926
And so you're just going to instead of trying to convince Okta to open up their APIs, you're just going to one shot them.
270
00:24:23.578 --> 00:24:27.019
Okay, so this seems similar to what you were just talking about as far as one shot.
271
00:24:27.079 --> 00:24:34.621
And it wasn't one shot at the time, but building the Slack API is the Okta API is you wanted to build on top of those API is you wanted some sort of validation framework around it.
272
00:24:34.661 --> 00:24:43.663
And so because language models can recreate a spec so quickly, it was feasible for you to do that, whereas a year ago or two years ago, it wouldn't have been.
273
00:24:43.943 --> 00:24:45.444
Yeah, yes, exactly.
274
00:24:45.664 --> 00:24:51.286
And so, you know, we are not constrained to the software that happens to be available off the shelf.
275
00:24:51.886 --> 00:24:57.428
We're not constrained to the visualization and the projection that, you know, someone else cooked up for their business or their goals.
276
00:24:57.808 --> 00:25:02.450
We can have all of these elements tailored to our needs, our business, our application that we're developing.
277
00:25:02.530 --> 00:25:16.280
And so, yeah, this is just an example where I want a building block in my 3D printing software factory that allows me to, you know, fluently go back and forth between off the shelf CAD tools, download them in the native formats, visualize them in the native formats.
278
00:25:16.620 --> 00:25:20.603
And then ultimately what I'm trying to do here, this is part of the maybe human in the loop.
279
00:25:20.943 --> 00:25:21.323
part of it.
280
00:25:22.104 --> 00:25:25.206
But what I want to do is I want to go back to the application.
281
00:25:25.387 --> 00:25:28.369
And this is actually something where I haven't tested it.
282
00:25:28.729 --> 00:25:35.414
But let's say actually I want to move this red ball here, let's say, onto the blue track.
283
00:25:36.175 --> 00:25:41.219
OK, I haven't tested something like that, but that's conceivably
something that's obviously an utterance I can make in English.
284
00:25:41.419 --> 00:25:48.164
OK, and so if CCAD, the agentic toolkit is is, you know, if it's doing its job, it's conceivable.
285
00:25:48.184 --> 00:25:55.150
I haven't tested it, but it's conceivable that we've infused the environment with enough tooling to enable something like that.
286
00:25:55.350 --> 00:25:58.812
OK, so without testing that, I'm going to just go ahead and try and YOLO it.
287
00:25:59.113 --> 00:26:00.654
So let's bring us back up to.
288
00:26:00.854 --> 00:26:02.335
Yeah, we can be on that one.
289
00:26:02.455 --> 00:26:03.055
extra high there.
290
00:26:03.416 --> 00:26:07.078
So, um, please use, uh, what did we call the file?
291
00:26:07.178 --> 00:26:09.999
It was, uh, it was automatic ball sorter.zip.
292
00:26:10.220 --> 00:26:10.480
Okay.
293
00:26:10.940 --> 00:26:23.948
Um, so we've got downloads, um, and move the, you know, independent red sphere, um, uh, onto any of the,
294
00:26:25.315 --> 00:26:27.798
or onto the blue ball tray, let's say.
295
00:26:28.439 --> 00:26:32.865
The blue ball tray, and then create a new zip.
296
00:26:35.248 --> 00:26:36.629
All the new objects.
297
00:26:37.701 --> 00:26:53.848
So the hope of the CCAD toolchain is that, and I have no idea if I have, but let's say by tonight, I will have infused it with enough capability that a task like this is inevitable and is obvious.
298
00:26:54.468 --> 00:27:00.411
While we're waiting for that, I also want to just show a couple of other examples.
299
00:27:00.591 --> 00:27:03.092
So I've got an intersection here
300
00:27:03.672 --> 00:27:07.016
here, which might represent, like, let's say this was an errant intersection.
301
00:27:07.276 --> 00:27:07.497
Okay.
302
00:27:07.917 --> 00:27:13.283
So this is something that, you know, maybe we, maybe we didn't want these objects to touch each other and intersect in this way.
303
00:27:13.784 --> 00:27:14.024
Okay.
304
00:27:14.425 --> 00:27:20.992
So this is another case where, you know, I'm going to download the file and then I just call that one oops intersection.
305
00:27:21.072 --> 00:27:23.595
So let's, let's grab that and just say.
306
00:27:23.615 --> 00:27:23.715
Okay.
307
00:27:24.316 --> 00:27:31.902
back here, please characterize the intersection, if any, in this.
308
00:27:32.983 --> 00:27:38.267
So same thing, we're going to have a little bit of a sense of, you know, are these objects overlapping?
309
00:27:38.467 --> 00:27:40.929
And then we can have a discussion about whether we intend to or not.
310
00:27:41.450 --> 00:27:44.332
CCAD is going to provide the tooling for that.
311
00:27:44.962 --> 00:28:01.934
Okay, while these are working, when you are day-to-day pursuing some sort of business outcome, what is sort of the heuristic they use to determine, okay, this is a problem with the design of the factory, the validation, the prosthetic, or, oh, this is, I'm actually just not prompting the language model in the correct way.
312
00:28:02.074 --> 00:28:06.157
Like, this is more of a legacy traditional problem that a vibe coder might...
313
00:28:06.777 --> 00:28:08.258
might already be.
314
00:28:08.278 --> 00:28:08.658
Sure.
315
00:28:09.078 --> 00:28:20.886
So first thing, when we're going closer and closer to production in real world, even though we're prompting, you know, you're trying not to do anything that feels like prompting because what you're trying to do is you're trying to do loop engineering.
316
00:28:20.906 --> 00:28:30.271
So you're trying to have some final state articulated, sufficiently articulated in your repository so that I like to say success is inevitable.
317
00:28:30.712 --> 00:28:35.795
So in the presence of an agent and that repository, it will become what it intends to become.
318
00:28:36.315 --> 00:28:41.157
OK, sometimes the agent will block on tokens that aren't in that repository.
319
00:28:41.197 --> 00:28:45.358
So some of the tokens may need to come from Zendesk or may come from GitHub issues.
320
00:28:45.558 --> 00:28:51.400
OK, so some of the inputs that needs to take a next step, those tokens have to come from elsewhere.
321
00:28:52.080 --> 00:28:52.340
Okay.
322
00:28:52.500 --> 00:28:57.803
But for sure, what I'm doing now, where you're like a REPL, you're typing in one sentence at a time.
323
00:28:58.223 --> 00:29:00.244
This is not factory engineering.
324
00:29:00.284 --> 00:29:06.787
Factory engineering is articulating destination state and then letting the system work backwards from that state.
325
00:29:06.827 --> 00:29:12.189
To your question on when you know, like what, which component of that loop to when you know that a problem.
326
00:29:12.649 --> 00:29:20.153
It essentially comes down to like any other thing, if the token budget is going too high or the wall time is going too high and we're not getting to a destination.
327
00:29:20.733 --> 00:29:23.855
The one thing that shouldn't be indicative is bugs.
328
00:29:24.095 --> 00:29:26.816
So bugs and errors, those you should never see.
329
00:29:27.256 --> 00:29:30.258
What you should see instead is loops taking forever.
330
00:29:30.838 --> 00:29:36.121
Because if you're seeing bugs, then you failed to declare a definition of done that's strong enough.
331
00:29:36.461 --> 00:29:45.987
OK. And so but once you strengthen those definitions of done to a very high standard, then you're at risk of, let's say, letting the models spin forever.
332
00:29:46.467 --> 00:29:52.110
This is particularly evident in both Fable and actually the ultra thinking level on 5.6.
333
00:29:52.451 --> 00:29:54.652
They can grind for a very, very, very long time.
334
00:29:55.192 --> 00:29:56.693
And sometimes that's exactly what you want.
335
00:29:57.053 --> 00:30:00.015
Sometimes that's an indication you need to be tuning the token environment.
336
00:30:00.275 --> 00:30:01.055
Say a little bit more.
337
00:30:01.155 --> 00:30:04.376
So this is not software
factory engineering.
338
00:30:04.996 --> 00:30:14.098
And what I'm hearing when you say that is this is this is one level in the software factory where we are bringing feedback into that one level.
339
00:30:14.878 --> 00:30:19.660
However, you don't need to be operating as a human one level above that providing input.
340
00:30:19.680 --> 00:30:21.900
You can actually on that level also create a loop.
341
00:30:22.040 --> 00:30:28.482
Can you talk a little bit about those different levels and what different prosthetics or pieces of feedback might look like
342
00:30:30.222 --> 00:30:34.144
So this is a fun case, right, where I had no idea if it was going to succeed at that.
343
00:30:34.424 --> 00:30:42.108
And we've given the agent sort of enough tooling in CCAD that it was able to, you know, you could argue like, hey, the ball's kind of hovering there.
344
00:30:42.188 --> 00:30:43.469
Maybe their balls are hovering too.
345
00:30:43.789 --> 00:30:46.931
But, you know, it seemed to vaguely comply with our instruction.
346
00:30:47.471 --> 00:31:04.229
So in terms of other ways of closing loops, what I'll say is this is a case where engineers have to get out of their comfort zone and out of their chair in some cases and ask, how can we know more reality?
347
00:31:04.269 --> 00:31:07.272
How can we ground more reality and then close the loop back?
348
00:31:07.532 --> 00:31:07.972
at.
349
00:31:07.992 --> 00:31:08.192
Okay.
350
00:31:08.873 --> 00:31:13.155
And so what that means is like, we might have to have far more calls with customers.
351
00:31:13.735 --> 00:31:15.696
Maybe we don't need to write notes after those.
352
00:31:15.776 --> 00:31:19.637
Maybe we don't need to compile notes, but we need to ask dozens of questions.
353
00:31:19.958 --> 00:31:35.244
And those questions have to be asked, let's say in an environment of trust where you're building trust with that customer, or they need to be asked systematically in bulk in the form of a survey, or maybe the questions aren't the things you want answers to because customers are going to say yes or say no, but neither of those are right.
354
00:31:35.465 --> 00:31:36.185
Maybe is maybe the
355
00:31:36.665 --> 00:31:44.647
And so for getting at those much more objective examples, maybe you should be writing some ad copy and maybe you should be publishing some ad campaigns.
356
00:31:45.107 --> 00:31:49.128
You know, maybe it's advertising an aspect of your product that you think you should build and you haven't built yet.
357
00:31:49.428 --> 00:31:56.370
And maybe you're finding a way to project some facet of that into an appealing ad to get some engagement or fail to get some engagement.
358
00:31:56.550 --> 00:32:00.953
So you have to have creativity in ways of testing and accessing tokens.
359
00:32:01.353 --> 00:32:06.897
And that creativity, it just has to extend in every direction possible, well outside of your Git repository.
360
00:32:07.458 --> 00:32:12.341
And when you say
publishing, maybe you should be publishing Facebook ads.
361
00:32:12.381 --> 00:32:14.943
What you're really talking about is maybe you should be building an agentic loop.
362
00:32:16.924 --> 00:32:28.496
In that stack, that author's ad copy publishes the ad, observes the behavior in the wild, and then uses that observation to close the loop and say, decide, oh, the customer wants the ball on the blue track.
363
00:32:28.856 --> 00:32:32.540
We are then going to move down one level in the stack and actually implement that in the cat design.
364
00:32:32.560 --> 00:32:32.880
Correct.
365
00:32:33.641 --> 00:32:46.265
Yeah, and I'll just say historically, we've had a little bit of, let's say, role specialization in organizations where, you know, Team A commonly writes software and Team B commonly, like, launches marketing campaigns.
366
00:32:46.806 --> 00:32:56.589
And I will just say, like, that segregation, while it was helpful for certain kinds of tasks, it's unhelpful when what you're trying to do is discover the right answer quickly.
367
00:32:57.129 --> 00:33:03.735
When you're trying to discover the right answer quickly, you have to use every tactic at your disposal, you know, within the envelope of the law.
368
00:33:04.335 --> 00:33:04.535
Right.
369
00:33:04.856 --> 00:33:07.398
And so you've got to get those campaigns out there.
370
00:33:07.738 --> 00:33:08.759
You've got to do sales.
371
00:33:08.799 --> 00:33:10.460
You've got to change the marketing copy.
372
00:33:11.321 --> 00:33:12.522
You've got to do all this stuff.
373
00:33:12.742 --> 00:33:15.805
Otherwise, you know, someone else is going to find that niche faster than you.
374
00:33:16.285 --> 00:33:17.706
So, Justin, last question for you.
375
00:33:17.746 --> 00:33:20.569
I know we're at time and I want to I want to be respectful of your time.
376
00:33:21.350 --> 00:33:24.953
When we met, you had home rolled your own coding harness.
377
00:33:25.933 --> 00:33:29.596
And now you're using the Foundation Lab coding harnesses.
378
00:33:30.217 --> 00:33:39.884
I'm curious for the novice engineer that's listening, what's the pattern where you can identify, okay, this is going to be a durable investment long term.
379
00:33:39.904 --> 00:33:49.172
This is something that we think is going to be useful into the future versus this is something that is just going to get absorbed into the capabilities of the Foundation Labs.
380
00:33:49.772 --> 00:33:56.434
So for right now, we're in mid 26, I would still say just remain on your toes because that's just going to be an open question.
381
00:33:56.674 --> 00:33:59.575
Like I'll just use the example of skill libraries, right?
382
00:33:59.595 --> 00:34:11.838
So combat engineering and G stack and superpowers and others, the amount of innovation that's happening inside of those skill ecosystems every week is also sort of conquering new territory and setting new precedent.
383
00:34:12.118 --> 00:34:16.499
And how much of that goes into the harness versus how much remains in prompts and skill libraries.
384
00:34:17.100 --> 00:34:18.060
All of that is TPD.
385
00:34:18.620 --> 00:34:22.604
I do recommend that folks, we've got some really good open source implementations.
386
00:34:22.644 --> 00:34:23.485
You can look at warp.
387
00:34:23.505 --> 00:34:24.887
You can look at codex.
388
00:34:24.907 --> 00:34:25.828
You can look at pi.
389
00:34:26.028 --> 00:34:27.770
Please just look at those coding harnesses.
390
00:34:28.250 --> 00:34:30.072
Look at the way codex is implemented.
391
00:34:30.412 --> 00:34:34.076
If you want, you can look at some of the reverse engineering of Claude that are out there.
392
00:34:34.116 --> 00:34:35.017
You should understand it.
393
00:34:35.037 --> 00:34:41.324
You should own that understanding because, you know, you shouldn't let the model tell you what to do in all these cases like you.
394
00:34:41.724 --> 00:34:43.447
You can't outsource your understanding.
395
00:34:43.487 --> 00:34:45.670
You need to own what's under the hood.
396
00:34:46.030 --> 00:34:48.814
But then once you own it, you can decide intellectually.
397
00:34:48.854 --> 00:34:53.400
Once you own it, you can decide how much you're outsourcing to Codex this week.
398
00:34:53.601 --> 00:34:56.725
OK, just don't let the hardest make the outsourcing decision for you.
399
00:34:57.563 --> 00:34:57.803
All right.
400
00:34:57.843 --> 00:34:59.724
That's AI worth using Justin McCarthy.
401
00:34:59.784 --> 00:35:00.825
Thank you so much.
402
00:35:01.205 --> 00:35:03.826
What I'm taking from this, the factory isn't a metaphor.
403
00:35:03.866 --> 00:35:04.746
It's a discipline.
404
00:35:05.187 --> 00:35:09.329
Intent goes in and validated software comes out and that's the most important point.
405
00:35:09.429 --> 00:35:15.311
The validation, um, humans are moving from writing software to designing the system that writes software.
406
00:35:15.351 --> 00:35:18.573
I, I suspect a lot of my time in the future is going to be focused on validation.
407
00:35:19.399 --> 00:35:21.720
The loop doesn't stop at the code though.
408
00:35:21.840 --> 00:35:27.681
It runs all the way to a 3D printer or a CAD file or add budget in the wild.
409
00:35:28.681 --> 00:35:33.223
Justin's building this for complex enterprises at Diffusion, diffusion.io.
410
00:35:33.883 --> 00:35:38.664
If this was useful though, send it to the person on your team who still thinks agents are just autocomplete.
411
00:35:39.424 --> 00:35:40.365
New episodes weekly.
412
00:35:40.785 --> 00:35:41.385
See you next time.