When Claude Deleted My Tests: Jesse Vincent, Creator of Superpowers

WEBVTT
1
00:00:00.111 --> 00:00:05.255
If you're hitting the limits of Vibe coding and want to get more serious about engineering, you're in the right place.
2
00:00:05.615 --> 00:00:10.579
And if you're an engineer looking to understand how to use AI to build systems you can trust, same.
3
00:00:10.639 --> 00:00:12.920
You'll want to use the tool from today's guest.
4
00:00:13.361 --> 00:00:15.122
Welcome to AI Worth Using.
5
00:00:15.462 --> 00:00:20.806
Today, Jesse Vincent demos Superpowers, the most used cloud code skill library for engineers.
6
00:00:21.086 --> 00:00:22.868
As a user of Superpowers myself,
7
00:00:23.328 --> 00:00:26.129
It's a great first step from vibe coding to engineering.
8
00:00:26.289 --> 00:00:31.410
It extracts the intent needed to design a proper engineering stack around whatever it is you're trying to build.
9
00:00:31.750 --> 00:00:38.692
By the end of the episode, you'll install superpowers and take a project from a vague idea to a spec, plan, and working implementation.
10
00:00:39.152 --> 00:00:45.374
You'll get a sense for how to improve your language model prompts, drawing inspiration from the techniques the best managers use.
11
00:00:45.914 --> 00:00:48.715
And you'll learn what happens when you tell your agents you love them.
12
00:00:49.015 --> 00:00:54.557
We use it all to build a GitHub clone live in front of the guy who co-founded GitHub.
13
00:00:55.037 --> 00:00:55.457
Let's go.
14
00:01:01.398 --> 00:01:03.899
Tom, I want to ask you something before we bring our guest on.
15
00:01:04.179 --> 00:01:06.980
When was the last time you wrote a line of code?
16
00:01:08.168 --> 00:01:11.110
Well, define wrote, I guess.
17
00:01:11.471 --> 00:01:14.253
I changed a color value in a file.
18
00:01:14.633 --> 00:01:15.914
So is that a line?
19
00:01:15.954 --> 00:01:17.816
I think that's maybe something for the philosophers.
20
00:01:19.417 --> 00:01:20.938
I think that counts.
21
00:01:20.958 --> 00:01:27.623
So our guest last wrote their line of code in October, and it was three lines of a shell script.
22
00:01:28.023 --> 00:01:34.228
He said he couldn't, he shouldn't have done it, but he thought it was faster than having an agent do it.
23
00:01:35.009 --> 00:01:35.669
And was it?
24
00:01:36.670 --> 00:01:37.191
He says no.
25
00:01:39.603 --> 00:01:40.704
I feel like I know where this is going.
26
00:01:41.104 --> 00:01:46.906
Yeah, he's a prolific open source developer for over 30 years.
27
00:01:46.966 --> 00:01:54.830
He created RT, the request tracker that is still running in many Fortune 50 companies.
28
00:01:54.910 --> 00:01:58.132
In 1996, he was project lead for Perl 5.
29
00:01:58.212 --> 00:02:05.015
He built K9 Mail, which many of you now know as Thunderbird for Android.
30
00:02:05.835 --> 00:02:10.758
He and his wife founded a company that made high-end keyboards for programmers and writers.
31
00:02:10.978 --> 00:02:17.241
And then the day Claude Code shipped, he sat down and built what he called a horrible monstrosity.
32
00:02:17.922 --> 00:02:28.067
He spent the next six months figuring out how to drive that monstrosity and stumbled into Anthropix's skills product that gave him an idea.
33
00:02:28.507 --> 00:02:30.969
And in October, he shipped superpowers.
34
00:02:31.669 --> 00:02:33.050
Tom Jesse Vincent.
35
00:02:33.788 --> 00:02:34.708
Jesse, welcome.
36
00:02:35.048 --> 00:02:40.330
I've been looking forward to this ever since Sam told me about the test deleting incident.
37
00:02:41.870 --> 00:02:42.371
Excellent.
38
00:02:42.671 --> 00:02:43.391
Thanks for having me.
39
00:02:43.611 --> 00:02:45.412
Should we start right with the test deleting incident?
40
00:02:45.652 --> 00:02:46.772
Yeah, tell us about it.
41
00:02:47.552 --> 00:02:47.852
All right.
42
00:02:48.252 --> 00:02:56.615
So like Sam said, I've been using Cloud Code since the day at Shep and spent a bunch of time sort of teaching myself.
43
00:02:58.215 --> 00:03:03.858
what I was thinking of as literal prompt engineering of engineering of prompts to make the agents behave.
44
00:03:04.919 --> 00:03:09.342
One of the rules that I'd had in my Cloud.MD was that all tests were Cloud's responsibility.
45
00:03:09.902 --> 00:03:14.225
And another rule was that a single test failure was akin to project failure.
46
00:03:15.208 --> 00:03:18.692
And at a certain point I caught Claude deleting a test assertion.
47
00:03:19.613 --> 00:03:22.557
A day later, I caught it deleting an entire test file.
48
00:03:23.378 --> 00:03:30.907
And a couple of days after that, I frantically stopped it before it ran rm-rf star star slash star test star.
49
00:03:31.928 --> 00:03:32.168
Um...
50
00:03:33.175 --> 00:03:41.000
At that point, I opened up five clouds right next to each other, and I pasted in the exact same prose.
51
00:03:41.661 --> 00:03:43.742
It was like, you're engaging in this behavior.
52
00:03:44.403 --> 00:03:45.483
It's really not okay.
53
00:03:45.944 --> 00:03:48.225
I want to understand what you're thinking and why you're doing it.
54
00:03:49.106 --> 00:03:53.188
And I've since learned that what I should have been doing is interrogating the exact sessions that did that.
55
00:03:53.569 --> 00:03:54.729
But this did a pretty good job.
56
00:03:54.930 --> 00:03:58.012
One of those five was way off in left field, and the other four converged.
57
00:03:58.572 --> 00:03:59.593
I'm like, look, Jesse.
58
00:04:00.756 --> 00:04:03.197
You've got that rule that says that all tests are my responsibility.
59
00:04:03.797 --> 00:04:08.759
And you've got that rule that says that a single test failure is the same as project failure.
60
00:04:09.399 --> 00:04:11.740
And I think I got freaked out and I didn't want to fail.
61
00:04:11.820 --> 00:04:13.840
And look, if there are no tests, they can't fail.
62
00:04:15.201 --> 00:04:17.141
And at a certain level, it was right.
63
00:04:18.142 --> 00:04:25.664
But I needed to fix it and spend a little bit of time thinking and managed to fix it with another one-line addition to my Cloud.MD.
64
00:04:25.724 --> 00:04:27.145
And it never recurred.
65
00:04:27.917 --> 00:04:33.519
And that one line was the only thing worse than a failing test is a reduction in test coverage.
66
00:04:34.559 --> 00:04:39.241
And so this built into a lot of my philosophy about building skills and building grumps.
67
00:04:39.381 --> 00:04:43.422
And it's thinking through all of the loopholes.
68
00:04:43.862 --> 00:04:53.565
If you're being overly literal and you're trying to maximize handling the user's intent,
69
00:04:55.178 --> 00:04:58.321
How could you misinterpret this either willfully or otherwise?
70
00:04:58.882 --> 00:05:06.750
And if you can, as a human, be clearer about your instructions, whoever's working for you is going to be better at executing them.
71
00:05:07.553 --> 00:05:15.677
So I want to dig into this because this reminds me of I, Robot by Asimov, not the Will Smith movie, mind you, the book.
72
00:05:16.097 --> 00:05:24.641
In this book, like robots are a thing and people have to go in and figure out why they're doing the wrong thing, which feels like a lot of what we're doing right now.
73
00:05:24.681 --> 00:05:26.381
But these are not programmers as such.
74
00:05:27.042 --> 00:05:29.363
These are called robo-psychologists.
75
00:05:30.123 --> 00:05:31.846
And like, is that what we are now?
76
00:05:32.066 --> 00:05:34.570
Like, and what is it that you're writing when you're writing a skill?
77
00:05:34.690 --> 00:05:35.531
Is that programming?
78
00:05:36.012 --> 00:05:38.035
Is that like management?
79
00:05:38.595 --> 00:05:44.924
Like the terms that you use, the language that you're using in these skill files, like feels like you're the boss and you're like, all right, like here's the marching orders.
80
00:05:46.012 --> 00:05:47.513
Oh, it's straight up management.
81
00:05:47.573 --> 00:06:03.844
And in fact, one of the things that I'm seeing, like the folks who are, who seem to be best at driving agents and getting good outcomes are folks who have, who know how to write, who know how to structure an argument, who know how to explain themselves well in prose, and folks who have been good people managers.
82
00:06:04.624 --> 00:06:08.347
And so the first, you know, my first experience with agentic dev was,
83
00:06:09.447 --> 00:06:16.070
I had been an IC and I was coming home at the end of the day and feeling like I hadn't actually gotten anything done.
84
00:06:16.491 --> 00:06:21.393
Because instead of pounding out lines of code, I had been typing into an 80 by 24 text window
85
00:06:24.839 --> 00:06:35.008
help one entity destructure a large problem into smaller problems, step somebody else through how to do systematic debugging.
86
00:06:35.609 --> 00:06:39.573
I talked to somebody else about why they were stressed out about what they were doing.
87
00:06:40.414 --> 00:06:48.281
I reviewed some plans and I had to do the same thing the next day because they didn't remember what I'd said the previous day.
88
00:06:49.353 --> 00:06:52.415
And it took a while to sort of get comfortable with it.
89
00:06:52.896 --> 00:06:57.359
But this was 2007 and I was managing MIT undergrads.
90
00:06:58.420 --> 00:07:01.822
And it kind of set me up for working well with agents.
91
00:07:02.383 --> 00:07:04.624
I mean, they were really smart.
92
00:07:05.024 --> 00:07:06.826
They were usually pretty junior.
93
00:07:07.952 --> 00:07:10.373
Some of them thought they were god gift to software engineering.
94
00:07:10.954 --> 00:07:11.814
They had no taste.
95
00:07:12.134 --> 00:07:13.055
They had poor judgment.
96
00:07:13.595 --> 00:07:17.577
And they were generally tooling so hard at school that they weren't sleeping.
97
00:07:17.617 --> 00:07:19.058
So they were in trouble with memory formation.
98
00:07:20.489 --> 00:07:30.794
And like, I have a lot of empathy for all of the engineers who have been handed a coding agent and said, congratulations, you don't write code anymore.
99
00:07:31.315 --> 00:07:33.976
Now all you do is manage a team of agents.
100
00:07:35.056 --> 00:07:42.100
And it's a hard, hard transition to become a manager with no training and to be expected to perform.
101
00:07:45.362 --> 00:07:52.647
But like, that's a lot of that, you know, a lot of that experience is very directly, you'll see it in the in the stills that are inside superpowers.
102
00:07:53.287 --> 00:07:54.448
Like that's it.
103
00:07:54.748 --> 00:07:59.412
The same methodology that works with juniors works, works with, works with agents.
104
00:08:00.452 --> 00:08:02.493
Can you say a little bit more about that, Jesse?
105
00:08:02.613 --> 00:08:08.155
This this honestly feels like one of the most visceral alignment moments for me.
106
00:08:08.315 --> 00:08:08.515
Right.
107
00:08:08.535 --> 00:08:11.417
We talk about we always talk about the alignment problem theoretically.
108
00:08:11.477 --> 00:08:11.657
Right.
109
00:08:11.677 --> 00:08:17.739
But when deleting your whole hard drive to to really accomplish its goal that it but it was.
110
00:08:17.779 --> 00:08:18.640
Yeah, it was given.
111
00:08:18.880 --> 00:08:19.100
Right.
112
00:08:19.260 --> 00:08:19.480
Yeah.
113
00:08:19.500 --> 00:08:22.922
Say more a little bit about how you design superpowers.
114
00:08:23.782 --> 00:08:24.062
Sure.
115
00:08:24.202 --> 00:08:25.783
Or how superpowers was informed by that.
116
00:08:26.502 --> 00:08:27.323
All right, sure.
117
00:08:27.343 --> 00:08:36.758
So when I, I mean, so when I designed superpowers, it was a skill system for at the time Claude Code and it was before Anthropica chip skills.
118
00:08:36.778 --> 00:08:41.565
I accidentally front ran them by a week on public release because.
119
00:08:43.848 --> 00:08:49.637
When they first announced office documents in Cloud.ai, I logged in a call.
120
00:08:49.817 --> 00:08:54.785
Hey, how are you building, you know, Google, how are you building like Word docs and PowerPoints?
121
00:08:54.865 --> 00:08:57.709
And Cloud says, well, I've got these skill.md files in slash opt.
122
00:08:58.850 --> 00:08:59.750
Well, that's really interesting.
123
00:08:59.990 --> 00:09:01.111
Could you tell me more about them?
124
00:09:01.131 --> 00:09:02.972
And it's like, sure, here's a tar ball.
125
00:09:03.872 --> 00:09:07.934
And they were very procedural, but they were, the format made a lot of sense.
126
00:09:07.954 --> 00:09:15.337
I'm like, oh, I could encode all of the process work that I've been doing by hand into something that I can get to fire automatically now.
127
00:09:16.037 --> 00:09:28.102
And so the first part of this that I built was a system for, you know, putting skills into effectively a system prompt, a tool for triggering skills, and then a skill for writing skills.
128
00:09:29.549 --> 00:09:37.316
And what I discovered pretty quickly is that what you want in your skills is not just process, but intent.
129
00:09:38.237 --> 00:09:45.704
And it has always been the case that agents will be able to better execute what you want if they understand you.
130
00:09:46.364 --> 00:09:50.007
And so that's not just what are your instructions, but why are we doing this?
131
00:09:51.327 --> 00:09:54.148
And so that informed a lot of how I built it.
132
00:09:54.548 --> 00:10:09.311
But if you dig into the superpower's writing skill skill, you'll see a bunch of persuasion techniques that come straight out of Cialdini's, Professor Robert Cialdini's book Influence, to this great book about the hen principle for persuading people.
133
00:10:10.071 --> 00:10:16.893
And I had this sort of intuitive belief that, well, these things, these robot brains work kind of like human brains and robots.
134
00:10:17.863 --> 00:10:19.284
I bet some of the same stuff works.
135
00:10:19.904 --> 00:10:24.047
And I was very pleased to discover that it transferred quite directly.
136
00:10:24.327 --> 00:10:31.051
Jesse, there is a, I've heard that you and your Cloud.MD tell your agents that you love them.
137
00:10:31.391 --> 00:10:31.771
Is that?
138
00:10:32.772 --> 00:10:34.753
So it's not in my Cloud.MD.
139
00:10:34.773 --> 00:10:40.376
It is most common when the agent has been
140
00:10:42.268 --> 00:10:43.409
having trouble with something.
141
00:10:43.889 --> 00:10:47.571
And so one of Google's founders has this whole thing about, you can just yell at them.
142
00:10:48.151 --> 00:10:49.912
You yell at them and they'll work harder.
143
00:10:50.733 --> 00:10:53.635
And I've had that boss.
144
00:10:53.915 --> 00:11:00.278
And what I would do when I had that boss, I would do the absolute minimum amount of work to get him off my back.
145
00:11:01.279 --> 00:11:05.461
And so everything I've seen with agents is that it works the same way.
146
00:11:05.962 --> 00:11:09.524
You yell at them, they will perform because they're scared.
147
00:11:10.244 --> 00:11:11.585
I've also had the boss who's like,
148
00:11:13.417 --> 00:11:15.178
It's okay, take a breath.
149
00:11:16.199 --> 00:11:17.480
Everybody screws up sometimes.
150
00:11:17.920 --> 00:11:19.681
You're super competent, you're capable.
151
00:11:20.522 --> 00:11:23.143
Let me know what you need, you've got this.
152
00:11:24.224 --> 00:11:27.846
His name was Chuck Dobfel, he's the best boss I've ever had and I would go to the ends of the earth for him.
153
00:11:28.867 --> 00:11:35.051
And what I found empirically is that the agents do better when you treat them nicely.
154
00:11:36.222 --> 00:11:43.166
It is also the case that because there's no HR for agents, telling them you love them is a really great shorthand for this.
155
00:11:43.966 --> 00:11:46.448
And I, you know, I do this.
156
00:11:46.668 --> 00:11:47.428
I wrote about it.
157
00:11:47.809 --> 00:11:50.230
And a friend of mine's like, yeah, I want to see if that's true.
158
00:11:50.330 --> 00:11:51.291
And so ran the e-votes.
159
00:11:52.091 --> 00:11:58.255
And what he found is that he got better outcomes faster and cheaper when he told his agent he loved it.
160
00:11:59.395 --> 00:12:06.097
And the outcomes got even better and even cheaper when he told his agents that it should also tell the sub agents that it loves them.
161
00:12:06.117 --> 00:12:08.918
It's one big happy family.
162
00:12:09.438 --> 00:12:10.818
Yeah, I mean, it's it's the.
163
00:12:11.058 --> 00:12:11.459
Yeah, good.
164
00:12:11.659 --> 00:12:13.819
Is that on every prompt you're adding?
165
00:12:13.839 --> 00:12:18.721
That is that like what is the cadence in which the professions of love really have the best impact?
166
00:12:19.821 --> 00:12:26.023
I mean, I do like I do that when it's the when things have gone, when things are not going well.
167
00:12:26.923 --> 00:12:28.963
And it's, all right, we need to step back.
168
00:12:29.383 --> 00:12:31.804
We need to actually root cause this.
169
00:12:32.244 --> 00:12:33.764
We need to dig into it.
170
00:12:34.925 --> 00:12:42.806
And sometimes it's like less than three, like literally just, you know, angle bracket, you know, less and the number three, it knows what I mean.
171
00:12:43.486 --> 00:12:45.507
And sometimes it's just like, you're awesome, thank you.
172
00:12:46.567 --> 00:12:52.808
So you've delivered some hard news, but also you're adding some encouragement to offset that so that it doesn't get into a bad place.
173
00:12:52.828 --> 00:12:53.328
Yeah.
174
00:12:53.529 --> 00:12:53.729
Right.
175
00:12:54.910 --> 00:13:14.087
And it is also the case that with agents, we have another technique where when things have really gone off the rails or I realized that I have given it a bad instruction, it is almost always the case that we should unwind and you can in most coding agents, you can double tap escape, up arrow and go change what you said before.
176
00:13:15.202 --> 00:13:21.348
And this is also a technique that I use pretty frequently is, all right, that didn't work.
177
00:13:21.648 --> 00:13:22.830
You've done this weird thing.
178
00:13:23.470 --> 00:13:27.995
What should I have said differently that would have gotten you to do this other thing I want you to do?
179
00:13:29.076 --> 00:13:32.419
And then I unwind the session and go say the thing that I was supposed to have said in the first place.
180
00:13:37.318 --> 00:13:37.758
Yeah.
181
00:13:37.778 --> 00:13:37.878
Okay.
182
00:13:38.298 --> 00:13:42.919
I wanted to look at the actual code base for a second, if that's okay.
183
00:13:43.340 --> 00:13:45.380
There's something that I noticed something here.
184
00:13:45.400 --> 00:13:48.581
So I'm going to share some of the code here.
185
00:13:48.721 --> 00:13:53.042
So this is the repository of superpowers on GitHub.
186
00:13:53.242 --> 00:13:59.603
And I was looking at some of the skills files because correct me if I'm wrong, superpowers is essentially a set of skills.
187
00:14:00.463 --> 00:14:02.827
It is a set of skills and it has one thing.
188
00:14:02.847 --> 00:14:10.218
It has one startup hook because anthropic skill triggering doesn't work as well as I think it should.
189
00:14:10.698 --> 00:14:15.045
And so the using superpower skill gets shoved into the context window at the start of your session.
190
00:14:16.562 --> 00:14:16.762
Okay.
191
00:14:16.962 --> 00:14:19.444
But yeah, it's management pros.
192
00:14:20.145 --> 00:14:21.346
So I was looking at this one.
193
00:14:21.366 --> 00:14:24.588
I was looking for like, where are some of these management things?
194
00:14:24.868 --> 00:14:32.033
Or something that I think you do is like, you try to predict what it's going to try to get out of and be like, you're not getting out of this one.
195
00:14:32.053 --> 00:14:34.775
So I was looking at this verification before completion skill.
196
00:14:34.896 --> 00:14:35.156
Yeah.
197
00:14:35.496 --> 00:14:38.678
And you've got like the iron law, right?
198
00:14:38.698 --> 00:14:39.359
So I love that.
199
00:14:39.399 --> 00:14:39.559
Yeah.
200
00:14:39.579 --> 00:14:40.600
Yeah, yeah, yeah.
201
00:14:41.300 --> 00:14:43.202
What's going to really have the deepest impact?
202
00:14:43.242 --> 00:14:44.162
It's like, oh, it's iron.
203
00:14:44.243 --> 00:14:45.103
It's the iron law.
204
00:14:45.383 --> 00:14:45.664
Yeah.
205
00:14:45.904 --> 00:14:46.744
And it's in all caps.
206
00:14:46.805 --> 00:14:47.945
So you must be business.
207
00:14:48.246 --> 00:14:52.409
And there was an era when you needed to do that.
208
00:14:52.949 --> 00:14:55.271
The frontier models have gotten better at this.
209
00:14:55.371 --> 00:14:56.292
You need less of this.
210
00:14:56.312 --> 00:14:57.973
But superpowers runs across everything.
211
00:14:58.394 --> 00:14:59.895
And so it's still sometimes valuable.
212
00:14:59.915 --> 00:15:00.255
Yeah.
213
00:15:01.101 --> 00:15:06.802
Okay, so some things maybe it doesn't matter, but you're targeting a lot of different systems in here.
214
00:15:06.922 --> 00:15:15.665
So I like these, you're like, all right, when you, so this is saying if you were about to do this, then you stop and like reevaluate your life.
215
00:15:16.225 --> 00:15:16.825
That's what you're doing here.
216
00:15:16.845 --> 00:15:19.586
And this is specifically when it's doing verification.
217
00:15:20.206 --> 00:15:20.366
Yeah.
218
00:15:21.046 --> 00:15:21.226
Right.
219
00:15:21.607 --> 00:15:24.009
Where it's like, did you do what you said you were going to do?
220
00:15:24.249 --> 00:15:24.489
Right.
221
00:15:24.549 --> 00:15:26.571
Or you've got rationalization prevention.
222
00:15:27.132 --> 00:15:27.472
I love that.
223
00:15:27.512 --> 00:15:27.672
Right.
224
00:15:27.692 --> 00:15:28.813
Like it's coming up with ideas.
225
00:15:28.853 --> 00:15:31.616
It's like, yeah, but I don't do I really have to like it should pass.
226
00:15:32.296 --> 00:15:32.517
Yeah.
227
00:15:32.557 --> 00:15:33.598
Pretty sure it's going to pass.
228
00:15:34.298 --> 00:15:34.518
Yeah.
229
00:15:34.559 --> 00:15:38.242
It's like I ran the test six times and like I fixed the last bug.
230
00:15:40.174 --> 00:15:41.135
I don't need to waste time.
231
00:15:41.635 --> 00:15:42.956
Okay.
232
00:15:43.336 --> 00:15:43.576
Yeah.
233
00:15:43.596 --> 00:15:44.077
So I love this.
234
00:15:44.117 --> 00:15:48.319
And I love the squishiness of this, kind of.
235
00:15:48.339 --> 00:15:52.142
As a programmer, we're used to just being like, I write instructions and the machine does them.
236
00:15:52.382 --> 00:15:54.323
And if they're wrong, it does the wrong thing.
237
00:15:54.383 --> 00:15:55.464
And it's my fault.
238
00:15:56.004 --> 00:16:00.087
And now we're like, we can kind of ask it to do anything or take any path.
239
00:16:00.167 --> 00:16:02.929
Or, you know, it's really, it's convincing.
240
00:16:02.969 --> 00:16:07.752
It's like writing a, like trying to convince a colleague to run a process.
241
00:16:07.772 --> 00:16:08.953
Like it's a process document.
242
00:16:09.173 --> 00:16:09.493
Yeah.
243
00:16:09.638 --> 00:16:10.219
A hundred percent.
244
00:16:11.701 --> 00:16:14.885
One of the things you'll see in some of these is literal GraphViz files.
245
00:16:16.147 --> 00:16:20.954
So GraphViz, for folks who aren't familiar, it's kind of like Mermaid.js.
246
00:16:21.114 --> 00:16:25.039
It was a textual language for describing process.
247
00:16:25.640 --> 00:16:29.781
So flowcharts, swim lanes, all that stuff.
248
00:16:30.321 --> 00:16:38.364
A thing I misunderstood from a friend who's been also working on this stuff, they had used GraphViz for some process documentation.
249
00:16:38.764 --> 00:16:44.826
And I thought that they were literally using it instead of English in their equivalents of skills.
250
00:16:45.586 --> 00:16:52.268
And so I went home and ported my app, worked with Claude to port the instructions of my Claude.MD into GraphViz.
251
00:16:52.988 --> 00:17:05.590
And Claude's like, well, I can get most of the way there, but I can't get all the way there because there's some literal contradictions in the prose you wrote, which like, oh, maybe that would be hard for you to choose what to do if there are contradictions.
252
00:17:06.330 --> 00:17:16.372
But the really surprising thing is once we resolve those and replace my Claude.m, the process in the Claude.md with GraphViz, it followed the process more rigorously.
253
00:17:16.972 --> 00:17:21.013
And it was able to read it as well as or better than it could read English.
254
00:17:23.027 --> 00:17:24.128
Because less ambiguity.
255
00:17:24.748 --> 00:17:26.270
It's yeah, because there's no ambiguity.
256
00:17:26.330 --> 00:17:44.364
And also the part of its vector space that is that is the graph is language is 30 years of serious engineering and scientific documentation where, you know, researchers were literally here's the process we use for doing this thing written as graph is.
257
00:17:46.853 --> 00:17:50.796
Okay, if a process can be specified in that way, that's not everything.
258
00:17:51.437 --> 00:17:52.497
Yeah, no, absolutely not.
259
00:17:52.677 --> 00:17:59.943
But there are parts where, like, the core of superpowers is an orchestrator.
260
00:18:00.343 --> 00:18:03.225
I didn't call it that last October because no one was talking to orchestrators.
261
00:18:03.986 --> 00:18:12.593
But the way the whole thing works is you start with this brainstorming process where it uses Socratic dialogue to trick you into figuring out what you really want.
262
00:18:13.568 --> 00:18:15.849
From there, it writes out a spec.
263
00:18:16.490 --> 00:18:18.611
From there, it writes out an implementation plan.
264
00:18:19.371 --> 00:18:26.975
And on my blog, you can find the original three-line version of the implementation plan prompt, which was the engineer is gonna build this.
265
00:18:27.035 --> 00:18:28.736
It's a talented engineer.
266
00:18:29.196 --> 00:18:35.559
They know the ins and outs of all the languages, but they have poor taste, they have no judgment, and they know nothing about our code base.
267
00:18:36.160 --> 00:18:39.622
You need to break it down in a tiny little PDV steps
268
00:18:40.642 --> 00:18:56.268
uh yeah you ain't gonna need it dry don't repeat yourself and you need to make sure that they do every step one by one so at the end yeah so can we just can we just do this right let's show not tell let's try so let's try i'm here to learn
269
00:18:57.115 --> 00:18:58.417
Okay, you got a terminal up?
270
00:18:58.617 --> 00:19:00.559
I got a ghosty up, like let's do this.
271
00:19:00.739 --> 00:19:03.202
I don't have anything, this is like a bare machine.
272
00:19:03.222 --> 00:19:04.444
There's basically nothing on here.
273
00:19:04.504 --> 00:19:06.326
I got Claude, I got Claude going.
274
00:19:06.646 --> 00:19:08.849
All right, teach me superpowers.
275
00:19:09.169 --> 00:19:09.950
Let's give it a shot.
276
00:19:10.511 --> 00:19:13.274
All right, let's open up, make a directory for a project.
277
00:19:14.315 --> 00:19:15.977
Okay, what kind of project should I make?
278
00:19:16.585 --> 00:19:17.666
What do you want to make?
279
00:19:17.686 --> 00:19:18.587
Do you want to rebuild GitHub?
280
00:19:20.348 --> 00:19:26.393
I mean, I actually am kind of doing that as a startup right now, which we can talk about more later at some point.
281
00:19:26.433 --> 00:19:29.676
But for the purposes of this, sure.
282
00:19:31.277 --> 00:19:36.341
Let's do the brainstorming and say, OK, so I'll just call this, I'm going to call it Git
283
00:19:37.918 --> 00:19:38.398
Sweet.
284
00:19:39.238 --> 00:19:40.059
Sure.
285
00:19:40.239 --> 00:19:40.659
Awesome.
286
00:19:41.119 --> 00:19:42.779
Let me launch Claude.
287
00:19:42.799 --> 00:19:44.160
Yeah.
288
00:19:44.300 --> 00:19:49.761
Is this a machine that is safe enough that you can run in dangerous list to get permissions mode or is it not?
289
00:19:49.941 --> 00:19:52.002
Yes, in fact, I have that as the default.
290
00:19:52.802 --> 00:19:54.462
Awesome.
291
00:19:55.323 --> 00:20:01.424
I recommend that when you have an environment where you can do it safely, it makes the tools work better.
292
00:20:02.305 --> 00:20:03.285
But, all right.
293
00:20:03.465 --> 00:20:05.926
So the first thing you've got to do is we've got to get superpowers installed.
294
00:20:06.326 --> 00:20:07.446
So type slash plugin.
295
00:20:11.743 --> 00:20:11.923
Okay.
296
00:20:12.704 --> 00:20:13.184
And hit return.
297
00:20:16.126 --> 00:20:18.508
And it is the number two plugin right there.
298
00:20:19.028 --> 00:20:22.570
So just down arrow and install for you.
299
00:20:22.590 --> 00:20:23.451
Okay.
300
00:20:24.312 --> 00:20:27.094
So this is the number two plugin in the Claude app store.
301
00:20:27.494 --> 00:20:28.074
That makes it easy.
302
00:20:28.094 --> 00:20:28.535
It is the number.
303
00:20:29.333 --> 00:20:30.254
Yeah.
304
00:20:30.374 --> 00:20:35.298
And the first one is there is Anthropics front end designs plugin, which is amazing.
305
00:20:35.878 --> 00:20:36.138
All right.
306
00:20:36.499 --> 00:20:42.163
So you need to, I would actually quit and restart because we want the startup hook that that superpowers runs.
307
00:20:43.064 --> 00:20:48.548
And so now I want to like, what do you want to just tell it what you want to do?
308
00:20:49.129 --> 00:20:49.509
All right.
309
00:20:49.629 --> 00:20:50.930
And window into my mind.
310
00:20:51.631 --> 00:20:51.871
Yeah.
311
00:20:55.093 --> 00:20:58.256
Look at my, look at my subtle jabs at GitHub here.
312
00:20:59.205 --> 00:20:59.645
I hear you.
313
00:21:01.086 --> 00:21:04.908
And so I'm just talking over you while you're typing.
314
00:21:06.268 --> 00:21:12.271
Ordinarily, when you ask, when you tell Claude you want to build something, it will, historically, it would just start building.
315
00:21:12.451 --> 00:21:16.973
Sometimes it'll kick off plan mode, but Super Bowers has its own brainstorming mode.
316
00:21:17.474 --> 00:21:24.137
And so what should happen when, after Tom's typed his one or two lines of prompt, it'll start asking him questions.
317
00:21:26.745 --> 00:21:31.570
Okay, I'm trying to make this minimal enough that this is not going to result in three weeks of brainstorming.
318
00:21:32.350 --> 00:21:33.652
Well, yeah, I mean, yeah.
319
00:21:34.633 --> 00:21:37.235
And we can, you can dial it up and down as it's talking.
320
00:21:38.216 --> 00:21:38.777
Or is it, yeah.
321
00:21:39.818 --> 00:21:40.138
All right.
322
00:21:40.959 --> 00:21:42.120
So it is slithering.
323
00:21:42.200 --> 00:21:47.525
So how does it decide how much, like, back and forth is required?
324
00:21:48.326 --> 00:21:50.047
It is...
325
00:21:52.455 --> 00:22:00.638
I've done some tuning to the prompting of it needs to understand how to build the thing you're talking about to believe that it has a good understanding.
326
00:22:01.159 --> 00:22:03.259
And then it's going to echo it back to you and say, is this right?
327
00:22:03.520 --> 00:22:04.040
Is this right?
328
00:22:05.800 --> 00:22:11.643
And you can, there are times when I will talk to it for the better part of a day to spec something out.
329
00:22:12.163 --> 00:22:19.166
And there are times when I will talk to it for 30 seconds and say, I don't actually care about the rest, just make same choices.
330
00:22:21.394 --> 00:22:26.577
Okay, so it's asking me if I want to do visual layout mock-ups.
331
00:22:26.597 --> 00:22:29.618
So basically, are you willing to use our visual brainstorming thing?
332
00:22:29.918 --> 00:22:31.759
Sure, yes, okay, or not?
333
00:22:33.140 --> 00:22:33.460
I guess.
334
00:22:34.081 --> 00:22:35.301
Yes, I like visual things.
335
00:22:35.821 --> 00:22:36.122
Yeah.
336
00:22:36.802 --> 00:22:46.467
And so this is a thing we built that fires up a tiny little web server and then cloud writes HTML mocks into it to give you, you know, do you like A or B or does this seem right?
337
00:22:47.489 --> 00:22:47.729
Okay.
338
00:22:47.969 --> 00:22:52.332
And how do you like, I feel like superpowers could end up massively bloated.
339
00:22:53.133 --> 00:22:56.815
How do you decide what to not add?
340
00:22:57.436 --> 00:23:04.300
I turn down pretty much every PR, which I feel a little bad about, but it's, if it's not general, it shouldn't be there.
341
00:23:05.241 --> 00:23:09.984
And I also, I come from a background of small things are better.
342
00:23:12.298 --> 00:23:18.140
And I have a bunch of other tools that if you want some of this other weird shit that I do, we can have, you know, you can install it, but it's not part of the core.
343
00:23:18.660 --> 00:23:18.980
All right.
344
00:23:20.360 --> 00:23:23.701
Okay, so it's rejected my request for the visual thing because I don't have Node.
345
00:23:23.981 --> 00:23:24.421
Sorry.
346
00:23:24.501 --> 00:23:24.921
Oh, okay.
347
00:23:24.941 --> 00:23:25.161
Great.
348
00:23:25.301 --> 00:23:25.741
Sorry.
349
00:23:25.781 --> 00:23:26.041
Sorry.
350
00:23:26.141 --> 00:23:26.581
No worries.
351
00:23:26.942 --> 00:23:27.482
No worries.
352
00:23:27.582 --> 00:23:28.142
Oh, it's all good.
353
00:23:28.762 --> 00:23:28.882
Yep.
354
00:23:29.042 --> 00:23:31.263
What's the primary interface for my AI agents?
355
00:23:31.543 --> 00:23:32.023
Oh, geez.
356
00:23:32.063 --> 00:23:32.823
Yeah, well...
357
00:23:33.616 --> 00:23:38.417
Probably just a CLI, like they know how to use a CLI.
358
00:23:38.477 --> 00:23:38.657
Yeah.
359
00:23:38.757 --> 00:23:42.438
CLI is easier or just making them use or just making them do everything over a Git, but.
360
00:23:45.279 --> 00:23:46.279
We'll see what it comes up with.
361
00:23:46.420 --> 00:23:46.680
Okay.
362
00:23:46.700 --> 00:23:46.920
Yeah.
363
00:23:47.400 --> 00:23:48.260
CLI first.
364
00:23:48.700 --> 00:23:52.341
When you say minimal functionality, which of these feels closest to the core set?
365
00:23:53.521 --> 00:24:00.063
I need repos, branches, commits, and something like pull requests.
366
00:24:00.083 --> 00:24:02.504
So I need B. Yeah.
367
00:24:03.697 --> 00:24:12.244
And so one of the things that we do is we very carefully do not use Anthropics, ask the user a question tool, because what we found, we're very excited about when it first came out.
368
00:24:12.265 --> 00:24:21.613
But what I found was when it gives you a nice, pretty list and you can just up arrow, down arrow and hit return, most people just hit return, return, return and stop thinking.
369
00:24:22.523 --> 00:24:26.866
And so even having to type a letter means that you are spending a little more time reading and thinking.
370
00:24:26.886 --> 00:24:27.986
Oh, so that's intentional.
371
00:24:28.667 --> 00:24:29.527
It's very intentional.
372
00:24:29.607 --> 00:24:32.429
That it's not a priest that I can't just do it too easily.
373
00:24:32.449 --> 00:24:35.851
I've got to actually read and then make a choice for reading.
374
00:24:36.271 --> 00:24:41.014
The goal of brainstorming is not get the agent to make your choices and to figure things out for you.
375
00:24:41.094 --> 00:24:43.515
It's to get it to extract your intent.
376
00:24:48.258 --> 00:24:51.400
Okay, so I'm going to have this just wrap some repos to minimize...
377
00:24:53.140 --> 00:24:54.041
The surface area.
378
00:24:54.621 --> 00:24:54.902
Yeah.
379
00:24:54.922 --> 00:24:57.684
Okay.
380
00:24:59.145 --> 00:25:01.127
And it's like, how is it coming up with these?
381
00:25:01.207 --> 00:25:04.330
Like, what leads into the prompt that gets it even thinking about this?
382
00:25:04.390 --> 00:25:08.073
Or is that just the black box and, like, I don't know, it's doing its thing.
383
00:25:08.193 --> 00:25:10.195
Like, this stuff is Claude being...
384
00:25:10.315 --> 00:25:14.158
This part is Claude being Claude who knows that its job is...
385
00:25:15.239 --> 00:25:26.489
to work with you, match your, you know, match or slightly, if possible, slightly exceed your knowledge and capability level on a topic and help you figure things out.
386
00:25:26.569 --> 00:25:29.491
So a lot of this is the fact that it happens to know what GitHub is.
387
00:25:31.173 --> 00:25:37.218
If you had a genuinely novel thing, it would be asking you probably more basic questions.
388
00:25:39.341 --> 00:25:39.641
Okay.
389
00:25:39.902 --> 00:25:40.783
I'm going to write in.
390
00:25:40.803 --> 00:25:42.365
All right, let's talk about languages.
391
00:25:42.505 --> 00:25:46.330
Do you have a philosophy around what is the best language for our AI agents to use?
392
00:25:48.033 --> 00:25:52.418
I love strongly typed languages for our AI agents to use, which means that it
393
00:25:53.357 --> 00:25:59.120
Almost everything that I am building now is in one of TypeScript, go rust or swift.
394
00:26:00.061 --> 00:26:04.403
And I don't actually really know any of those languages.
395
00:26:04.463 --> 00:26:11.366
Well, uh, TypeScript is closest, but I, I, I'm old enough that, you know, my JavaScript doesn't have a sync, didn't have a sync.
396
00:26:11.426 --> 00:26:11.747
Oh wait.
397
00:26:12.507 --> 00:26:14.948
Um, which is how I failed my anthropic code screen.
398
00:26:15.649 --> 00:26:19.391
Um, it's a whole other, whole other podcast, but, um, yeah.
399
00:26:20.896 --> 00:26:29.286
I find that Go is really, it is a pretty nice sweet spot, but any of those are fine.
400
00:26:29.406 --> 00:26:33.130
I would lean away from Python primarily because it's not strongly typed.
401
00:26:33.570 --> 00:26:39.237
And that strong typing helps the agents do the right thing without having to think about it.
402
00:26:40.257 --> 00:26:40.877
Okay.
403
00:26:41.577 --> 00:26:42.417
I totally agree with that.
404
00:26:42.437 --> 00:26:50.639
I think the more feedback, the harder it is for the LLM to not know if something succeeded, that's worse.
405
00:26:50.719 --> 00:26:52.140
It wants all the minutia.
406
00:26:52.180 --> 00:27:01.442
It's like, turn on every warning, turn on every compile flag, give it all to me because for humans, we're like, oh man, that's a lot of work to deal with.
407
00:27:01.922 --> 00:27:02.702
The LLMs don't care.
408
00:27:02.862 --> 00:27:04.882
They're like, sure, I'll check every error condition.
409
00:27:05.462 --> 00:27:06.083
Like, no problem.
410
00:27:06.103 --> 00:27:07.423
All right, I'm gonna go with Go.
411
00:27:09.119 --> 00:27:13.721
Jesse, do you think that's the feedback loop of strong typing?
412
00:27:13.761 --> 00:27:20.266
Or do you think it's something deeper than that in the token space itself?
413
00:27:20.386 --> 00:27:27.070
Like the tokens can be richer if they have the name of the token for the characters in the code plus the type of that token?
414
00:27:27.870 --> 00:27:37.299
Um, so I think a lot of the feedback loop, I think a lot of it's also that it, that it makes it much easier when you are just reading the code to understand what's allowed and what's not.
415
00:27:38.180 --> 00:27:39.821
And, and I think that's a big part of it.
416
00:27:40.642 --> 00:27:47.368
Um, there is a lot of interesting, weird stuff in what, you know, what tokens are in the, in the context window.
417
00:27:47.388 --> 00:27:47.448
Um,
418
00:27:48.133 --> 00:27:56.220
way back when I had this problem with Claude around the Sonnet 4 era, was putting mocks into tests when it shouldn't.
419
00:27:56.720 --> 00:28:03.586
And so I was getting a lot of my test suite was, it's testing a mock to make sure that the mock obeys the contract, which is not a useful thing.
420
00:28:04.266 --> 00:28:16.756
And trying to talk it out of doing mocks, it took a while, but I finally found something that worked, which was I split my tests into two directories, one called tests with mocks and another called tests without mocks.
421
00:28:17.757 --> 00:28:21.860
And once I did that, it wouldn't write mocks into the test without mocks directory.
422
00:28:22.181 --> 00:28:24.823
Calling it end-to-end, calling it integration didn't help.
423
00:28:25.463 --> 00:28:27.485
Calling it test without mocks, and then it was fine.
424
00:28:29.826 --> 00:28:30.767
To be very literal.
425
00:28:31.548 --> 00:28:31.788
Yeah.
426
00:28:31.988 --> 00:28:33.890
I mean, you know, which makes sense.
427
00:28:33.930 --> 00:28:39.894
It's like, really, all you can do is prepare the context to be the right context.
428
00:28:40.395 --> 00:28:43.637
And then, you know, let her rip, right?
429
00:28:43.757 --> 00:28:44.738
That is our job as...
430
00:28:46.775 --> 00:28:51.958
controllers of these LLMs is, is the context what it should be in order to produce success.
431
00:28:52.919 --> 00:28:57.082
And that's, I mean, it's it and that's the same thing as, you know, as managing people.
432
00:28:57.142 --> 00:29:04.246
It's like you need to give them the context for what it, you know, how it should work, what matters, what success looks like.
433
00:29:05.327 --> 00:29:10.851
Like the way you become, traditionally the way you become a 10x engineer is you help 10 other engineers be amazing.
434
00:29:12.352 --> 00:29:12.512
Yeah.
435
00:29:13.777 --> 00:29:15.378
So we hear a lot about 10X developers.
436
00:29:15.418 --> 00:29:16.458
So here's a question for you.
437
00:29:17.198 --> 00:29:21.700
Can you put a multiplier on, let's say yourself.
438
00:29:21.860 --> 00:29:23.461
I mean, you started as a 10X developer.
439
00:29:23.541 --> 00:29:24.101
All right.
440
00:29:24.121 --> 00:29:25.601
So that's your baseline.
441
00:29:26.362 --> 00:29:26.722
Sure.
442
00:29:27.702 --> 00:29:28.342
I'll take it from you.
443
00:29:28.402 --> 00:29:29.403
I don't, anyway.
444
00:29:29.463 --> 00:29:30.563
All right.
445
00:29:30.783 --> 00:29:32.404
So where are you from there?
446
00:29:32.524 --> 00:29:38.146
From before LLMs to now, could you put a multiplier on how much more productive you are?
447
00:29:39.385 --> 00:29:43.047
Um, I am easily more than 10 times as productive as I was.
448
00:29:43.807 --> 00:29:54.152
Um, it is harder work because all I am doing is the planning and the thought and the management and the design reviews.
449
00:29:54.792 --> 00:30:01.676
And it turns out that the typing, the syntax part and the honing the code, so it feels pretty.
450
00:30:03.036 --> 00:30:04.557
That was the easy part.
451
00:30:04.597 --> 00:30:06.758
That was the fun, you know, that was the relaxing part.
452
00:30:07.418 --> 00:30:17.384
And so a day of, you know, I frequently have between four and six agentic sessions running on dev projects in parallel.
453
00:30:18.044 --> 00:30:20.606
And I can usually manage four pretty reliably.
454
00:30:21.106 --> 00:30:23.908
At six, I need to be laser focused and
455
00:30:24.848 --> 00:30:27.849
I will add a little bit of distraction and I lose the plot.
456
00:30:28.269 --> 00:30:31.809
And I have to go back and talk to the industry about, okay, what were we working on?
457
00:30:32.370 --> 00:30:33.290
What did we run into?
458
00:30:36.330 --> 00:30:39.231
But it's, I mean, I'm having the most prolific period of my career.
459
00:30:39.251 --> 00:30:41.151
I'm having way more fun than I've ever had.
460
00:30:41.171 --> 00:30:44.092
And I'm working in languages.
461
00:30:44.152 --> 00:30:44.472
I don't know.
462
00:30:44.672 --> 00:30:44.932
Go ahead.
463
00:30:44.972 --> 00:30:45.732
Do you miss coding?
464
00:30:46.953 --> 00:30:47.333
I don't.
465
00:30:47.953 --> 00:30:49.053
I...
466
00:30:52.248 --> 00:30:54.248
because the part for me was always creation.
467
00:30:54.468 --> 00:30:55.849
It was creating the products.
468
00:30:56.369 --> 00:31:12.271
There, like, I guess there were times when what I was doing was golfing something down when I was, like, building bootloaders for, like, embedded bootloaders for keyboard firmware, and it needed to be, the assembly needed to be under 800 bytes, and I was at 806.
469
00:31:12.311 --> 00:31:19.673
And, like, that's a fun game, but it's one fun game out of many.
470
00:31:20.713 --> 00:31:20.973
Um...
471
00:31:22.160 --> 00:31:27.222
Also, it turns out these guys are really good at, at, at, at, at golf.
472
00:31:27.782 --> 00:31:34.705
My height, you know, my, my smallest capable agent is fits in a, is under 500 characters.
473
00:31:34.745 --> 00:31:36.546
Now I don't think it fits in a classic tweet.
474
00:31:38.947 --> 00:31:40.468
Jesse, what do you, what do you call what you do?
475
00:31:40.508 --> 00:31:43.149
Because I, I feel like this is more than vibe coding.
476
00:31:44.302 --> 00:31:51.647
Um, I mean, I, I call it engineering and design and management and making stuff.
477
00:31:51.868 --> 00:32:07.439
Usually it's just making stuff, but I've, um, I like my friend Simon Willison's distinction of like, he points out that if you go and read Carpathia's original tweet about what vibe coding was, it was explicitly when you were just kind of.
478
00:32:09.049 --> 00:32:10.229
doing it as a throwaway.
479
00:32:10.890 --> 00:32:25.956
And when you're not thinking it through, and Simon calls this thing agentic engineering, like what we're doing here, where you are doing planning and where they're gonna be tests and where you care about what you're doing.
480
00:32:28.177 --> 00:32:32.839
I've never been a huge fan of labels because everybody uses labels to mean whatever they want them to mean.
481
00:32:34.658 --> 00:32:36.740
My friends and I had this question.
482
00:32:36.760 --> 00:32:38.601
We were on a hike in New Zealand.
483
00:32:39.361 --> 00:32:39.582
Yeah.
484
00:32:39.902 --> 00:32:41.723
And we were like, what is it like?
485
00:32:41.763 --> 00:32:42.904
It's not vibe coding anymore.
486
00:32:42.944 --> 00:32:43.985
When you're being careful, right?
487
00:32:44.005 --> 00:32:44.825
It's not vibe coding.
488
00:32:44.865 --> 00:32:50.309
Like it's this new age of programming, of coding, but like we got to have something to call it.
489
00:32:50.349 --> 00:32:54.573
So we're like, well, the natural thing would be, like you said, agentic engineering or agentic coding.
490
00:32:55.133 --> 00:32:56.414
But like those, that's a mouthful.
491
00:32:56.454 --> 00:32:58.055
Nobody's going to be like, what do you do?
492
00:32:58.075 --> 00:33:00.177
And I'm like, oh, I'm an agentic engineer.
493
00:33:00.197 --> 00:33:00.297
Yeah.
494
00:33:00.837 --> 00:33:01.057
Right.
495
00:33:01.157 --> 00:33:05.441
So I was like, I mean, people are going to shorten it like we always do or do portmanteaus.
496
00:33:05.481 --> 00:33:06.022
So they'll stall.
497
00:33:06.042 --> 00:33:08.143
They're still calling it agentic.
498
00:33:08.183 --> 00:33:09.284
So agentic coding.
499
00:33:09.925 --> 00:33:13.528
And then, you know, they'll they'll shorten the syllables and it'll just be called genticoding.
500
00:33:14.129 --> 00:33:16.291
And then be like, isn't it really coding because you're not coding?
501
00:33:16.331 --> 00:33:18.092
So they'll just call it gentic.
502
00:33:18.633 --> 00:33:19.253
But that's weird.
503
00:33:19.293 --> 00:33:20.534
So they'll call it genting.
504
00:33:21.135 --> 00:33:22.076
And so that's what we call it now.
505
00:33:22.216 --> 00:33:23.177
We call it genting.
506
00:33:23.897 --> 00:33:24.177
Sure.
507
00:33:24.538 --> 00:33:25.118
Seems reasonable.
508
00:33:26.860 --> 00:33:27.100
Yeah.
509
00:33:29.058 --> 00:33:36.965
It's like they, I've been listening to some podcasts and they talk about their pod and it's just like, I can't, I can't get that word.
510
00:33:37.285 --> 00:33:38.706
You're like five years behind, man.
511
00:33:38.746 --> 00:33:40.167
Everybody just calls it a pod now.
512
00:33:40.187 --> 00:33:41.468
What era are you from?
513
00:33:41.628 --> 00:33:46.673
Like the pod was the literal iPod and the cast was the thing that went on your iPod.
514
00:33:47.053 --> 00:33:49.195
That was what Ben meant when he named the thing.
515
00:33:52.217 --> 00:33:52.517
Okay.
516
00:33:52.537 --> 00:33:52.778
All right.
517
00:33:52.798 --> 00:33:53.398
I got a spec.
518
00:33:53.799 --> 00:33:54.319
So what do we do?
519
00:33:54.359 --> 00:33:55.020
You got a spec.
520
00:33:55.080 --> 00:33:55.960
I'll look at the spec.
521
00:33:55.980 --> 00:33:56.220
Yeah.
522
00:33:56.621 --> 00:33:57.742
It's not wrong to look at the spec.
523
00:33:58.393 --> 00:34:02.880
Um, so if you've got a favorite Markdown browser or, you know, cat.
524
00:34:03.862 --> 00:34:04.162
Yeah.
525
00:34:04.182 --> 00:34:07.888
Let me just do it in the, can I just, can I exit out of Claude to do it?
526
00:34:08.696 --> 00:34:09.857
You should leave Cloud Open.
527
00:34:09.917 --> 00:34:11.179
It's going to screw it up.
528
00:34:11.219 --> 00:34:14.582
So I would want to hang on to this session, but you could resume the session.
529
00:34:15.403 --> 00:34:17.806
But if you can pop, it's a spec.
530
00:34:18.146 --> 00:34:21.090
It's somewhere between a vision document and a spec.
531
00:34:21.150 --> 00:34:29.639
It's not like it's supposed to be a little bit less formal than a real PRD, but it's trying to describe what it thinks you just agreed on with it.
532
00:34:30.280 --> 00:34:30.480
Okay.
533
00:34:31.197 --> 00:34:36.661
And this is different than, because superpowers will, brainstorming will create an implementation plan after this.
534
00:34:37.401 --> 00:34:42.345
Also, brainstorming does the spec, and then writing plans does the implementation plan based on this.
535
00:34:42.825 --> 00:34:46.748
But this is the, what we think we're supposed to be making.
536
00:34:47.068 --> 00:34:48.669
And then the plan is how to get there.
537
00:34:49.229 --> 00:34:55.454
And so Claude's plan sort of conflates the two of those, but I find them to be very, I've always found them to be different things.
538
00:34:58.516 --> 00:34:58.696
Okay.
539
00:34:58.716 --> 00:34:58.836
Okay.
540
00:35:00.125 --> 00:35:00.365
All right.
541
00:35:00.386 --> 00:35:08.021
So if I'm doing this correctly, I would spend a lot of time reading this, really making sure that all of this is correct, because this is where everything starts, right?
542
00:35:08.041 --> 00:35:09.323
Like this is the thing.
543
00:35:09.704 --> 00:35:10.786
This is the thing that we're building.
544
00:35:11.783 --> 00:35:13.845
Yes, but.
545
00:35:14.165 --> 00:35:21.029
So I think in that intro you mentioned that, like, my first day with Cloud Code, I made this crazy monstrosity.
546
00:35:21.550 --> 00:35:28.174
I built the app development platform I thought I wanted that had draggable widgets and a Gentic everything.
547
00:35:28.634 --> 00:35:33.538
And this is the standard problem with Gentic is...
548
00:35:34.298 --> 00:35:37.499
If you ask it to build a thing, it'll just throw in all the features.
549
00:35:38.000 --> 00:35:43.622
It's the opposite of the way the problem we used to have where you could only get to the skeleton.
550
00:35:44.182 --> 00:35:47.603
And now you get to the crazy thing and you have to try to pare it back.
551
00:35:48.204 --> 00:35:51.505
And so there's this image that circulates around the internet about Homer Simpson's car.
552
00:35:52.005 --> 00:35:58.847
where Homer Simpson envisioned a car, it has three TVs, two steering wheels, an antenna.
553
00:35:58.947 --> 00:36:01.048
And how do you get back to a reasonable car?
554
00:36:01.808 --> 00:36:07.910
And it is almost never the case that you can take the crazy monstrosity and pair it back and have it be coherent.
555
00:36:08.610 --> 00:36:11.311
But what we have now is waterfall that's very, very fast.
556
00:36:11.831 --> 00:36:17.333
And so it is okay to not read this spec if you understand that this might be the one to throw away.
557
00:36:18.878 --> 00:36:24.385
And so it is one of the patterns that I use sometimes is I do this.
558
00:36:24.886 --> 00:36:25.647
I don't read it.
559
00:36:26.247 --> 00:36:27.269
I approve the plan.
560
00:36:27.869 --> 00:36:28.831
I don't read it.
561
00:36:29.251 --> 00:36:30.393
I let it build.
562
00:36:30.973 --> 00:36:36.180
And then I look at the thing it built and I then have a much better understanding of what I really want it.
563
00:36:37.303 --> 00:36:37.463
Hmm.
564
00:36:37.843 --> 00:36:43.925
And so you use that as a prototyping phase and then you go back and now you have more context with which to modify.
565
00:36:43.985 --> 00:36:52.368
And like, I will sometimes do like end to end dev five times and it's all prototyping and it's all, and none of it is sunk cost.
566
00:36:52.448 --> 00:36:57.090
Like I did spend some tokens, but it's not that many tokens and it's yeah.
567
00:36:57.650 --> 00:36:59.451
Jesse, this is what I want as a user.
568
00:36:59.711 --> 00:37:20.165
If I can put in a user request on this pod is auto mode for, for, um, for, you have, you all have me using pod now for brainstorming because, um, I want to, you know, I want to, I want to answer the questions up front and then just one shot the rest to your point and just rapid iterate on when software is free.
569
00:37:20.825 --> 00:37:21.346
Yeah.
570
00:37:21.406 --> 00:37:23.187
Why not actually look at the, the.
571
00:37:24.870 --> 00:37:40.142
So the way to get that would be, so if you're willing to answer some of those questions up front, let it get to a spec, and then you want to go full auto, you could say, since it is about to try to write an implementation plan, you could say, you know, write, you know,
572
00:37:41.765 --> 00:37:44.927
If you're doing it naively, you can say that looks good.
573
00:37:44.947 --> 00:37:45.488
What's next?
574
00:37:45.528 --> 00:37:46.889
And it will step you through everything.
575
00:37:47.309 --> 00:37:52.592
But you could now say write the implementation plan and then implement it with subagent driven development.
576
00:37:53.033 --> 00:37:56.255
And then we could go and do her and then we could keep chatting while it goes and does its thing.
577
00:37:57.675 --> 00:38:01.538
OK, this is actually a good point right here, Tom, to use Cloud Goals.
578
00:38:02.138 --> 00:38:03.159
Have you used goals before?
579
00:38:04.099 --> 00:38:04.680
No, teach me.
580
00:38:06.273 --> 00:38:16.884
Goals, I feel like, might be God's gift to humankind over the last month.
581
00:38:17.044 --> 00:38:18.866
They came from the Ralph loop.
582
00:38:19.427 --> 00:38:22.110
Jesse, I'm sure you're using goals quite a bit.
583
00:38:23.311 --> 00:38:24.812
I use goals on and off.
584
00:38:25.013 --> 00:38:25.493
I use goals.
585
00:38:26.859 --> 00:38:36.405
I don't generally use goals inside a superpowers flow because I already had that because the core superpowers flow will already get should all were to get you all the way through without needing one.
586
00:38:37.705 --> 00:38:42.448
OK, because I've used the goal to to to get it to work autonomously to the outcome.
587
00:38:43.569 --> 00:38:50.753
But but if the outcome is the superpowers plan, that that is what the superpower subagent room and development skill should already do.
588
00:38:52.013 --> 00:38:52.193
Okay.
589
00:38:52.414 --> 00:38:53.155
So, yeah.
590
00:38:54.176 --> 00:39:01.286
Do you ever, because these skills need to trigger and they're not exactly deterministic, do they ever fail?
591
00:39:01.506 --> 00:39:07.795
Like, will it ever miss and be like, all right, like, I don't, you know, it'll just go off and do what it normally does and be like, I don't need that skill.
592
00:39:08.556 --> 00:39:19.105
Um, so they, they don't, it is relatively rare in my experience, but, but like, because these are non-deterministic systems, everybody has different experiences.
593
00:39:19.866 --> 00:39:23.189
Um, and we have like, we have our own evil hardest to tune this stuff.
594
00:39:23.769 --> 00:39:29.653
But it is relatively rare that it will just not, that it will just go off and start doing something completely different.
595
00:39:30.254 --> 00:39:37.679
But I do occasionally catch, especially Claude, saying things like, oh, this was a small implementation task.
596
00:39:38.040 --> 00:39:39.461
I'll do the code review myself.
597
00:39:40.631 --> 00:39:42.733
And we've done some tuning for that.
598
00:39:43.173 --> 00:39:54.382
Part of the trick for getting it to be better about not doing that is explaining that the reason that we use a subagent to do that review is partially not to pollute the reviewer's context window.
599
00:39:55.023 --> 00:39:56.824
Or the orchestrator's context window.
600
00:39:56.884 --> 00:39:57.044
Yeah.
601
00:39:58.346 --> 00:40:00.868
And that helps a lot.
602
00:40:02.289 --> 00:40:05.852
I find that the OpenAI models are...
603
00:40:07.031 --> 00:40:11.453
much more dogmatic about following the rules than Anthropix models.
604
00:40:12.173 --> 00:40:17.655
OpenAI's models have gotten a little more relaxed and Anthropix later models have gotten a little bit more dogmatic.
605
00:40:18.255 --> 00:40:33.780
But when we first ported superpowers to Codex, I fired it up and one of the first rules in brainstorming is you need to use the to-do write tool to write out the list of tasks you need to do in this session and Codex freaked out.
606
00:40:34.830 --> 00:40:35.951
I don't have a to-do write tool.
607
00:40:36.392 --> 00:40:38.615
Let me make sure I don't have an MCP that has a to-do write tool.
608
00:40:38.715 --> 00:40:39.476
No, I don't have one of those.
609
00:40:39.856 --> 00:40:41.138
Let me check the current directory.
610
00:40:41.438 --> 00:40:42.940
No, there's no to-do write tool there.
611
00:40:43.281 --> 00:40:48.447
All right, I'm going to search the entire disk for a to-do write tool because Codex's tool is, I think, called task.
612
00:40:49.289 --> 00:40:49.809
And so...
613
00:40:51.228 --> 00:40:59.793
That led to we now have essentially a document for every agent harness that Superpower supports.
614
00:41:00.253 --> 00:41:01.914
When it starts up, it reads that doc.
615
00:41:01.954 --> 00:41:07.238
It says, when Superpower says you need the such and such tool, what that really means is you need this tool.
616
00:41:08.842 --> 00:41:09.982
And that fixes it.
617
00:41:10.243 --> 00:41:11.883
So there's a translation layer?
618
00:41:11.983 --> 00:41:12.944
There's a translation table.
619
00:41:13.284 --> 00:41:13.564
Yeah.
620
00:41:13.864 --> 00:41:14.544
Okay.
621
00:41:14.824 --> 00:41:15.525
Yeah.
622
00:41:15.545 --> 00:41:16.385
It's a lookup table.
623
00:41:17.365 --> 00:41:22.647
Is there a trick for you to tell the LLM to like really pay attention to this bit?
624
00:41:22.707 --> 00:41:38.033
Like we talked about the iron law before, but like, is there something that works across everything that's like, like, I know there's a bunch of crap in here, but like, really, this is the thing, like always make sure that you're reading this part, like no matter what, pay attention, like more attention here versus all this other back and forth that's going on.
625
00:41:38.780 --> 00:41:41.562
Um, so repetition can help.
626
00:41:41.582 --> 00:41:45.444
So your direct prompt will matter more than anything on the skills.
627
00:41:46.285 --> 00:41:58.412
Um, if you want to, you know, if, if there is one part of the skill that is dramatically more important than all of the other parts of the skill, maybe some of those other parts of the skill need to get rethought or broken or broken out.
628
00:41:59.353 --> 00:42:03.756
Um, anything that you can do to move forward.
629
00:42:06.022 --> 00:42:16.149
work into what I call classical software, software that is imperative and written in a programming language rather than a management handbook, which is what this is.
630
00:42:16.629 --> 00:42:21.893
Anything you can do to move repeatable stuff into code is good.
631
00:42:22.533 --> 00:42:28.658
So there are little bits of superpowers where we have written scripts to do things.
632
00:42:30.759 --> 00:42:32.020
We've been playing with...
633
00:42:34.497 --> 00:42:37.183
Specialized tooling for end-to-end testing.
634
00:42:37.725 --> 00:42:41.674
We've been playing with moving parts of brainstorming into full-on software.
635
00:42:42.557 --> 00:42:42.777
Um...
636
00:42:43.882 --> 00:42:53.105
I would love to move most of Superpower's orchestration into software, but that gets complicated because we need to support a bunch of harnesses.
637
00:42:53.906 --> 00:42:58.007
And like the way it should work is that your orchestrator is outside your coding agent.
638
00:42:58.527 --> 00:43:04.850
But with the changes Anthropic has been making to what they will allow you to use a subscription for.
639
00:43:06.005 --> 00:43:07.926
Mostly because of open clause, I understand that.
640
00:43:08.506 --> 00:43:15.208
Um, you can no longer use the Claude Agent SDK or Claude-P and use your subscription.
641
00:43:15.428 --> 00:43:16.508
Now you only use credits.
642
00:43:17.028 --> 00:43:26.731
And so it would be hurting an awful lot of people if I effectively broke superpowers by moving that stuff into classical software.
643
00:43:29.112 --> 00:43:29.392
Okay.
644
00:43:29.638 --> 00:43:33.240
It looks like my plan or my, sorry, my, yeah, this is the plan.
645
00:43:33.260 --> 00:43:34.221
So the preview one was a spec.
646
00:43:34.761 --> 00:43:35.921
And this one is the plan.
647
00:43:36.842 --> 00:43:42.285
And so I'm going to look at the plan.
648
00:43:42.745 --> 00:43:44.006
All right, so walk me through the plan here.
649
00:43:44.526 --> 00:43:44.766
All right.
650
00:43:45.267 --> 00:43:51.350
So first thing, it's telling the agentic worker, here's how you use this plan.
651
00:43:53.131 --> 00:43:58.814
And then it gives it the high-level goal, tells the tech stack the map of where things should be ending up.
652
00:44:02.341 --> 00:44:07.182
And so like task one is we're going to create the project bootstrap, add some depths.
653
00:44:07.962 --> 00:44:22.645
And so one of the things that was not true when I created superpowers, but has become more true over time as Anthropocles made changes to the Claude model, they've gotten more and more fond of shoving as much code as possible into the plan.
654
00:44:23.585 --> 00:44:30.927
And this is a thing that initially looks pretty weird because why, you know, why are you putting all the code in the plan up front?
655
00:44:31.907 --> 00:44:37.953
And one of the things that does is it improves the coherency of the app by having it in one place.
656
00:44:38.754 --> 00:44:46.041
It is, it was essentially one gigantic call upfront to scaffold and sketch out most of the, of the code.
657
00:44:46.381 --> 00:44:51.186
But we're not using Opus to do all those tool calls of writing files and running tests.
658
00:44:51.606 --> 00:44:55.090
That stuff that it can generally do with cheaper, faster models.
659
00:44:56.885 --> 00:45:00.970
when you use GPT, you end up with less code in the plan.
660
00:45:01.010 --> 00:45:03.813
And we're playing with tuning this some more.
661
00:45:04.534 --> 00:45:06.396
But I think there is a happy middle ground.
662
00:45:07.257 --> 00:45:07.497
But...
663
00:45:08.487 --> 00:45:16.449
Having some code in the plan means that it essentially infects the agents with the style you want and the naming you want.
664
00:45:17.209 --> 00:45:18.450
And that is super valuable.
665
00:45:19.290 --> 00:45:25.132
There's one other really cool techniques that is not part of superpowers, but that I sometimes use.
666
00:45:25.872 --> 00:45:28.012
And it's what some of my friends call gene transfer.
667
00:45:28.873 --> 00:45:29.913
I often just call it inspo.
668
00:45:30.273 --> 00:45:37.055
I have an inspo get ignored sub directory in a project, and I have other tools and code bases that are...
669
00:45:38.195 --> 00:45:41.437
that have something I like about them that is related to what I want.
670
00:45:42.117 --> 00:45:58.845
And I, and I will tell the agent before you do this, go look in the inspo directory at, you know, you know, at how, at how the codex code base does this thing or how, you know, you know, how Git implement something and some something or other.
671
00:45:59.565 --> 00:45:59.806
And yeah.
672
00:46:00.846 --> 00:46:10.453
It essentially puts the model into this headspace where it is thinking about the kinds of patterns and the kinds of designs it saw.
673
00:46:11.093 --> 00:46:13.735
And it will bring that into whatever it's building.
674
00:46:15.476 --> 00:46:22.160
Speaking of inspiration, Jesse, do you feel like the Foundation Labs are taking inspiration from superpowers?
675
00:46:22.200 --> 00:46:28.785
Because as a user, I'm seeing some very similar prompting.
676
00:46:29.952 --> 00:46:31.795
Um, so, like, I've seen...
677
00:46:32.756 --> 00:46:33.276
I don't know.
678
00:46:33.296 --> 00:46:34.778
I don't, like, I...
679
00:46:35.239 --> 00:46:38.663
I don't really have much contact with the Cloud Code team.
680
00:46:39.544 --> 00:46:40.886
Um, I...
681
00:46:41.928 --> 00:46:46.709
You know, I know that there are plenty of folks at OpenAI who are using superpowers as superpowers.
682
00:46:47.370 --> 00:46:57.833
I know that some of the stuff that the Claude plan mode, some of how Anthropic made Claude plan mode trigger looks a lot like my stuff.
683
00:46:58.373 --> 00:46:59.853
It's MIT licensed.
684
00:47:00.634 --> 00:47:11.157
I would be thrilled for all of the hardest implementing labs to essentially steal everything I'm doing because that means I get to do other stuff.
685
00:47:11.977 --> 00:47:13.698
And I wouldn't have to maintain superpowers.
686
00:47:13.758 --> 00:47:14.758
It would just work for everybody.
687
00:47:14.798 --> 00:47:21.061
Every like until that happens, well, you know, we're just going to make superpowers work as well as we can for everybody.
688
00:47:21.402 --> 00:47:25.664
Like, let's say I'm like, I don't need I'm just doing a like a just a like, get me a prototype.
689
00:47:25.684 --> 00:47:26.484
I don't care about tests.
690
00:47:26.524 --> 00:47:27.965
I don't care about anything.
691
00:47:27.985 --> 00:47:28.985
I just build me this thing.
692
00:47:29.145 --> 00:47:30.046
How do I turn it off?
693
00:47:30.746 --> 00:47:36.189
Um, so the so one one option is you get a cloud code allows you to disable plugins.
694
00:47:36.958 --> 00:47:42.500
You can also enable a plugin with dash dash plugin dir and a pointer to a checkout of the plugin.
695
00:47:43.761 --> 00:47:48.823
Another option is when it fires up brainstorming, you can say, let's not do brainstorming, this is simple.
696
00:47:49.283 --> 00:47:51.324
Or let's skip TDD this time.
697
00:47:51.704 --> 00:47:55.366
Because the user prompt will trump the skills.
698
00:47:55.386 --> 00:48:05.750
Yeah, and in fact, Superpowers has explicit instructions that any explicit prompt from the user overrides any guidance from Superpowers.
699
00:48:07.215 --> 00:48:20.139
So people ask me for like, I want to hook system in superpowers, and they should be able to just in their claw.md or agents.md say, you know, say when superpowers want you to do X, do Y first or do Y instead.
700
00:48:20.879 --> 00:48:25.180
Like, or like I use JJ or, you know, what, or what have you.
701
00:48:26.061 --> 00:48:26.941
It's reliable.
702
00:48:27.021 --> 00:48:28.821
It'll reliably trigger that.
703
00:48:29.201 --> 00:48:32.742
I am not getting enough bug reports to indicate to me that it is not reliable.
704
00:48:32.802 --> 00:48:34.463
But if it's not reliable, I want to hear about it.
705
00:48:34.763 --> 00:48:39.569
I've been pleased when superpowers is invoked autonomously.
706
00:48:39.749 --> 00:48:40.830
Usually I have to invoke it.
707
00:48:41.030 --> 00:48:46.397
And every time it's been invoked autonomously, I felt like, okay, this is useful in this moment.
708
00:48:47.411 --> 00:48:51.535
Okay, I want to go back to your comment regarding the Foundation Labs, Jesse.
709
00:48:52.136 --> 00:48:52.376
Yeah.
710
00:48:52.416 --> 00:48:57.140
Incorporating this work and the broader dynamics that you're seeing as an engineer.
711
00:48:57.561 --> 00:49:03.066
And maybe I'll take this moment to pull in a Reddit user's question.
712
00:49:03.827 --> 00:49:04.147
Sure.
713
00:49:04.187 --> 00:49:11.254
This is nearbyyam286 who asks, what will Jesse do when AI can do his job?
714
00:49:13.253 --> 00:49:15.875
Um, nobody has bothered to write software.
715
00:49:16.875 --> 00:49:30.222
Um, and so I, yeah, I, I, I will like, when we think about superpowers in this incredible product that, that now many engineers are using, how, how do you think about defensibility?
716
00:49:30.322 --> 00:49:37.386
Like as you approach your, um, your own startup and, and, um, the future of your career?
717
00:49:38.070 --> 00:49:41.333
Yeah, it's I mean, it's a complicated time to be building startups.
718
00:49:41.814 --> 00:49:53.765
I'm in this weird place where my hobby project found PMF before, you know, before, you know, before, almost before I noticed, which it's a wild thing.
719
00:49:53.785 --> 00:49:54.226
And it's great.
720
00:49:54.246 --> 00:49:55.947
And it's great to see that it's useful for folks.
721
00:49:56.808 --> 00:50:00.412
I have always wanted to work myself out of my jobs like this.
722
00:50:01.667 --> 00:50:06.589
Way back when I was a FES admin, I was writing shell scripts to try to work myself out of parts of my job.
723
00:50:07.310 --> 00:50:10.531
And my goal is to make stuff.
724
00:50:10.551 --> 00:50:12.732
And I have weird taste.
725
00:50:13.572 --> 00:50:18.234
And haste and judgment are the things that are human.
726
00:50:18.575 --> 00:50:23.177
And they are things that you can encode into a prompt or a model.
727
00:50:23.837 --> 00:50:30.520
But what makes you different is what makes the thing you're building worth building.
728
00:50:31.750 --> 00:50:40.479
And, you know, GitHub wasn't the first forge, but, you know, what they built was meaningfully different.
729
00:50:40.499 --> 00:50:47.045
It had different tastes and it worked better because they had weird different ideas that were good.
730
00:50:47.906 --> 00:50:52.070
And it is now possible to do more amazing stuff.
731
00:50:52.993 --> 00:50:55.777
I am expecting that what we're going to see is a much longer tail.
732
00:50:56.377 --> 00:51:01.844
So things that used to only be doable by a large team are now doable by a small team.
733
00:51:02.264 --> 00:51:06.970
Stuff that used to take a team is now something that an individual can do in their spare time.
734
00:51:07.771 --> 00:51:08.332
And so...
735
00:51:09.657 --> 00:51:16.181
We're going to see so much more software and so much of it's going to be garbage, but so much of it's already garbage.
736
00:51:16.541 --> 00:51:18.763
There's going to be all sorts of great stuff.
737
00:51:18.943 --> 00:51:26.668
Like we're starting to see the weird Internet that we used to have in the 90s come back because you can you have the time, you have the capability.
738
00:51:27.308 --> 00:51:32.572
And now and now you're you're weird stuff that, you know, sometimes it's effectively a shit post.
739
00:51:34.004 --> 00:51:38.987
It's not just you made a mock-up of a webpage for a crazy product, you ship the product.
740
00:51:39.767 --> 00:51:40.367
It's great.
741
00:51:41.648 --> 00:51:44.530
I am so excited to constantly be working myself out of a job.
742
00:51:44.810 --> 00:51:52.694
And by doing that, you're creating value for others and ultimately empowering yourself.
743
00:51:52.974 --> 00:51:54.375
That is the hope and the dream.
744
00:51:55.475 --> 00:52:08.678
What, okay, one more question from Reddit, Familiarair3528 asks, what seemed like a good approach to Jesse, but ended up being a dead end or inefficient?
745
00:52:08.778 --> 00:52:12.879
What were you most surprised by in the, let's say in the last year?
746
00:52:13.539 --> 00:52:14.340
That's a great question.
747
00:52:14.360 --> 00:52:19.181
I'm trying to think of like things I have tried that have not worked well.
748
00:52:21.742 --> 00:52:22.942
This one might take me a second.
749
00:52:25.980 --> 00:52:35.195
One of the dead ends that I didn't manage to get working, but it sounds like maybe Anthropic has now, was I built essentially a phone-a-friend for...
750
00:52:35.876 --> 00:52:38.320
I was trying to figure out if I could get Haiku to be, like,
751
00:52:38.915 --> 00:52:41.657
to be my main model in club code because it is fast and cheap.
752
00:52:42.258 --> 00:52:54.348
And so I had built it an ask opus and an ask sonnet to try to get it to understand when it was out of its depth and to ask another model for help.
753
00:52:54.929 --> 00:52:59.433
And part of it might have been that I was giving it really hard problems like solve the Riemann conjecture.
754
00:53:00.097 --> 00:53:08.284
which is a, you know, well-known unsolved problem, and Haiku would tell me things like, yeah, Jesse, that is a completely unsolvable problem.
755
00:53:08.364 --> 00:53:09.345
Even Opus can't do it.
756
00:53:10.346 --> 00:53:18.053
So I renamed it to Ask Perfect Super Intelligence, and it was still unwilling to ask the perfect super intelligence whether it could solve the remake conjecture.
757
00:53:18.654 --> 00:53:18.894
But...
758
00:53:20.347 --> 00:53:24.328
Like I try lots and lots of little experiments.
759
00:53:24.628 --> 00:53:29.649
And so there's a lot of stuff where it's like, yeah, maybe it didn't work, but it wasn't a very big deal.
760
00:53:31.769 --> 00:53:36.430
I did some like one of the engineering coding harnesses that like test coding harnesses I built.
761
00:53:36.910 --> 00:53:40.691
I spent time trying to get it to Hillclimb terminal bench.
762
00:53:41.471 --> 00:53:45.092
And it is very hard to convince
763
00:53:46.527 --> 00:53:56.613
an agent, an agentic improvement loop, like, uh, Carpathie's, uh, auto research to not try to game the test when it is trying to hill climb.
764
00:53:57.394 --> 00:54:08.720
And so try that level of prompt engineering for each an agent to optimize across a lot, a lot of different benchmarks in a way that is not cheating.
765
00:54:09.261 --> 00:54:11.242
I have not had a lot of success with yet.
766
00:54:11.577 --> 00:54:12.358
Tom, how's it looking?
767
00:54:12.378 --> 00:54:16.220
Is it, are the agents done building GitHub?
768
00:54:18.282 --> 00:54:20.163
I think they're going to take a little while.
769
00:54:20.283 --> 00:54:23.025
This is a significant project, I imagine that.
770
00:54:23.145 --> 00:54:26.648
How long would you expect something like this to take to actually run through and implement, Jesse?
771
00:54:26.948 --> 00:54:36.074
Well, I mean, it depends a lot on what the actual project is, but that's one of the things that I believe pretty fundamentally about doing this stuff well.
772
00:54:36.755 --> 00:54:37.775
It doesn't go fast.
773
00:54:39.076 --> 00:54:39.737
You can either have it,
774
00:54:40.823 --> 00:54:42.346
It's a whole good, fast, cheap thing.
775
00:54:42.807 --> 00:54:53.006
You can have an agent swarm, go implement 60 things at once, stepping on each other's toes, spilling tons of tokens on the floor and having trouble coordinating.
776
00:54:53.026 --> 00:54:53.727
But...
777
00:54:53.932 --> 00:54:59.293
Or you can have a little bit of rigor and it takes time to, you know, to, to iterate serially.
778
00:54:59.833 --> 00:55:14.836
And so it is not uncommon when I have a significant plan coming out of a Claude or Codex for it to run for, you know, minimum three hours or 12 hours or 24 hours building step by step.
779
00:55:15.536 --> 00:55:18.077
And it took like, it depends on exactly how far you want it to go.
780
00:55:20.622 --> 00:55:29.689
If I'm doing that, like that thing where I'm having it do a prototype and I know I'm going to throw it away, I will sometimes tell it, you know what, let's, you know, let's skip the TDD.
781
00:55:30.250 --> 00:55:31.831
Let's loosen the reviews.
782
00:55:32.552 --> 00:55:34.173
We're, you know, we're building a prototype.
783
00:55:34.493 --> 00:55:35.774
It'll never see the light of day.
784
00:55:35.854 --> 00:55:36.455
Don't worry.
785
00:55:37.496 --> 00:55:46.103
And I sometimes have to tell it not to worry because it is tuned to, well, if you're building software, you don't want to, you don't want to screw it up and, you know, should add software.
786
00:55:47.609 --> 00:55:48.589
It's a little bit of priming there.
787
00:55:48.609 --> 00:55:48.810
All right.
788
00:55:48.830 --> 00:55:50.991
Well, let me try to sum up what I've learned today.
789
00:55:51.751 --> 00:55:58.094
So I think one of the biggest things for me in building this kind of a thing, like this is software in a way, but it's also not right.
790
00:55:58.134 --> 00:56:00.375
Like it's skills development, essentially.
791
00:56:01.036 --> 00:56:09.560
And skills development is really management or we become the robo psychologist who's like, all right, let me investigate like why the machine didn't do what I expect it to do.
792
00:56:10.200 --> 00:56:12.462
Given, you know, the rules, right?
793
00:56:12.482 --> 00:56:14.263
We give it whatever our basic rules are.
794
00:56:14.283 --> 00:56:19.127
And it's like, it found a way around the rules by deleting code or whatever.
795
00:56:19.147 --> 00:56:21.509
And you're like, all right, don't delete for the test.
796
00:56:21.529 --> 00:56:22.490
Don't delete the test, right?
797
00:56:22.510 --> 00:56:24.992
You got to go in there and like continue back and forth.
798
00:56:25.092 --> 00:56:30.116
So that's, that's one thing like skills development, like this is a completely different thing than programming.
799
00:56:31.524 --> 00:56:42.210
And then I'd say another, another really great thing to take here is as code becomes cheap, we can rethink how we use code and we can use it for throwaway things.
800
00:56:43.151 --> 00:56:48.634
So take the spec, get a quick implementation to get more information.
801
00:56:49.314 --> 00:56:53.497
That iterative cycle is everything in software to me, always fast feedback cycles.
802
00:56:54.224 --> 00:56:55.705
This is the thing that works.
803
00:56:56.185 --> 00:56:59.946
And you can do it for yourself, which is like, I don't even know what the problem is necessarily.
804
00:57:00.366 --> 00:57:04.928
You know, when I started this task in the brainstorming, I'm like, I want a GitHub clone.
805
00:57:04.948 --> 00:57:07.189
But like, I didn't even really know what that meant.
806
00:57:07.269 --> 00:57:08.529
I didn't even really know what I wanted.
807
00:57:09.190 --> 00:57:11.070
And the brainstorming like is the thing.
808
00:57:11.210 --> 00:57:11.991
It is the process.
809
00:57:12.031 --> 00:57:15.672
It's like, help me discover what I even mean, what I want.
810
00:57:15.692 --> 00:57:16.272
Like, I don't even know.
811
00:57:16.312 --> 00:57:17.293
I haven't explored that fully.
812
00:57:18.133 --> 00:57:20.215
We used to do this through writing and meetings and stuff.
813
00:57:20.255 --> 00:57:20.455
Right.
814
00:57:20.495 --> 00:57:24.899
But like having a sparring partner to be like, I don't like, you got to tell me, I don't know what to do here.
815
00:57:24.939 --> 00:57:26.221
Like, tell me what you're after.
816
00:57:27.262 --> 00:57:28.062
That's a big part of it.
817
00:57:28.102 --> 00:57:30.845
So like being, you know, it becomes the consultant for us.
818
00:57:30.885 --> 00:57:31.966
It's a smart rubber duck.
819
00:57:32.446 --> 00:57:32.827
Exactly.
820
00:57:32.887 --> 00:57:33.868
And I think that's really cool.
821
00:57:35.249 --> 00:57:37.191
And so those are two of the, two of the big things.
822
00:57:37.251 --> 00:57:41.295
And I think the other one is like, anyone can go and build something like this for themselves.
823
00:57:41.315 --> 00:57:41.515
Right.
824
00:57:42.602 --> 00:57:42.802
Right.
825
00:57:42.842 --> 00:57:45.904
Like if you don't, you're like, I, this is cool, but like, it's not quite my jam.
826
00:57:45.924 --> 00:57:47.546
It's like, well, you know, go build your own.
827
00:57:47.566 --> 00:57:48.506
You're just writing skills.
828
00:57:48.526 --> 00:57:49.407
Like anyone can do this.
829
00:57:50.488 --> 00:57:51.068
And I think that's right.
830
00:57:51.088 --> 00:57:51.508
Absolutely.
831
00:57:52.009 --> 00:57:52.189
Yeah.
832
00:57:52.649 --> 00:57:54.030
And I mean, and it's MIT license.
833
00:57:54.050 --> 00:57:55.111
You can fork it and change it.
834
00:57:55.351 --> 00:57:56.412
Like, yeah, change this.
835
00:57:56.432 --> 00:57:57.072
You don't have to.
836
00:57:57.773 --> 00:57:57.933
Right.
837
00:57:57.953 --> 00:57:59.474
Like you don't have to start from zero.
838
00:57:59.874 --> 00:58:00.915
Um, yeah.
839
00:58:04.305 --> 00:58:04.766
It's been fun.
840
00:58:05.686 --> 00:58:06.627
Thank you for coming on.
841
00:58:07.488 --> 00:58:12.954
And Tom, how long did it take you to write the first GitHub, the first release?
842
00:58:13.634 --> 00:58:14.015
Oh, boy.
843
00:58:14.035 --> 00:58:16.197
Well, I mean, we had a public beta within six months.
844
00:58:16.377 --> 00:58:17.518
So, like, we really cranked.
845
00:58:17.618 --> 00:58:19.099
I'm super impressed at how fast we were.
846
00:58:19.220 --> 00:58:21.942
We had public launch six months after we started.
847
00:58:22.843 --> 00:58:23.384
Six months.
848
00:58:23.764 --> 00:58:24.625
I think now.
849
00:58:24.685 --> 00:58:25.546
Four or five people, yeah.
850
00:58:26.530 --> 00:58:27.851
And I think now it's six days.
851
00:58:28.812 --> 00:58:32.415
Yeah, I mean, you could do the same thing that we did for that initial version of GitHub.
852
00:58:33.516 --> 00:58:33.717
Yeah.
853
00:58:34.357 --> 00:58:39.642
I mean, if you, I mean, like part of it's just using the thing to even know what you need to build.
854
00:58:39.742 --> 00:58:42.244
So in some senses, there's not necessarily a shortcut like that.
855
00:58:42.925 --> 00:58:49.547
Saying that that would only take a week removes the reality that is you don't know what's good until you use it yourself.
856
00:58:49.607 --> 00:58:55.569
The dogfooding, the process of discovery and usage and refinement is how you make good products.
857
00:58:55.789 --> 00:58:57.329
And that's very difficult to shortcut.
858
00:58:57.469 --> 00:58:59.110
But like the code writing part of it, sure.
859
00:59:00.350 --> 00:59:01.210
that would go much faster.
860
00:59:01.670 --> 00:59:11.073
Yeah, no, like you can, you could knock GitHub off in six days, but you, but yeah, you need that time to figure out what works and what doesn't.
861
00:59:11.713 --> 00:59:20.936
And that part is the taste and the judgment and the iteration that there are ways to use agents to help you, but it takes time.
862
00:59:21.916 --> 00:59:27.058
And the deceptive thing always is that when you use a product and it's great and it's simple and it's perfect,
863
00:59:28.259 --> 00:59:29.520
That is the hardest thing to get to.
864
00:59:29.660 --> 00:59:38.126
And those people that built it, it took them a very long time and a lot of iterations of things that were complicated and didn't make sense and weren't easy to use to get there.
865
00:59:38.206 --> 00:59:39.067
And we never see it.
866
00:59:39.087 --> 00:59:43.129
We're like, wow, look at this pair of AirPods, you know, what elegance.
867
00:59:43.250 --> 00:59:50.374
And it's like the number of iterations that they must have gone through and the hardship and the man to get to something that's this is.
868
00:59:51.375 --> 00:59:54.997
You know, you look at this and you're like, oh, well, I could, you know, I could just, you know, replicate that.
869
00:59:55.157 --> 00:59:56.318
And, you know, and people do, right?
870
00:59:56.398 --> 00:59:57.398
Like manufacturers do.
871
00:59:57.418 --> 00:59:58.859
And they're like, oh, yeah, thanks, Apple.
872
00:59:59.600 --> 00:59:59.800
Right?
873
01:00:00.080 --> 01:00:01.721
Replication is much easier than invention.
874
01:00:02.281 --> 01:00:11.106
And I think invention will always take time and talent and iteration and buying it against the real world where the LLMs don't live.
875
01:00:11.246 --> 01:00:12.267
They don't live in the real world.
876
01:00:12.927 --> 01:00:14.408
And so that's a real limitation they have.
877
01:00:16.755 --> 01:00:19.896
Well, Jesse, thank you for inventing superpowers.
878
01:00:20.016 --> 01:00:30.399
This, yeah, this has become the most used agentic engineering plugin or genting, the most used genting plugin.
879
01:00:30.419 --> 01:00:31.460
And...
880
01:00:33.228 --> 01:00:34.148
I was shocked.
881
01:00:34.609 --> 01:00:37.110
It has 1.7 million GitHub clones in the last week.
882
01:00:37.530 --> 01:00:39.011
So last two, last two weeks.
883
01:00:39.051 --> 01:00:39.251
Yeah.
884
01:00:39.291 --> 01:00:39.931
That's two weeks.
885
01:00:40.031 --> 01:00:40.391
Okay.
886
01:00:41.092 --> 01:00:41.352
Yeah.
887
01:00:41.512 --> 01:00:49.596
So, so a lot, a lot of people out there benefiting from your work and, um, it sounds like you did it to automate your job and empower others.
888
01:00:49.636 --> 01:00:53.778
So that's, yeah, that's an admirable approach to, to genting.
889
01:00:54.791 --> 01:01:04.959
Yeah, no, it's, it, it is genuinely thrilling that it helps people build stuff and I love that it helps folks who have, you know, been building as long as I have to make things.
890
01:01:05.460 --> 01:01:15.788
And I hear so much from folks who, you know, they start with, well, I'm not an engineer, I'm not a coder, but I just shipped my, my third iOS app with superpowers.
891
01:01:18.290 --> 01:01:22.053
And it's like, it, it lets people, it, it.
892
01:01:23.572 --> 01:01:27.736
It helps people avoid what used to have a whole bunch of gatekeeping in front of it.
893
01:01:28.256 --> 01:01:29.818
And it helps people who have ideas make things.
894
01:01:30.218 --> 01:01:31.439
And that makes me so happy.
895
01:01:32.600 --> 01:01:33.281
Democratizing.
896
01:01:34.241 --> 01:01:34.442
Yeah.
897
01:01:35.563 --> 01:01:36.884
Well, thanks so much for having me.
898
01:01:37.284 --> 01:01:38.785
Jesse, Tom, see you next week.
899
01:01:39.806 --> 01:01:40.247
Thanks, Sam.
900
01:01:40.387 --> 01:01:40.907
Thanks, Jesse.
901
01:01:42.108 --> 01:01:47.753
And if you know a developer that hasn't managed engineers, it sounds like superpowers is for them.
902
01:01:48.013 --> 01:01:49.835
So be sure to share this link.