Hari Krishnan is improving AI's skills with intent

WEBVTT
00:00.000 --> 00:01.450
doesn't mean the assertions are clear,
00:01.450 --> 00:03.201
like what you mentioned, the mutation
00:02.476 --> 00:03.476
Right, right.
00:03.201 --> 00:04.201
testing
00:03.661 --> 00:07.032
is there. So then the next level of
tightening needs to happen to say,
00:07.032 --> 00:10.363
hey, if I run mutations, how much
percentage of mutants survive?
00:10.363 --> 00:12.543
That is the next level.
00:12.783 --> 00:16.515
Any test development skill you take like
Matt Pocock or superpowers
00:16.515 --> 00:19.547
or anything or for that matter your own
skill.
00:19.547 --> 00:24.140
So what I usually do is I use like a
Claude dynamic workflow or some sort
00:24.140 --> 00:27.374
of a Ralph loop wherein I put in a rubric
of this measurements.
00:27.374 --> 00:29.623
I put in the base problem statement,
00:29.623 --> 00:33.736
which is the substrate. And then I put in
the skill and I put it through
00:33.948 --> 00:36.210
like a few iterations to see where they
are missing.
00:36.210 --> 00:41.766
And then I have of rules which I can use
and pick one at a time,
00:41.766 --> 00:46.609
keep adding it, keep removing them and
then I come out with a skill file which
00:46.609 --> 00:49.660
I then use for that project. So long story
short,
00:49.660 --> 00:53.743
tuning the skill is necessary for your
code base,
00:52.999 --> 00:55.959
Yeah. Yeah. Mm.
00:53.743 --> 00:55.844
your team and your
00:55.985 --> 00:59.487
models that you're using gives you a
little bit more consistent results than
00:59.903 --> 01:03.966
Just generally saying follow TDD or
generally taking a TDD skill
01:02.941 --> 01:03.941
Right, right.
01:03.966 --> 01:04.966
off the market
01:04.727 --> 01:07.648
Hari Krishnan is the creator at
intentdriven.dev.
01:07.648 --> 01:10.268
He's also been a guest on this podcast
before,
01:10.268 --> 01:13.691
and his episode does really well He also
has some really big news,
01:13.691 --> 01:15.892
so we wanted to have him on today to talk
about that.
01:15.892 --> 01:17.545
Hari, thanks for being on the show today.
01:17.545 --> 01:18.545
I really appreciate your time.
01:18.890 --> 01:21.262
Thanks, Kyle. And I'm really glad to be
back here.
01:21.262 --> 01:27.018
And ~ yeah, I'm glad to see that ~ your
audience are able to resonate with what
01:27.018 --> 01:29.031
we spoke about last time. Yeah,
01:29.031 --> 01:30.031
glad to be back.
01:30.826 --> 01:33.959
Yeah, for sure. Well and I hear you've got
some really big news right now,
01:33.959 --> 01:37.172
Hari Like you've been working on like do
do you have a book deal?
01:37.172 --> 01:39.164
Like come on, man, tell me the tell me the
story.
01:39.915 --> 01:43.430
yeah, I just launched the MEAP,
01:43.430 --> 01:48.481
the Manning Early Access program ~ of my
book called Spec Driven Development,
01:48.481 --> 01:52.202
Engineering with Intent. It went live on
the 31st of August.
01:52.202 --> 01:54.693
So that's the big news. Again,
01:54.693 --> 01:59.624
thank you for your wishes. I've been
working on it for about three to six
months.
01:59.624 --> 02:02.732
I was writing the book even before I
closed on the publisher,
02:02.732 --> 02:06.140
but thanks to the publisher now it's out
there.
02:06.140 --> 02:08.803
looking forward to feedback and it's early
access.
02:08.803 --> 02:13.378
So obviously anyone who's reading it can
give us the feedback and figure
02:13.378 --> 02:15.421
out how we shape the book and take it
forward.
02:16.275 --> 02:18.996
And w early access program, I'm familiar
with it.
02:18.996 --> 02:21.222
I I can almost see the the icon in my
head.
02:21.222 --> 02:25.098
But I I forget, does this mean all the
chapters are there and and it's
02:25.098 --> 02:28.573
a you're sort of getting a pre read before
the the paper book or is this like
02:28.573 --> 02:30.386
a only some selected chapters?
02:31.646 --> 02:32.807
That's a great question. Thanks.
02:32.807 --> 02:35.749
I almost miss mentioning it because it's
an MEAP.
02:35.749 --> 02:38.212
It's only the first three chapters that
are live right now.
02:38.212 --> 02:40.461
It's running at a 50 % site wide sale,
02:40.461 --> 02:43.868
which means people could get on it read
the chapters now.
02:43.868 --> 02:45.430
And of course you get it at 50%.
02:45.430 --> 02:48.572
And eventually when the book goes into
print,
02:48.572 --> 02:51.505
you will get your copy. Yeah.
02:49.274 --> 02:50.384
Okay, you get the whole thing.
02:50.384 --> 02:51.585
Nice, nice, nice.
02:51.585 --> 02:54.647
so much has changed and you weren't on all
that long ago.
02:54.647 --> 02:57.893
what stands out to you as something that ~
is is one of the bigger changes.
02:59.168 --> 03:02.281
one of the things I noticed like I think
your radar itself is brilliant.
03:02.281 --> 03:04.593
It was not there last time and now it's
on.
03:04.593 --> 03:08.176
So thank you for doing that. I think the
space needs it.
03:08.176 --> 03:12.420
I refer to it quite regularly just to see
where people are at.
03:12.420 --> 03:15.443
And of course, like other thought leaders
who've been on your show,
03:15.443 --> 03:20.278
I get a sense of what their headspace is
like in terms of concepts that
03:20.278 --> 03:21.278
are going on.
03:21.045 --> 03:25.357
So going by that, think a topic we
discussed last time is intent
articulation.
03:25.357 --> 03:26.408
And pretty
03:25.894 --> 03:26.894
Yeah.
03:26.408 --> 03:29.318
much like my website goes, it's called
intent-driven development.
03:29.318 --> 03:34.902
on, I used to think I'm indexing or
over-indexing on this term called intent
because
03:34.902 --> 03:37.103
it naturally felt that that's what it
means.
03:37.103 --> 03:40.324
Whether you're doing specification or
you're doing loops,
03:40.324 --> 03:43.886
you're articulating intent to the agent so
that then the agent
03:43.886 --> 03:47.557
can build something rather than trying to
interpret what your intent is.
03:47.877 --> 03:49.287
You make that intent explicit.
03:48.323 --> 03:51.005
Mm-hmm. Just say it. Yeah.
03:49.287 --> 03:50.608
That was the core idea of it.
03:51.028 --> 03:56.023
Yeah. And I think it's good to see that
resonate and a lot more information coming
03:56.023 --> 03:57.514
out there, which is in support of that.
03:57.514 --> 04:01.573
Like for example, the recent playbook that
Anthropic released called
04:01.573 --> 04:04.086
AI native SDLC playbook or something like
that.
04:04.086 --> 04:07.907
And one of the primary things is it's
called this thing called intent.md,
04:07.907 --> 04:09.588
which you capture your intent.
04:09.588 --> 04:14.199
Thereafter that drives the rest of the
software development life cycle itself.
04:14.629 --> 04:19.583
~ Well, of course, like the details are
important and each of us needs
04:19.583 --> 04:22.054
to read it and see if it makes sense to us
and whatnot.
04:22.054 --> 04:27.058
But what I take away is intent
articulation is of utmost importance
04:27.518 --> 04:31.061
and paying attention to that is something
that all of us need to do.
04:31.061 --> 04:36.345
How we go about doing it is all
essentially building the harness is about.
04:36.345 --> 04:37.455
The
04:37.076 --> 04:38.076
Yeah.
04:37.455 --> 04:41.909
harness is about getting intent out and
that's another concept which I've been
04:41.909 --> 04:42.909
toying with.
04:42.757 --> 04:44.596
which I've been calling the intent
harness.
04:44.596 --> 04:45.932
So I'm glad to see
04:45.616 --> 04:46.616
Mm.
04:46.281 --> 04:47.513
is taking shape right now.
04:48.073 --> 04:50.154
Yeah, yeah. You had your finger on the
pulse.
04:50.154 --> 04:51.886
I do remember when that came out,
04:51.886 --> 04:54.097
like and I think it's like an intent MD
file,
04:54.097 --> 04:56.458
like you say. and it's not like this was
deep inside,
04:56.458 --> 04:58.800
you know, the article. It was like in the
first few paragraphs,
04:58.800 --> 05:02.333
right? Because I only skimmed it and I saw
it in a like r and as soon as
05:02.333 --> 05:04.264
I saw this file name, I'm like I'm
thinking,
05:04.264 --> 05:07.136
okay, Hari would love this. This is
definitely up your alley.
05:07.136 --> 05:09.882
So the audience, because of course I
remember,
05:09.882 --> 05:11.203
actually I'm not sure I do.
05:11.594 --> 05:15.846
How wha how do we think about intent
versus like specification,
05:15.846 --> 05:20.459
right? Like i you know, I think of the
story format as ~ I would like
05:20.459 --> 05:24.971
so that is that towards intent or or is
that or am I missing some sort
05:24.971 --> 05:27.052
of nuance when I think about it that way?
05:27.052 --> 05:28.683
What what is meant by intent here?
05:28.683 --> 05:31.064
You you described it as the loop,
05:31.064 --> 05:34.726
you know, other kinds of things that
that's what the intent is helping
05:34.726 --> 05:38.999
you ~ really nail down, but I'm not sure
I've got the semantics right
05:38.999 --> 05:39.999
in my head yet.
05:40.815 --> 05:46.679
Got it. I think a simple one line
explainer in my head is usually when
05:46.679 --> 05:50.902
we are prompting or vibe coding our way to
getting work done with agents,
05:50.902 --> 05:54.575
we are going from a vague idea we have
directly to implementation,
05:54.575 --> 05:58.868
right? And then whatever's missing based
on the implementation,
05:58.868 --> 06:04.253
we course correct and then we iterate on
top of it to arrive at what we want to do.
06:04.253 --> 06:08.716
But the issue here is the interpretation
gap between idea,
06:08.942 --> 06:12.835
implementation the agent has to interpret
whatever has not been said
06:12.835 --> 06:15.546
by person who wants the implementation to
be built.
06:15.546 --> 06:19.554
So in my head intent is the ability to
reduce that
06:19.654 --> 06:23.977
gap between your idea and the
implementation
06:23.988 --> 06:28.561
by interpretation that agent is trying to
do.
06:28.561 --> 06:32.360
So the thought process behind
intent-driven development is that
06:32.382 --> 06:36.675
As agents get more capable and models
become more powerful,
06:36.675 --> 06:40.517
they obviously are working towards a
factory like setup,
06:40.517 --> 06:44.091
right? Long running complex features could
be built and we cannot
06:44.091 --> 06:48.247
be the bottleneck and to keep clarifying
what's going on and you know,
06:48.247 --> 06:51.343
we cannot have this ping pong relationship
going on between us and the agent.
06:51.343 --> 06:53.015
Agent needs to independently deliver,
06:53.015 --> 06:54.156
but also stay
06:53.845 --> 06:54.845
Mm-hmm.
06:54.717 --> 06:58.249
aligned with what we want. So which means
we need a process.
06:58.611 --> 07:00.292
which will elicit intent from us,
07:00.292 --> 07:01.593
make that clear.
07:00.979 --> 07:01.979
Yeah.
07:02.073 --> 07:06.816
And where it becomes clear is an artifact
such as specifications.
07:06.816 --> 07:08.737
So the authoring of specification,
07:07.168 --> 07:08.168
I see, I see.
07:08.737 --> 07:13.140
for example, is a conversation between the
human and the agent,
07:13.140 --> 07:14.140
thereby
07:13.706 --> 07:14.706
Yeah.
07:14.060 --> 07:19.844
bringing out those details, reducing
implicit understanding so that then
07:19.844 --> 07:21.635
the agent can go on to build the complex
stuff.
07:21.635 --> 07:24.767
So I'll stop there if that's making sense.
07:24.191 --> 07:25.302
Okay, no, that
07:25.302 --> 07:28.493
helps. That helps. so one thing I'm
getting from this is don't worry,
07:28.493 --> 07:31.845
you don't have to like sit down and know
your full intent perfectly ahead of time.
07:31.845 --> 07:37.311
it is still an iterative process and in in
the the intent.md is is where
07:37.311 --> 07:40.105
we work on that together with with the
agent that we're working with.
07:40.105 --> 07:41.667
okay, so that's one thing I'm getting from
this.
07:41.667 --> 07:44.539
Another thing I'm getting from this is
that it's it's sort of maybe
07:44.539 --> 07:46.100
a bigger picture than the user story,
07:46.100 --> 07:47.221
right? As
07:47.936 --> 07:51.168
You know, as a person who at Google.com,
07:51.168 --> 07:57.441
I would like to enter ~ the name of the
product that I'm looking for so that
07:57.441 --> 07:59.973
I can find out if my favorite shop carries
it.
07:59.973 --> 08:04.024
That's a great user story, but it doesn't
do much in terms of the intent of like,
08:04.024 --> 08:06.176
well, our intent is to be the best
e-commerce,
08:06.176 --> 08:10.859
search in the world. it seems like there's
more of the directional ~
08:10.859 --> 08:12.330
and not just the specification.
08:12.686 --> 08:14.370
Or it's a nice combination of the two,
08:14.370 --> 08:15.933
but I just made that up on the spot.
08:15.933 --> 08:18.028
So am I getting closer now or am I still
missing it?
08:19.407 --> 08:26.283
You are getting closer in the sense that
intent ~ by its nature is implicit.
08:26.283 --> 08:29.365
I know what I want, but ~ I don't know
whether,
08:29.365 --> 08:31.839
let's say understands what I want.
08:31.839 --> 08:34.329
And there's that conversation between us.
08:34.329 --> 08:37.426
I think it's a throwback to
behavior-driven development,
08:37.426 --> 08:39.938
if you wish. The whole idea of BDD,
08:38.758 --> 08:40.845
Hm. Yeah.
08:39.938 --> 08:40.938
if I'm not wrong,
08:41.990 --> 08:46.053
is about ~ pushing that conversation
between people rather than...
08:46.383 --> 08:48.084
I think, okay, so taking one step back,
08:48.084 --> 08:49.315
right? I think the
08:49.241 --> 08:50.241
Okay.
08:49.315 --> 08:50.315
wipe coding
08:50.115 --> 08:55.138
or prompting approach to talking to an
agent is very instructional to say,
08:54.947 --> 08:57.420
Yeah. Yeah.
08:55.138 --> 08:57.680
like, you know, I want you to do this
versus ~
08:57.680 --> 08:59.080
if you get into a conversation,
08:59.080 --> 09:01.542
then if the agent asks you, is this what
you meant?
09:01.542 --> 09:03.953
And then you clarify ~ that
09:02.488 --> 09:03.883
Yeah. Yeah.
09:03.953 --> 09:07.766
could be pretty much just the intent
elicitation itself.
09:07.766 --> 09:09.206
You don't technically
09:08.494 --> 09:09.494
Got
09:09.206 --> 09:13.449
need user stories, but user stories give a
format which
09:13.881 --> 09:16.750
makes it a lot more easier. But is that
the only format?
09:15.992 --> 09:19.595
Yeah. No no, yeah. Sure.
09:16.750 --> 09:19.158
Maybe not. You have lot more going on
there.
09:19.915 --> 09:22.674
Yeah, okay. Okay, so I'm I'm forgetting
the important part.
09:22.674 --> 09:26.870
A user story is ~ a reason to have a
conversation between a software developer
09:26.870 --> 09:28.201
and an analyst. ~
09:27.192 --> 09:28.192
Exactly.
09:28.282 --> 09:30.293
and so that I'm forgetting the important
part.
09:30.293 --> 09:34.957
So the intent MD is a representation of
that elicitation of the intent wherein
09:34.957 --> 09:36.998
we arrive on are we talking about the same
thing?
09:36.998 --> 09:37.998
Okay,
09:37.401 --> 09:41.164
the best of what I understood from the
Anthropic playbook yet,
09:41.164 --> 09:44.206
I don't think I would say I have fully
grok what they have,
09:44.206 --> 09:48.169
you know, mentioned there, but they
09:46.363 --> 09:47.363
Right, right.
09:48.429 --> 09:51.391
clearly have an intent.md and they also
have a spec.md.
09:51.391 --> 09:52.592
So I'm still trying
09:52.167 --> 09:53.167
~
09:52.592 --> 09:55.394
to discern what's the difference between
how they are interpreting it.
09:55.180 --> 09:56.249
yeah. Right.
09:56.314 --> 10:01.448
So far, what I'm trying to say is intent
as I understand it.
10:01.884 --> 10:08.136
and intent as like eventually we'll all
converge I guess but the idea is
10:06.230 --> 10:08.265
Yeah, yeah. Yeah.
10:08.397 --> 10:09.560
to make things explicit.
10:10.466 --> 10:13.551
So one other article that's been coming to
mind while we've been talking
10:13.711 --> 10:18.286
is earlier this week, our former colleague
Martin published something from Giles
10:18.306 --> 10:21.107
Edwards Alexander, ~ another former
thought worker,
10:20.716 --> 10:21.716
Mm-hmm.
10:21.339 --> 10:22.988
~ I think he's still with Thoughtworks,
10:22.988 --> 10:26.189
talking about this this concept of the
accidental blackboard.
10:26.189 --> 10:29.169
he's talking about is like having a single
document which becomes
10:29.169 --> 10:32.720
a shared repository for many agents,
10:32.720 --> 10:36.531
including people, right? ~ that might want
to use it as a reference.
10:36.531 --> 10:40.802
And the idea of like many scientists or
engineers in one room around
10:40.802 --> 10:43.453
a single blackboard and we're all sort of
using it to share ideas.
10:44.548 --> 10:47.260
And yeah, that's what's coming to mind
when when you talk about this.
10:47.260 --> 10:49.261
The intent MD file becomes that
blackboard,
10:49.261 --> 10:52.373
you know, that other agents can read from
and the agent and I that
10:52.373 --> 10:55.024
are discussing the intent can use describe
and so on.
10:55.024 --> 11:00.146
Yes, and I think I could hazard a guess as
to how that makes sense.
11:00.146 --> 11:03.168
And this point is that agents,
11:03.168 --> 11:06.069
just like people, if you're
anthropomorphizing them,
11:06.069 --> 11:08.610
need to communicate. And
11:08.510 --> 11:09.510
Yeah. Mm-hmm.
11:08.610 --> 11:10.041
~ essentially,
11:10.041 --> 11:12.182
the way I've been using tools like Linear,
11:12.182 --> 11:18.295
or Asana, Jira, not more like it's
essentially the coordination between them.
11:18.971 --> 11:21.602
~ So we need to communicate, we need to
say,
11:21.602 --> 11:25.154
hey, Hari is done with this, so Kyle needs
to pick up and run with the next task.
11:25.154 --> 11:27.416
So we need that kind of concept to,
11:27.416 --> 11:32.208
you know, play off of and maybe I myself
want to put something out of there
11:32.208 --> 11:34.320
on the blackboard, which I might otherwise
forget.
11:34.320 --> 11:36.141
I come back to it and pick it up.
11:34.942 --> 11:36.987
Right. Yep.
11:36.141 --> 11:37.141
So anything
11:37.061 --> 11:40.763
that humans have done as a way to think
through,
11:40.763 --> 11:43.104
I think that applies very much to agent.
11:43.104 --> 11:45.846
So far, my hypothesis at least,
11:45.846 --> 11:46.846
and it's been holding.
11:47.389 --> 11:50.411
Yeah, yeah, yeah, yeah. I I tried to use
Notion that way for a little while
11:50.651 --> 11:52.582
and it worked somewhat pretty well.
11:52.582 --> 11:56.114
I I was trying to use Google Docs before
and that didn't work very well at all.
11:56.114 --> 11:57.905
We ended up like creating new documents
all the time.
11:57.905 --> 12:02.018
~ at least with Notion we could sort of
update document pieces a little
12:02.018 --> 12:03.419
bit at a time. I haven't tried linear
that.
12:03.419 --> 12:05.170
That's a new neat idea there. Cool.
12:05.280 --> 12:10.253
Another thing that I'm reflecting on is
that often the best conversations that
I've
12:10.253 --> 12:13.574
had on the show, I will find out later
someone was writing a book.
12:13.574 --> 12:19.478
And I'm wondering if just that process of
going and drafting is is you know,
12:19.478 --> 12:21.619
people talk about how writing is thinking,
12:21.619 --> 12:24.620
right? And so some of the most thoughtful
conversations I've had with people,
12:24.620 --> 12:26.221
the one like you and I had before,
12:26.221 --> 12:29.428
I'll later find that there was a book
coming and ~ and it's it
12:29.725 --> 12:32.126
Yeah, there's there's a bit of a
premonition there for me now.
12:32.126 --> 12:33.745
Interesting. Hm.
12:34.200 --> 12:35.761
Yeah, glad you mentioned that.
12:35.761 --> 12:40.573
think the whole idea of writing the book I
started six months ago ~
12:40.573 --> 12:42.453
was more about like, you know,
12:42.453 --> 12:44.414
just thinking through the process myself,
12:44.414 --> 12:46.155
like what are we really talking about?
12:46.155 --> 12:49.476
And essentially in this area of intent
driven development,
12:49.476 --> 12:51.857
there's very fragmented knowledge out
there.
12:51.857 --> 12:55.929
And I myself wanted something which I
could refer back to to say,
12:55.929 --> 12:58.550
hey, this is what I mean. And this is what
I want to index back
12:58.550 --> 13:00.881
on for every concept. ~
13:01.037 --> 13:04.757
so that I am consistent about how I prompt
my agents and whatnot.
13:04.757 --> 13:07.737
that's how it started. So I guess you're
13:07.363 --> 13:08.363
Yeah.
13:07.737 --> 13:12.377
absolutely right. know, thinking through
happens with the writing.
13:12.377 --> 13:13.917
It's a very,
13:13.454 --> 13:14.454
Yeah.
13:13.917 --> 13:16.417
how do you say, a recursive
13:15.831 --> 13:16.831
Yeah.
13:16.417 --> 13:17.417
process.
13:17.262 --> 13:20.804
Yeah. And I think there there's it's
almost in defense of the book as well,
13:20.804 --> 13:23.065
you know, or or at least of of long form.
13:23.065 --> 13:25.787
~ I think there there's something you
know,
13:25.787 --> 13:28.328
I I could probably rip out a page about
most things.
13:28.328 --> 13:31.500
but if you if you ask me to write two
hundred pages about something,
13:31.500 --> 13:32.871
I'm really gonna have to push myself,
13:32.871 --> 13:35.492
right? And I'm gonna have to think deeply
and I'm gonna check my research
13:35.492 --> 13:37.273
and I'm part my do research as well.
13:37.273 --> 13:42.836
So yeah, I think there's something to that
that volume or that size that that that
13:42.836 --> 13:43.836
is also a forcing function.
13:44.131 --> 13:47.102
Cool. So other things have changed since
we last spoke.
13:47.102 --> 13:51.034
~ a lot of guests and and myself are
thinking about like,
13:51.034 --> 13:54.865
hey, we we were living in the era of token
maxing there for a while where
13:54.865 --> 13:57.306
we were all rewarded by how many tokens
did we spend.
13:57.306 --> 14:00.257
~ I don't remember actually getting a
bonus based on that,
14:00.257 --> 14:02.008
but but you we remember those times.
14:02.008 --> 14:07.510
That seems to have changed. ~ how how is
this impacting you in your practice?
14:07.510 --> 14:10.682
How is this impacting teams which you have
a chance to observe?
14:10.682 --> 14:11.682
Like what are you seeing?
14:12.833 --> 14:15.575
I think it's quite a reckoning for all of
us.
14:15.575 --> 14:19.499
~ For the longest of time we had a little
bit of a free ride I would say.
14:19.499 --> 14:23.002
I at least would not really think about
tokens as much.
14:23.002 --> 14:26.895
But I think the first conversation now
that happens with most of my clients
14:26.895 --> 14:30.528
at least is that what is the perceived
budget that we're looking for?
14:30.528 --> 14:35.483
~ How does your ~ know harness everything
comes down to that right?
14:35.483 --> 14:39.163
How does your harness and your SDLC the
native SDLC
14:39.651 --> 14:44.055
~ You know justify what is the amount of
token and even to that matter
14:44.315 --> 14:48.889
the amount of unit work done per certain
unit token is how we had
14:48.889 --> 14:52.909
to start looking at it because tokens are
not made equal so to speak right because
14:53.370 --> 14:57.483
some models tend to be token hungry but
still they're very cheap so we have
14:57.483 --> 15:02.597
to sort of normalize those things so the
economics of it is fairly involved
actually
15:02.898 --> 15:06.891
and I wouldn't say I have a full handle on
it but what I'm trying to
15:06.891 --> 15:08.222
do at this point is
15:08.616 --> 15:11.568
try and expend energy carefully,
15:11.568 --> 15:15.368
so to speak. So if there is specification
process,
15:15.368 --> 15:17.891
for example, I think in my case,
15:17.891 --> 15:21.292
I do a lot of specification, I do a lot of
grant work execution
15:21.573 --> 15:25.715
and then verification. So the execution
part is what can afford to
15:25.715 --> 15:28.897
be on lighter models or cheaper models.
15:28.774 --> 15:29.774
Mm-hmm.
15:28.897 --> 15:29.897
The
15:29.727 --> 15:34.240
specification and exploration has to be
more on the frontier models.
15:34.240 --> 15:36.501
division of labor is how
15:36.979 --> 15:41.932
I've been able to get some amount of
rationale in terms of keeping
15:41.932 --> 15:43.883
the costs under control. But honestly,
15:43.883 --> 15:46.265
do I have a control on how much I will be
using?
15:46.265 --> 15:50.317
I don't yet know that because it's quite
difficult because month
15:50.317 --> 15:55.210
on month the economics of it has been
changing and how much work gets done keeps
15:54.868 --> 15:56.679
Yeah. Yep.
15:55.210 --> 15:56.871
varying. So I
15:56.871 --> 16:00.133
think it will need another quarter at
least for me to arrive at saying
16:00.133 --> 16:02.842
hey this this harness for this kind of
work.
16:03.408 --> 16:05.199
you will be able to predictably say,
16:05.199 --> 16:07.702
know, a perceived cost of 100 to 200,
16:07.702 --> 16:10.755
maybe $300 at most. you could comfortably
do it,
16:10.755 --> 16:15.446
right? the one thing that I'm guarding for
is worst case scenario,
16:15.446 --> 16:17.754
should you run out of tokens and whatnot,
16:17.754 --> 16:22.366
you should not be so dependent on the
model that I cannot do anything else,
16:22.366 --> 16:27.028
right? Like if I have to switch models and
So which is why I am heavily investing in
16:27.087 --> 16:30.633
the strategic context in terms of my
specifications and other ways
16:30.633 --> 16:35.246
of indexing that code so that even if I
don't have a full mental model,
16:35.246 --> 16:39.971
I can operate on it. Even if I switch
vendor harness model,
16:39.971 --> 16:43.415
I'm not impacted by it. I'm trying to
guard for that.
16:43.415 --> 16:44.415
That's the current
16:43.897 --> 16:44.897
Yeah.
16:44.096 --> 16:45.096
going
16:44.376 --> 16:45.376
on.
16:45.128 --> 16:49.089
Yeah. Yeah. It's interesting. And I'm I'm
working with a team that that over
16:49.089 --> 16:51.990
the last little while has been working
their way through not only
16:51.990 --> 16:56.773
the economics but also how they think
about ~ sort of information security
16:56.773 --> 17:01.953
and PII and their own ~ competitive you
know the nature of their their data,
17:01.953 --> 17:03.208
you know, and those kinds of things.
17:03.208 --> 17:07.380
Anyway, so th they sort of started with
anthropic Claude team and they're paying
17:07.440 --> 17:08.661
a certain amount per developer.
17:09.089 --> 17:11.210
So it it's not the enterprise account.
17:11.210 --> 17:13.151
As I understand it, once you hop to
enterprise,
17:13.151 --> 17:14.432
like you're paying by the token.
17:14.432 --> 17:15.833
But if you're at the team level,
17:15.833 --> 17:19.255
you're not yet doing that. And so it's
closer to like a pro max plan,
17:19.255 --> 17:22.036
which these these amounts sound expensive,
17:22.036 --> 17:24.698
you know, $200 a month and is not
uncommon.
17:24.698 --> 17:25.748
that I think that's what I'm paying,
17:25.748 --> 17:29.600
for instance, personally. but when you
start to look at how many tokens
17:29.600 --> 17:31.821
you actually used and what they normally
cost on the API,
17:31.821 --> 17:34.413
you start to realize, my God, I've been
getting away with murder.
17:34.413 --> 17:35.413
That's cheap.
17:35.537 --> 17:38.199
compared to what the the per token pricing
would be.
17:38.199 --> 17:40.450
But as you say, a token isn't a token.
17:40.450 --> 17:43.462
Like and I've I've also seen those same
folks sort of say,
17:43.462 --> 17:45.923
like, I'm here on open router and I can
see that,
17:45.923 --> 17:47.470
you know, prices for this, you know,
17:47.470 --> 17:49.755
tokens over here are better and tokens
over there are cheap.
17:49.755 --> 17:53.102
But to your point, that's not necessarily
the same thing.
17:53.102 --> 17:56.589
The models use them differently and do
different amounts of work with them.
17:56.589 --> 17:58.650
And they may be flaster, they may be
slower,
17:58.650 --> 18:02.172
they might do better work with those more
tokens or not.
18:02.642 --> 18:05.414
So yeah, it's it's a very confusing time.
18:05.414 --> 18:08.187
Another ~ experiment that I'm aware of,
18:08.187 --> 18:10.599
~ well, it's it's it's one that I'm doing,
18:10.599 --> 18:13.441
is trying to rent, you know, very large
machines,
18:13.441 --> 18:16.013
from EC2 at the moment that have,
18:16.013 --> 18:17.935
you know, sort of very large GPUs in them,
18:17.935 --> 18:19.966
you know, H one hundreds, call it four of
those.
18:19.966 --> 18:22.328
And just running your own, you know,
18:22.328 --> 18:26.792
go download Mimo from Xiaomi, go download
~ you know,
18:26.792 --> 18:29.014
whichever one of the latest Qwen or
something.
18:29.662 --> 18:32.483
And just see how far you can push the
economics that way.
18:32.483 --> 18:36.765
Now, rack rate EC2 instances are probably
not the cheapest way to get
18:36.765 --> 18:39.806
a machine with four H100s, but it is
something which with which you
18:39.806 --> 18:43.308
can do some experimentation. The part that
I haven't gotten to yet,
18:43.308 --> 18:45.489
~ and I'll I'll tell you how it goes once
I do,
18:45.489 --> 18:48.250
is how many people can you throw at a
machine like that?
18:48.250 --> 18:51.526
Right? Like if all it can do is answer one
person very well,
18:51.526 --> 18:53.272
then this is not the right economics.
18:53.272 --> 18:55.533
But if it if it can answer 100 people,
18:55.533 --> 18:57.754
and then maybe, right? So
18:58.084 --> 19:00.696
But to me that's that's the part which I
have not yet discovered.
19:00.696 --> 19:03.667
But yeah, to think that we all have to do
this kind of economic work
19:03.667 --> 19:06.609
to figure that that's th this is such
interesting times,
19:06.609 --> 19:09.261
you know. They're as you say, the
economics are a big part of it.
19:09.261 --> 19:10.261
Yeah.
19:10.924 --> 19:14.956
And eventually you might get down to the
point where you have local models right
19:14.956 --> 19:17.980
on your laptop. ~ And there are
experiments on those,
19:17.980 --> 19:20.582
right? I think the one got popular called
Colibri.
19:20.582 --> 19:25.706
The Colibri project was essentially to
take a GLM 5.2 instance and load
19:25.706 --> 19:28.758
it up on demand, like not load the entire
thing into memory,
19:28.758 --> 19:32.591
but sort of stream the model into memory
as needed.
19:32.591 --> 19:37.295
Of course it is much slower, but if you
can plan your workloads efficiently,
19:37.295 --> 19:39.427
then you could very well like,
19:39.427 --> 19:40.598
know, hand it off overnight.
19:41.010 --> 19:44.651
because it's going to be slow but it's not
bad right you're getting JLM
19:44.651 --> 19:50.083
5.2 and get that work done so I was
experimenting with that a month ago or
19:49.753 --> 19:50.753
Yeah.
19:50.083 --> 19:55.134
so but it might be leading to something is
what I'm thinking maybe
19:55.134 --> 19:59.745
you will have models which run locally or
maybe on a small tower which
19:59.745 --> 20:01.826
is sitting right on your desktop maybe not
even
20:01.380 --> 20:02.380
Right.
20:01.826 --> 20:07.448
on a cloud but might be slower but that's
where again it's important for
20:07.448 --> 20:09.058
us to get our harness
20:09.844 --> 20:14.497
right so that we don't really have to wait
for the agent to finish let
20:14.497 --> 20:17.044
it take its time but it's free so that's
okay.
20:17.016 --> 20:18.016
Yeah.
20:17.956 --> 20:21.927
I made the mistake of searching ~ eBay or
Kiji or one of these the other
20:21.927 --> 20:25.918
day for like used Mac Studios and I went
down this rabbit hole.
20:25.918 --> 20:29.589
There's all these ~ Mac Studios for a
little while they were selling them with
five
20:29.589 --> 20:31.160
hundred and twelve gigabytes of RAM,
20:31.160 --> 20:34.998
right? which is that that's usually the
size of a small SSD,
20:34.998 --> 20:36.731
right? and that's unified memory.
20:36.731 --> 20:39.002
And those things are selling for like
fifty thousand dollars.
20:39.002 --> 20:40.582
Second hand
20:39.958 --> 20:40.958
Wahaha!
20:42.816 --> 20:43.816
That's brilliant.
20:42.823 --> 20:43.823
Apple doesn't sell
20:43.643 --> 20:46.194
that version anymore because memory has
gotten expensive,
20:46.194 --> 20:47.194
but
20:46.926 --> 20:48.958
But I'm not sure that explains that kind
of pricing.
20:48.958 --> 20:51.300
Yeah. So you can tell
20:50.944 --> 20:51.944
Bye.
20:51.300 --> 20:52.772
we're all sort of looking into this.
20:52.772 --> 20:55.464
I doubt anyone is doing that thinking
like,
20:55.464 --> 20:57.446
~ I I really want the fastest,
20:57.446 --> 20:59.378
you know, logic pro performance I can get.
20:59.378 --> 21:01.850
That's that's likely someone who's trying
to do local models.
21:01.850 --> 21:02.850
Yeah.
21:03.296 --> 21:08.127
there is also this concept that a
colleague or a friend of mine was talking
about
21:08.127 --> 21:11.548
~ called Rob Winkle. He is talking about
RLMs,
21:11.548 --> 21:13.428
which is recursive language models,
21:12.316 --> 21:14.227
Mm. Yeah.
21:13.428 --> 21:14.428
which you could
21:14.309 --> 21:18.270
take like smaller models. We could
recursively keep asking the questions
21:18.270 --> 21:20.140
and you know, that reduces
21:19.478 --> 21:20.478
No.
21:20.140 --> 21:24.662
your need for very advanced model.
21:24.662 --> 21:26.132
You could just take the same question,
21:26.132 --> 21:28.973
you know, a reminiscence on top of it and
get
21:28.350 --> 21:29.350
Interesting.
21:28.973 --> 21:30.073
the solution eventually.
21:30.510 --> 21:33.698
That's another area that I'm just trying
to learn right now.
21:33.698 --> 21:36.275
So RLM is another area to look into.
21:37.321 --> 21:41.403
last time we spoke we spoke a test upon a
topic which is essentially
21:41.640 --> 21:46.395
a Spec Driven development and you could
Define the spec but then once
21:46.395 --> 21:49.236
you define the spec like what happens
thereafter like how do you know
21:49.236 --> 21:54.589
if the Application is built according to
that spec or not So I've been
experimenting
21:54.589 --> 21:58.630
a little just to see how far you could
push that and just have like
21:58.201 --> 21:59.201
Yeah.
21:58.630 --> 22:01.352
a quick thing that I could Demonstrate or
rather show you
22:00.610 --> 22:01.610
Yeah, yeah, yeah.
22:01.352 --> 22:02.352
like how it looks like
22:02.752 --> 22:05.377
Let me quickly share my screen and talk
about it.
22:06.244 --> 22:07.244
That'd be great
22:06.731 --> 22:07.731
I
22:07.224 --> 22:11.535
brilliant. So, okay, I think just to cut
to the chase and show you
22:11.535 --> 22:14.306
an example of what it looks like,
22:14.306 --> 22:17.627
~ I'll quickly just expand this,
22:17.627 --> 22:21.648
reduce this out so that you can see my
full screen.
22:21.648 --> 22:22.867
Alright, so
22:22.409 --> 22:23.409
Mm-hmm.
22:22.868 --> 22:25.289
my specification in OpenSpec format,
22:25.289 --> 22:28.790
again, OpenSpec by default doesn't really
capture it in Gherkin format,
22:28.790 --> 22:30.570
close to it, but not really Gherkin,
22:30.570 --> 22:32.001
right? What I have done
22:31.551 --> 22:32.551
Yeah.
22:32.001 --> 22:36.292
here is a ~ customized schema of ~
22:36.658 --> 22:38.949
OpenSpec, which I call Intent-Driven
Schema.
22:38.949 --> 22:44.560
And that schema changes the specification
format to contain gherkin within
22:44.560 --> 22:50.412
the markdown. And this gherkin is what
then gets translated into something that
22:50.412 --> 22:53.943
we are all familiar with, which is step
definitions with CucumberJS
22:53.943 --> 22:56.053
or whichever tool you choose. Again,
22:56.053 --> 23:00.375
I'm using JavaScript here, so it's written
the same use cases here in JavaScript.
23:00.375 --> 23:03.465
So that makes it equivalent. So what
23:03.100 --> 23:04.100
Uh-huh.
23:03.465 --> 23:05.546
happens is in this approach,
23:05.992 --> 23:08.614
I think in your radar, I also noticed you
have ATDD,
23:08.614 --> 23:10.576
right? Which is acceptance test driven
development.
23:10.576 --> 23:13.819
So this becomes the acceptance test for
the,
23:13.819 --> 23:15.960
you know, the agent to build the
application against.
23:15.960 --> 23:18.162
And so thereby like, you know,
23:18.162 --> 23:20.024
it doesn't deviate from what's in the
spec.
23:20.024 --> 23:21.875
So even though like, you know,
23:21.875 --> 23:25.558
if the spec itself is ~ like it's plain
English,
23:25.558 --> 23:27.860
so to speak, I can still read this,
23:27.860 --> 23:33.415
right? But this becomes more, executed
according to ~ an acceptance test.
23:33.661 --> 23:34.852
So that is keeping
23:34.053 --> 23:35.053
Yeah.
23:34.852 --> 23:37.453
the application alive. But that's only
23:36.848 --> 23:37.848
Mm-hmm.
23:37.453 --> 23:39.003
one part of the equation, right?
23:39.003 --> 23:42.995
Which is essentially, if you look at
what's happening,
23:42.995 --> 23:46.356
I think what we spoke about last time was
this guy,
23:46.356 --> 23:51.229
is spec first or spec anchored.
23:51.229 --> 23:53.309
I think these are the two areas that we
looked at.
23:53.309 --> 23:55.260
So spec driven development by itself,
23:55.260 --> 24:00.012
by default is spec first because you're
using specification to think through
24:00.012 --> 24:01.463
the process and... ~
24:01.651 --> 24:03.691
That itself is a good value, right?
24:03.691 --> 24:07.331
But worst case scenario, even if you are
not really following strictly,
24:07.331 --> 24:10.791
you could be anchored. We're trying to
keep spec and the implementation
24:10.791 --> 24:16.031
in close coordination. But what we really
want is more around,
24:16.031 --> 24:20.751
you know, spec as source, which is I only
start with the spec
24:20.751 --> 24:22.611
for all implementations, right?
24:22.611 --> 24:27.311
That's where BDD comes in. So wherein I
showed you how I take the spec,
24:27.311 --> 24:28.971
it into an acceptance test.
24:28.389 --> 24:29.389
Mm.
24:29.455 --> 24:30.455
and then
24:29.456 --> 24:30.456
Yeah.
24:29.905 --> 24:31.897
that drives the implementation.
24:31.897 --> 24:34.378
So thereby, you know, your code gets
built.
24:34.378 --> 24:39.102
But the problem I faced with this is
obviously the outer shell
24:39.102 --> 24:43.185
is constraining the code sufficiently well
enough in terms of behavior.
24:43.185 --> 24:45.865
But what gets built internally is largely
spaghetti.
24:45.865 --> 24:48.268
It doesn't really ~
24:47.670 --> 24:48.670
Mm.
24:48.268 --> 24:50.330
make a lot of nice sense, right?
24:50.330 --> 24:52.851
Of course, you can introduce DDD concepts
and whatnot.
24:52.851 --> 24:58.045
But what I found most useful is putting
DDD into that equation.
24:58.277 --> 25:04.231
really makes sense because BDD acts as the
macro level check on the unit of work,
25:04.231 --> 25:07.613
keeping the overall behavior aligned with
what you want to build.
25:07.613 --> 25:08.613
And
25:08.295 --> 25:09.295
right.
25:08.754 --> 25:13.457
TDD kind of makes sure that, you and I am
a mockist school of TDD,
25:13.457 --> 25:17.860
so to speak. So when you're mocking and
you're building out units,
25:17.860 --> 25:22.664
so sort of somewhere between these two ~
guidelines,
25:22.664 --> 25:26.987
so to speak, or how do I say envelope of
light within which those
25:26.595 --> 25:27.595
Yeah.
25:27.779 --> 25:32.983
the aircraft has to navigate. ~ These
units essentially
25:32.983 --> 25:35.024
are built fairly reasonably well.
25:35.024 --> 25:36.185
Of course, there's more to this,
25:36.185 --> 25:41.729
but at least at a top level, the thought
that evolved in my head is more like
25:41.869 --> 25:43.810
how do these concepts fit in? know,
25:43.810 --> 25:49.274
SDD, of course, gets you the thinking
process to first spec first to spec
anchored.
25:49.274 --> 25:52.416
BDD helps you reach a little bit of spec
as source.
25:52.416 --> 25:55.979
And of course, if you want cleaner code
and ~
25:55.596 --> 25:56.596
Yeah.
25:55.979 --> 25:56.979
nice design,
25:57.117 --> 26:01.550
That's where the testing, the test-driven
development niceties that helps.
26:01.550 --> 26:05.123
So I'm still building on this concept.
26:05.123 --> 26:08.746
It's still very early stage, but you have
to sort of live with this ugly little
26:08.746 --> 26:12.719
diagram that I have to kind of convey the
concept that.
26:11.786 --> 26:12.786
No, no,
26:13.357 --> 26:17.901
I I totally get it. I I can ~ I have often
given the advice to teams
26:18.181 --> 26:22.044
if you have some code which is a bit of a
ball of spaghetti,
26:22.044 --> 26:26.408
that ~ going in and applying ~ unit test
to it,
26:26.408 --> 26:30.482
~ posthumously, this is now this isn't you
know TDD in advance,
26:30.482 --> 26:32.433
this is this is adding unit test after the
fact,
26:32.433 --> 26:36.787
will force the spaghetti to detangle
because each time you're adding a unit
test,
26:36.787 --> 26:39.639
you're gonna have to sort of like clean up
this area in order to make
26:39.639 --> 26:40.639
it test a bowl.
26:40.894 --> 26:44.515
instead of an object that news up another
object that news up another object
26:44.515 --> 26:46.286
and you know inception all the way down,
26:46.286 --> 26:48.284
you're gonna end up putting in factories
and you're gonna end
26:48.284 --> 26:51.687
up putting in parameters and and you know
inversion of control and so on.
26:51.687 --> 26:53.017
And so that's
26:52.605 --> 26:53.605
Yeah.
26:53.017 --> 26:57.989
gonna it it's a way to do it. And and and
test driven ~ on the other side sort
26:57.989 --> 26:59.669
of forces that that happens from the
beginning.
26:59.669 --> 27:01.470
So you don't have to stop and go back and
do it.
27:01.470 --> 27:05.021
~ so yeah, to me that's I like that.
27:05.021 --> 27:06.021
The BDD
27:06.229 --> 27:08.891
kind of keeping the behavior and making
sure that the intent
27:08.891 --> 27:10.982
is captured and executed correctly,
27:10.982 --> 27:14.645
but doesn't give you like good structure
and then T D does.
27:14.645 --> 27:18.157
Yeah, I and I like yeah, that's that's I
like that.
27:18.157 --> 27:21.889
I and and and it really sort of shows you
where Spec Driven development fits.
27:21.889 --> 27:23.711
It sort of fits in, like you say,
27:23.711 --> 27:27.683
this sort of outer envelope, which is
driving these existing processes that
27:27.683 --> 27:29.164
we already know and love. Ooh,
27:29.164 --> 27:30.175
that's interesting. Nice.
27:31.599 --> 27:33.760
So taking off from what you mentioned,
27:33.760 --> 27:37.051
right? It makes sense even for in a
pre-existing code,
27:37.051 --> 27:38.842
which does not have specifications or
tests,
27:38.842 --> 27:43.584
which is the large majority of what we
have to deal with on a day-to-day basis.
27:43.584 --> 27:47.495
I actually found using characterization
tests,
27:47.495 --> 27:52.467
as you would say, like from Michael
Feather's kind of legacy code refactoring,
27:51.285 --> 27:53.283
Yeah. Mm-hmm.
27:52.467 --> 27:53.668
that concept works
27:53.868 --> 27:57.029
like a charm because then you can isolate
what area you're working on,
27:57.029 --> 27:59.304
characterize it and isolate it.
27:59.716 --> 28:04.490
in the process of isolation you also
nicely modularize it and you know quality
keeps
28:04.490 --> 28:08.492
incrementally changing again none of what
i mentioned has to be day
28:08.492 --> 28:13.246
one you could sort of eat your like you
know have your medicine and
28:13.246 --> 28:15.718
eat your veggies and eventually get better
28:16.414 --> 28:18.165
Yeah, yeah, yeah. I like it. I like it.
28:18.165 --> 28:21.506
Okay, cool. The the other thing that's
coming up for me,
28:21.506 --> 28:25.188
and this is this has kind of ~ come up a
couple of times now as we've been talking,
28:25.188 --> 28:28.969
is that ~ recently in my practice with
Claude,
28:28.969 --> 28:32.311
and I don't know if this is them improving
Claude code or if this is
28:32.631 --> 28:34.792
one of the plugins like superpowers that I
pulled in.
28:34.792 --> 28:36.773
I think it's probably a little bit of both
there.
28:36.773 --> 28:38.834
But Claude has started asking me,
28:38.834 --> 28:41.855
like, okay, we've got the spec now and
you've approved the spec,
28:41.855 --> 28:44.296
~ and I have a plan based on that spec.
28:45.342 --> 28:47.042
Are we ready to proceed? And then they'll
say,
28:47.042 --> 28:48.533
do you want to go sub-agent driven?
28:48.533 --> 28:53.105
Or since this is small, we can sort of do
it in this one context session.
28:53.105 --> 28:57.157
And my understanding is that part of what
we're doing there is that making sure that
28:57.157 --> 29:00.679
each implementing ~ agent gets to start
with a fresh context.
29:00.679 --> 29:04.190
It doesn't have to go through the entire
conversation that we use to
29:04.190 --> 29:06.761
get to that intent, just the intent
itself,
29:06.761 --> 29:09.432
and not through the entire conversation
that led to the spec,
29:09.432 --> 29:10.843
just the spec itself, and so on.
29:11.359 --> 29:13.950
~ and so thereby it can have a cleaner
context window.
29:13.950 --> 29:17.351
But I can still keep talking to this one
that remembers the conversation,
29:17.351 --> 29:20.742
which is great. ~ and I've I've started to
think,
29:20.742 --> 29:23.312
like for one thing, with your open weights
conversation,
29:23.312 --> 29:26.903
talking about using frontier models for
that sort of container conversation,
29:26.903 --> 29:32.205
I could imagine using a a very good open
weights model that's more of
29:32.205 --> 29:33.705
a sonnet level thing, I think,
29:33.705 --> 29:36.706
and less of a fable or an opus for the
actual execution,
29:36.706 --> 29:39.597
for the actual development, especially
with these guardrails in place,
29:39.597 --> 29:40.597
I think.
29:40.445 --> 29:44.308
But the other thing that that's this is
all bringing to mind is that like this
29:44.308 --> 29:45.890
is starting to very much to feel like,
29:45.890 --> 29:48.401
yeah, we're capturing the intent in this
outer container.
29:48.401 --> 29:52.465
And then, you know, when each one of these
little subagents does pick up,
29:52.465 --> 29:54.647
like, yeah, superpowers is feeding in TDD.
29:54.647 --> 29:58.650
And so that is helping me build a larger
test suite and helping me build
confidence.
29:58.650 --> 30:03.514
~ so yeah, it it's I but it seems like
this this new concept
30:02.836 --> 30:03.836
Thank you.
30:03.514 --> 30:07.898
of intent on top of it could really speed
up the process of understanding like where
30:07.898 --> 30:09.279
are we right now? Are we
30:09.798 --> 30:12.482
Are we still building a spec up or are we
shifted to building a plan or
30:12.482 --> 30:16.557
or are are we implementing ~ and really
sort of better organizing all that?
30:16.557 --> 30:17.778
So that's exciting. Yeah.
30:19.010 --> 30:20.010
I think I
30:19.580 --> 30:21.322
mentioned this concept last time,
30:21.322 --> 30:26.705
which is I let agents fight it out sort of
to figure out what's the right thing
30:26.705 --> 30:32.019
to do. So I call that skill something
called the adversarial authoring skill.
30:31.168 --> 30:33.705
Adversario. Review okay, okay.
30:32.019 --> 30:33.560
yeah, so
30:35.061 --> 30:36.722
that was a real authoring. In my case,
30:36.722 --> 30:39.924
I'm using two ~ agents, which is the
author.
30:39.924 --> 30:40.924
In this case, it's
30:41.015 --> 30:45.646
5.5 I was just experimenting and review is
GPT 5.6 Tera.
30:45.646 --> 30:47.887
So this is reviewing what the other fellow
wrote.
30:47.887 --> 30:51.308
And ~ if you look at my artifacts which
get generated,
30:51.308 --> 30:54.989
~ you will notice that, let me just pull
that up.
30:54.989 --> 30:59.930
So this is the proposal. This is one of
the artifacts that OpenSpec generates
31:00.130 --> 31:04.091
for each feature that I'm, or change that
I'm proposing,
31:04.091 --> 31:06.892
right? So the other proposed poll voting,
31:06.892 --> 31:09.713
this is a voting application. So I'm
trying to figure out what's going on.
31:10.125 --> 31:12.005
Now what happens is these two agents,
31:12.005 --> 31:14.505
like one agent writes it, the other one
reviews it.
31:14.505 --> 31:16.925
And then they have like, the document is,
31:16.925 --> 31:19.765
this is the proposal and this is the
council notes.
31:19.765 --> 31:24.605
So the council notes are essentially the
two agents argued as to what were
31:24.605 --> 31:27.385
the challenges raised by the reviewer and
then
31:25.994 --> 31:27.316
Wow. Yeah.
31:27.385 --> 31:29.485
the resolutions that were accepted between
them.
31:29.485 --> 31:31.605
And so basically what comes
31:30.986 --> 31:31.986
Wow.
31:31.605 --> 31:37.085
to me for review is already something that
two agents have kind of like fought
31:36.950 --> 31:37.950
Yeah.
31:37.085 --> 31:38.085
it out a
31:37.585 --> 31:38.585
little bit.
31:38.331 --> 31:39.331
so that I
31:38.416 --> 31:39.416
Yeah.
31:38.911 --> 31:45.434
can do that. And ~ this is also a way for
me to kind of ~ evaluate newer models,
31:45.434 --> 31:46.434
right? For example,
31:46.103 --> 31:47.103
Yeah.
31:46.524 --> 31:48.065
if I'm onboarding a newer model,
31:48.065 --> 31:50.146
I usually keep my reviewer constant.
31:50.146 --> 31:54.388
I onboard a newer model. So then I see if
my review notes is going bigger
31:54.388 --> 31:56.449
or smaller, what's going on there.
31:56.449 --> 31:58.729
So kind of gives me a vague sense.
31:58.729 --> 32:01.251
I can't say this is scientific and all
that,
32:01.251 --> 32:02.431
but gives
32:01.290 --> 32:02.290
No, yeah.
32:02.431 --> 32:06.042
me a pulse on how these agents are
behaving and where they
32:06.042 --> 32:07.213
are disagreeing or agreeing.
32:08.057 --> 32:09.057
So that's
32:08.364 --> 32:09.364
Yeah.
32:08.717 --> 32:13.660
one thing. And I also have like agents for
like developer and QA.
32:13.660 --> 32:14.901
So they again,
32:14.391 --> 32:15.391
Yeah.
32:14.901 --> 32:17.542
like, you know, sort of keep the guy,
32:17.542 --> 32:20.114
the QA is more responsible for the
acceptance test.
32:19.266 --> 32:21.355
Quin three eight max. Yep. Totally.
32:20.114 --> 32:21.664
The developer is more unit test.
32:21.355 --> 32:22.612
Yep.
32:21.664 --> 32:22.664
So they kind of
32:23.105 --> 32:25.597
fight it out with each other. And yeah,
32:25.597 --> 32:28.448
this, this sort of helps like since you
mentioned.
32:26.914 --> 32:29.066
Sort of a sonnet level model. That's
brilliant.
32:28.448 --> 32:30.940
Yeah. So the sub
32:29.066 --> 32:30.857
Yeah. Yeah.
32:30.940 --> 32:33.191
agent is what makes sense, right?
32:33.191 --> 32:37.073
And I think I was testing out QN 3.8 Max
and kind of made sense to
32:37.283 --> 32:38.621
verify it
32:38.783 --> 32:41.745
so similar to the adversarial authoring,
32:41.745 --> 32:45.589
we also have like the senior developer and
senior QA.
32:45.589 --> 32:46.790
So these are like,
32:46.379 --> 32:47.379
Yeah.
32:47.231 --> 32:51.094
you know, opposing roles and the QA is
responsible for acceptance,
32:51.094 --> 32:52.245
the developer writes the tests,
32:52.245 --> 32:54.358
the unit tests in the code. And in this
case,
32:54.358 --> 32:59.852
I was testing out QN3.8 max against the QA
~ and I could assign a different model
32:59.852 --> 33:00.852
to it. So.
33:00.993 --> 33:03.755
just like what you mentioned, which is we
could use sub-agents to do
33:03.755 --> 33:08.638
the work and also structure the harness
such that you have a little
33:08.638 --> 33:13.691
bit more control in terms of how much
context window we use,
33:13.691 --> 33:15.232
right? It's really, really critical.
33:15.232 --> 33:16.232
So instead of
33:15.946 --> 33:16.946
Yeah.
33:16.393 --> 33:19.515
trying to use up the context window to its
extent,
33:19.515 --> 33:23.738
~ sub-agents are a great way. And I think
since you are using superpowers,
33:23.738 --> 33:28.000
it does a pretty decent job of like
handing off and bringing back
33:28.641 --> 33:30.292
the important aspects and...
33:30.775 --> 33:34.771
and not have to overload your context into
a dumb zone,
33:34.771 --> 33:35.771
so to speak.
33:36.083 --> 33:40.306
Yeah, for sure. For sure. a lot obviously
of of the challenge is like
33:40.306 --> 33:42.158
how do you keep this so that we're in the
smart mode.
33:42.158 --> 33:46.681
Yes. So a couple of things are coming to
mind ~ with this.
33:46.681 --> 33:52.205
One, ~ your your technique of like looking
at the outputs ~ is is
33:52.205 --> 33:56.498
is definitely so one of the thing when I
started out on my project to rent large
33:56.498 --> 34:00.591
EC two machines, I tried to cheap out and
go with this ~ this
34:00.812 --> 34:02.904
This set of chips called the L40S.
34:02.904 --> 34:04.065
This is out of Lovelace.
34:03.549 --> 34:04.549
Mm-hmm.
34:04.845 --> 34:08.258
and I thought to myself, ~ those are just
like two generations back.
34:08.258 --> 34:10.280
These should be fine. But I misunderstood.
34:10.280 --> 34:12.041
They they were two generations back,
34:12.041 --> 34:16.775
but they were also one of the chipsets
that NVIDIA was aiming at like lighter
weight
34:16.775 --> 34:20.438
work. ~ I think the memory bandwidth is
less,
34:20.438 --> 34:23.520
I think is what's going on. Like they
still have plenty of memory,
34:23.520 --> 34:24.731
but the the bandwidth is less,
34:24.731 --> 34:26.863
and maybe there's less processing power,
34:26.863 --> 34:29.005
something. I'm I'm sure I'm getting the
details wrong.
34:29.336 --> 34:30.336
But point being,
34:29.575 --> 34:30.575
Mm-hmm.
34:29.946 --> 34:32.929
the L40S were not really meant for what I
was doing with them.
34:32.929 --> 34:36.612
And it worked. You know, I was able to get
open code to talk to it.
34:36.612 --> 34:40.255
And I was able to, you know, get it to go
through I had a sample code base that
34:40.255 --> 34:43.097
was an open source project I'd worked on a
year and a half ago.
34:43.097 --> 34:45.179
And I fed that in there and said,
34:45.179 --> 34:47.891
like, go analyze this code and tell me
what's wrong with it.
34:47.891 --> 34:51.364
And it I mean it did, but it wasn't good.
34:51.364 --> 34:53.106
~ you know, the results. It
34:53.429 --> 34:58.323
I I did the same code base and the exact
same prompt in Opus and it came back
34:58.323 --> 35:02.076
in about four minutes and it had like
really specific advice,
35:02.076 --> 35:05.359
you know, not just like, hey, you might
have a SQL injection here,
35:05.359 --> 35:08.021
but it's in this line and it's also in
these files,
35:08.021 --> 35:10.253
and here's how you might go about fixing
it.
35:10.253 --> 35:14.346
~ but then Quen36, which is a very well
thought-of model,
35:14.346 --> 35:18.189
you know, but but running on this hardware
that was not well thought of.
35:18.790 --> 35:20.821
And it took seven ish minutes.
35:20.821 --> 35:23.543
So okay, seven versus four. It's not
terrible.
35:23.543 --> 35:25.784
But the output was not nearly as smart.
35:25.784 --> 35:28.886
It was like, yes, there's SQL injection
and you need to fix it in that file.
35:28.886 --> 35:31.868
And that's it. There was no like pointer
to which line of code.
35:31.868 --> 35:34.089
There was no explanation of how it got it
right or wrong.
35:34.089 --> 35:39.903
And, you know, and and and there was so
it's it's just that the quality wasn't
35:39.903 --> 35:44.405
as good. Which ~ this mirrors something
Birgit talked about ~ in one
35:44.405 --> 35:45.676
of her articles recently that
35:46.262 --> 35:47.473
you know, the exact same model,
35:47.473 --> 35:50.794
but on different hardware will produce
better or worse results.
35:50.794 --> 35:53.513
So I think we're we're used to thinking
about this as like,
35:53.513 --> 35:54.766
how many billion parameters? Well,
35:54.766 --> 35:56.957
that one's gonna be good. And sure,
35:56.957 --> 36:01.079
maybe, ~ but just like we talked about how
tokens are not really fungible,
36:01.079 --> 36:02.819
I'm not sure those parameters are
fungible.
36:02.819 --> 36:06.486
But the other thing is just running that
on different hardware,
36:06.486 --> 36:07.522
you're gonna get different results.
36:07.522 --> 36:10.103
Like bigger hardware for some models,
36:10.103 --> 36:11.523
it's not just gonna be faster,
36:11.523 --> 36:12.714
it's gonna be smarter.
36:12.915 --> 36:15.236
And what it says back, which is really
kind of amazing.
36:15.236 --> 36:17.397
Again, it's the economics of all of this
in a way,
36:17.397 --> 36:20.658
you know, ~ the is is becomes one of the
interesting questions.
36:20.658 --> 36:23.299
But that reminds me of one more thing she
said.
36:23.299 --> 36:27.600
When she was on the show, she cautioned me
against T D for agents.
36:27.600 --> 36:30.562
She said she had done some research.
36:30.562 --> 36:32.338
She had done now, if I remember right,
36:32.338 --> 36:36.124
it was about three runs of maybe four
different scenarios.
36:36.124 --> 36:39.685
So, you know, our our our our size here is
small
36:39.685 --> 36:43.488
But when she looked through the results of
these like 12 different sessions
36:43.688 --> 36:47.711
in this three by four grid, she found that
when she asked,
36:47.711 --> 36:49.893
I think it was it was an anthropic model,
36:49.893 --> 36:54.017
which one I don't remember, but when she
asked it to use TDD in advance,
36:54.017 --> 36:55.538
it actually produced results
36:55.538 --> 36:58.300
that were on average worse, right?
36:58.300 --> 37:02.564
versus when she did not, the the results
were on average better.
37:02.564 --> 37:05.757
Now I asked her, are you using like is
this part of like
37:05.876 --> 37:09.218
The superpowers version, or is this just
like some other prompt that she wrote?
37:09.218 --> 37:12.239
And it wasn't. So she's talked about how
she wants to go back and sort
37:12.239 --> 37:15.931
of like rerun this against like more
battle-proven prompts,
37:15.931 --> 37:18.432
I guess, or or approaches at least,
37:18.432 --> 37:22.154
~ to to how you might have an agent to TDD
But yeah,
37:22.154 --> 37:27.147
her her early caution was we might be
anthropomorphizing them a little
37:27.147 --> 37:28.918
bit too much when we ask them to TDD.
37:28.918 --> 37:30.158
It may not actually be better.
37:30.158 --> 37:34.531
~ but like you, I'm finding that it at
least gives me confidence to
37:34.531 --> 37:35.531
see the test suite growing.
37:35.713 --> 37:40.987
~ and especially when I have it go run
like a mutation ~ check against those
tests
37:40.987 --> 37:44.479
to tell me like how good were they and it
goes and finds mutants and kills them
37:44.479 --> 37:46.741
and approves them, it gives me confidence,
37:46.741 --> 37:49.373
you know. ~ but I haven't done what she's
done.
37:49.373 --> 37:52.905
I haven't actually gone in and like run
the test to find out.
37:52.905 --> 37:56.728
So yeah, it's it's it's an interesting
sort of is it a debate?
37:56.728 --> 37:58.169
Is it a finding? I I don't know.
37:58.169 --> 38:01.551
But it's an interesting little ~ nugget
that I came across that I can't kind
38:01.551 --> 38:02.822
of can't let go of. Yeah.
38:03.827 --> 38:07.055
Got it. I think my experience with test
development has been...
38:07.561 --> 38:12.143
~ all over the place, right? ~ The same
models have done a very good job when
38:12.143 --> 38:16.414
I just asked them to do TDD, the same
model on a different day and probably
based
38:16.414 --> 38:20.706
on it's, you know, how angry it is or how
hard the processor is running,
38:20.706 --> 38:21.727
it gives
38:21.373 --> 38:22.373
Uh-huh.
38:21.727 --> 38:25.808
me a different result. But what I've
noticed is TDD,
38:25.808 --> 38:27.619
it's tried to game through that,
38:27.619 --> 38:33.561
right? ~ It's trying to finish off as soon
as possible and not really realizing that
38:33.561 --> 38:35.612
TDD is important to get the result right.
38:36.040 --> 38:37.101
~ So that's
38:36.627 --> 38:37.627
Mm.
38:37.101 --> 38:43.246
where I kind of use like constant ways of
asking it for evidence,
38:43.246 --> 38:46.689
right? So for example, the simplest of
hacks it does
38:46.153 --> 38:47.153
Yeah.
38:46.689 --> 38:49.911
is it does a vertical slice. It writes all
the tests first and then
38:49.911 --> 38:53.394
it writes the code later. Obviously the
code is going to be ~ not
38:53.394 --> 38:55.876
so great because the design is not being
driven by the test.
38:55.876 --> 38:58.868
When you go, ~ sorry, it does a horizontal
slice.
38:58.868 --> 39:00.980
I it does all the tests at once.
39:00.980 --> 39:03.722
Instead we want vertical slices of smaller
units.
39:03.722 --> 39:05.644
But how do you get vertical slices?
39:06.144 --> 39:08.646
for it to be done without it cheating,
39:08.646 --> 39:11.338
right? So then I had to make it like,
39:11.338 --> 39:14.230
you you have to give me commits for each
vertical slice.
39:14.230 --> 39:16.151
And unless you give me commits,
39:15.613 --> 39:16.613
Hmm.
39:16.151 --> 39:19.514
another agent will reject that particular
PR itself.
39:19.514 --> 39:22.496
So ~ that forced it into vertical slices.
39:22.496 --> 39:23.847
Then when we
39:23.659 --> 39:24.659
huh.
39:23.847 --> 39:25.438
got into writing tests, of course,
39:25.438 --> 39:29.161
it's not like ideally if it wrote the
tests and only wrote enough code
39:29.501 --> 39:32.213
to get that test passing, this should be
39:31.773 --> 39:32.773
Yeah.
39:32.213 --> 39:33.584
100 % branch coverage.
39:34.012 --> 39:36.253
There is no reason for it to have no
branch coverage,
39:34.577 --> 39:37.145
Yeah. Yeah.
39:36.253 --> 39:37.253
right? But it still
39:37.223 --> 39:40.574
does. So obviously, I need to have like a
branch coverage
39:39.706 --> 39:40.706
Mm.
39:40.894 --> 39:43.645
to verify what's going on and keep it at
that.
39:43.645 --> 39:45.555
Now, even if branch coverage is right,
39:45.555 --> 39:47.566
it doesn't mean the assertions are clear,
39:47.566 --> 39:49.317
like what you mentioned, the mutation
39:48.592 --> 39:49.592
Right, right.
39:49.317 --> 39:50.317
testing
39:49.777 --> 39:53.148
is there. So then the next level of
tightening needs to happen to say,
39:53.148 --> 39:56.479
hey, if I run mutations, how much
percentage of mutants survive?
39:56.479 --> 40:00.870
That is the next level. So what I did is I
took
40:01.209 --> 40:04.941
Any test development skill you take like
Matt Pocock or superpowers
40:04.941 --> 40:07.973
or anything or for that matter your own
skill.
40:07.973 --> 40:12.957
There's a certain amount of tuning you
need to do for your models and your code
base
40:13.177 --> 40:15.919
for you to get a little bit of consistent
results over time.
40:15.919 --> 40:20.832
So what I usually do is I use like a
Claude dynamic workflow or some sort
40:20.832 --> 40:24.066
of a Ralph loop wherein I put in a rubric
of this measurements.
40:24.066 --> 40:26.315
I put in the base problem statement,
40:26.315 --> 40:30.428
which is the substrate. And then I put in
the skill and I put it through
40:30.640 --> 40:32.902
like a few iterations to see where they
are missing.
40:32.902 --> 40:38.458
And then I have of rules which I can use
and pick one at a time,
40:38.458 --> 40:43.301
keep adding it, keep removing them and
then I come out with a skill file which
40:43.301 --> 40:47.874
I then use for that project. So ~ long
story short,
40:47.874 --> 40:51.957
tuning the skill is necessary for your
code base,
40:51.213 --> 40:54.173
Yeah. Yeah. Mm.
40:51.957 --> 40:54.058
your team and your
40:54.199 --> 40:57.701
models that you're using gives you a
little bit more consistent results than
40:58.117 --> 41:02.180
Just generally saying follow TDD or
generally taking a TDD skill
41:01.155 --> 41:02.155
Right, right.
41:02.180 --> 41:04.942
off the market and seeing what's going on.
41:04.942 --> 41:07.715
~ That's my experience so far,
41:06.437 --> 41:08.297
that's interesting. Yeah.
41:07.715 --> 41:08.715
yeah.
41:08.978 --> 41:10.858
Yeah, yeah, yeah. It's so in gosh.
41:10.858 --> 41:14.728
Th this whole notion of skills and how it
continues to be useful.
41:14.728 --> 41:16.539
When when I first saw them, I'm like,
41:16.539 --> 41:20.500
I mean, cool, I guess, but but that's
that's not gonna work.
41:20.500 --> 41:25.142
We're just gonna type English into a file
and but but it honestly it's kept going.
41:25.142 --> 41:27.502
Like it's kept proving value. And and to
your point,
41:27.502 --> 41:30.223
the the tuning of them is an important
thing.
41:30.223 --> 41:33.144
I think some of that can be just sort of
about about the code base itself,
41:33.144 --> 41:34.624
about the other expected practices.
41:35.224 --> 41:39.287
To your point, I s I just sort of let ~
superpowers push w in whatever it's doing,
41:39.287 --> 41:41.259
but sometimes that means it'll start
suggesting,
41:41.259 --> 41:42.590
well, you haven't done a PR. Well,
41:42.590 --> 41:46.973
I don't do PRs. I'm a solo coder on and I
j I always merge to main So I don't
41:46.973 --> 41:51.737
do I I merge to main frequently and and I
don't so so stop telling me to do PRs.
41:51.737 --> 41:55.190
But so yeah, I mean to your point these
things all need adaptation.
41:55.190 --> 41:57.952
But I think you're saying something which
is deeper and more nuanced than that,
41:57.952 --> 41:59.844
not just like, this isn't my process.
42:00.258 --> 42:04.238
But that literally tuning the skill can
make it more effective on your code base.
42:04.238 --> 42:07.507
And it though the same skill might have
run perfectly well on somebody else's.
42:07.507 --> 42:08.991
So that's interesting. Yeah.
42:07.730 --> 42:09.071
Yeah, and it's pretty much the
42:09.071 --> 42:10.912
same for us humans also, right?
42:10.912 --> 42:16.114
I could be a very good engineer as far as
comes to extreme programming
42:16.114 --> 42:18.135
and test development, but it takes me a
while
42:17.697 --> 42:18.697
Yeah.
42:18.135 --> 42:21.187
for me to tune myself into the code base
when I start
42:20.780 --> 42:21.780
Yeah.
42:21.187 --> 42:25.589
a new project. And that tuning or rather
that onboarding experience is what
42:25.589 --> 42:28.651
I think may be missing for agents
42:28.192 --> 42:29.192
Yeah.
42:28.651 --> 42:31.802
and skills. So that's what I'm
experimenting again.
42:31.802 --> 42:34.404
I don't want to say like I have a handle
on it,
42:34.404 --> 42:37.125
but this has helped is what I would.
42:37.498 --> 42:38.498
say
42:38.773 --> 42:41.706
I love it. Hari, you're you're you're
staying a step ahead of at least me,
42:41.706 --> 42:46.110
~ and and possibly some of our other our
our other audience members as well.
42:46.110 --> 42:48.352
So yeah, I I love I love how you're doing
that.
42:48.352 --> 42:49.352
Nice work.
42:50.095 --> 42:51.273
So Hari, another thing
42:51.273 --> 42:54.767
that's coming to mind for me, ~ in one of
my recent interviews,
42:54.767 --> 43:00.418
~ this is with Ian Johnson. He is a
software developer in Austin,
43:00.418 --> 43:03.530
Texas, and he's currently working on a a
book called Harness Engineering,
43:03.530 --> 43:07.182
~ which I'm sure it is one of many with
that title right now.
43:07.182 --> 43:09.733
But but I really liked him. He seems
sharp.
43:09.733 --> 43:12.864
And and I think he's the first person to
come on the show and ~
43:13.184 --> 43:18.005
a at least explain explicitly to me that
they have as part of their process,
43:18.005 --> 43:19.767
as part of their team in, you know,
43:19.767 --> 43:24.197
in a startup, they have built s something
that would resemble a factory,
43:24.197 --> 43:27.511
right? And you and I talked about dark
factories and we talked about like,
43:24.871 --> 43:27.295
Mm-hmm. Mm-hmm.
43:27.732 --> 43:29.291
are we ready? Is this coming later?
43:29.291 --> 43:33.114
And these kinds of questions. And I think
that's still something of a question.
43:33.114 --> 43:38.267
But let me explain what he's done as I
understand it and and and bounce
43:38.267 --> 43:41.759
it off of you. ~ as I say other guests may
have done it and I just
43:41.931 --> 43:43.392
They didn't call it out so explicitly,
43:43.392 --> 43:46.134
but he called it out explicitly as we are
doing a factory.
43:46.134 --> 43:50.196
And and what they do, ~ they have a
software developer assign
43:50.196 --> 43:54.259
the work to themselves. ~ so they're
they're not like assigning it to Claude
43:54.279 --> 43:56.160
or to Agent Alpha or something.
43:56.160 --> 43:58.442
They're they're I'll I'll assign it to
Kyle.
43:58.442 --> 44:01.704
From there, I'll then somehow,
44:01.704 --> 44:03.575
~ and I don't remember exactly how,
44:03.575 --> 44:05.546
but do some sort of process. I'll say,
44:05.546 --> 44:08.859
agent, go work on this. Do I mark it with
a tag or something?
44:08.859 --> 44:10.420
And the next cron job finds it,
44:10.420 --> 44:11.420
who knows?
44:11.179 --> 44:13.820
But an agent is gonna pick that up and
it's gonna do the work.
44:13.820 --> 44:15.361
It's gonna pull in the story from Jira,
44:15.361 --> 44:16.751
it's gonna pull in the code that it needs,
44:16.751 --> 44:19.922
it's gonna make the change that it needs
and write a PR,
44:19.922 --> 44:23.484
or maybe it'll give me some sort of
preliminary step before a PR,
44:23.484 --> 44:27.565
~ where I get to do some review before we
actually get to like,
44:27.565 --> 44:30.306
okay, let's put it into into into main by
a PR.
44:30.306 --> 44:32.747
But the agent basically does the work.
44:32.747 --> 44:34.308
Now, I'm still assigned though.
44:34.308 --> 44:38.409
And what I think what I like about the way
he's describing this is it's like
44:38.409 --> 44:40.830
I get to look at this as a software
developer and be like,
44:41.268 --> 44:43.649
Library upgrade for a library that,
44:43.649 --> 44:46.521
you know, I feel pretty confident is going
to be an easy one.
44:46.521 --> 44:47.882
You know, throw that to the agent.
44:47.882 --> 44:51.314
You know, bug fix for like something
that's probably going to be
44:51.314 --> 44:52.945
a fairly simple change, you know,
44:52.945 --> 44:55.116
throw that one to the agent. you know,
44:55.116 --> 44:59.489
a new user interface change, but it's one
that we've really well thought through
44:59.549 --> 45:02.520
and this agent this area of the user
interface is really clean,
45:02.520 --> 45:05.032
throw that to the agent. New feature.
45:05.032 --> 45:06.853
Maybe I'll do this one interactively.
45:07.025 --> 45:10.899
You know, and and we we sort of get to
leverage our own decision process around
like
45:10.899 --> 45:14.131
what to throw to the agent and what to
work on interactively,
45:14.131 --> 45:18.525
just like you would today. Like sit down
with w your favorite agent of choice,
45:18.525 --> 45:22.759
whether that's, you know, Qwen three eight
or or Mimo or GLT or whi whichever
45:22.759 --> 45:25.832
GLM or or whichever one, ~ you know,
45:25.832 --> 45:28.975
and work interactively on it. And I think
I think I liked that.
45:28.975 --> 45:31.077
I think I liked that it's still assigned
to a human.
45:31.077 --> 45:32.077
~
45:32.140 --> 45:34.441
But I can imagine greater parallelism this
way,
45:34.441 --> 45:36.612
right? I can imagine greater work getting
done,
45:36.612 --> 45:40.004
but there's still sort of a who is the
responsible developer to it.
45:40.004 --> 45:44.456
So this may be me naively describing
something that like everyone's already
doing,
45:44.456 --> 45:47.658
~ or it may be me describing, you know,
45:47.658 --> 45:50.950
something that's that's unique or at least
specific to this team.
45:50.950 --> 45:54.011
But that's how I see it. I I hadn't heard
a description like that
45:54.011 --> 45:57.673
of other teams which have decided on how a
factory should be.
45:57.673 --> 46:01.205
This doesn't particularly seem like a dark
factory to me,
46:01.205 --> 46:02.205
but
46:01.525 --> 46:04.496
Maybe, kind of. Like it doesn't have the
overnight quality,
46:04.496 --> 46:06.908
but I guess it is. But I wanted your take.
46:06.908 --> 46:09.919
Like are have you seen others doing things
like this?
46:09.919 --> 46:14.822
~ what do you think of this as what does
this fit your definition of factory?
46:14.822 --> 46:17.183
Like what do you think of what you've
heard in the description so far?
46:18.325 --> 46:19.325
So. ~
46:19.601 --> 46:23.923
I think the way I look at it is the
factory is able
46:23.923 --> 46:30.386
to indenting what you want out of the
factory and then you place
46:30.386 --> 46:32.927
an order and something gets manufactured
and out of the other side.
46:32.927 --> 46:34.608
Something like
46:33.633 --> 46:34.633
Yeah.
46:35.028 --> 46:40.661
that. Everything internally is automated
so to speak.
46:40.661 --> 46:41.661
Of course
46:40.662 --> 46:41.662
Yeah.
46:41.341 --> 46:46.854
this QC and if you want to be at the end
of the line as the last person stamping
46:46.854 --> 46:48.775
your seal of approval before it goes out,
46:48.775 --> 46:49.775
that's
46:50.015 --> 46:54.778
But everything in between obviously will
be in various stages of automation.
46:54.778 --> 46:58.461
And of course when a new factory is being
commissioned,
46:58.461 --> 47:01.633
we're not going to have everything run off
by themselves.
47:01.633 --> 47:04.865
We're probably going to have humans and
robots work side by side
47:04.865 --> 47:10.079
and then slowly wean off on the humans and
eventually trust the process enough
47:10.079 --> 47:14.912
so that it of runs itself. I don't think
it's very different from
47:15.913 --> 47:18.414
our continuous delivery so to speak.
47:19.081 --> 47:20.081
Yeah.
47:19.415 --> 47:22.777
to think about it that way. ~ Almost every
commit goes to production,
47:22.777 --> 47:26.980
right? Like, and the kind of rigor that
requires is a lot more than what people
47:26.980 --> 47:30.562
would give credit for. ~ Same thing goes
for the factories also.
47:30.562 --> 47:34.585
So the small microcosm of that is how I
would say my website works.
47:34.585 --> 47:38.268
Like I don't actively look at the code
that happens at all.
47:38.268 --> 47:42.070
I keep pushing tasks of what needs to
happen.
47:42.070 --> 47:44.282
Let's say my book launched, right?
47:44.282 --> 47:48.765
So I need to put a section on the website
saying the book is live and go
47:48.765 --> 47:49.765
buy it and watch
47:49.052 --> 47:50.052
Yeah.
47:50.029 --> 47:53.611
So I just push a task into my linear and
leave it there.
47:53.611 --> 47:56.343
Sometime whenever the agent comes on,
47:56.343 --> 47:58.445
it will go pick it up, will do the job,
47:58.445 --> 48:03.419
it will send me a PR with the design and
I'll approve it and send it off.
48:03.419 --> 48:08.263
Then at some time around it will build it
and it will host a temporary version
48:08.263 --> 48:11.485
of it and say is this okay? I'll say okay
and then it does it,
48:11.485 --> 48:15.628
right? And I think it also does a periodic
check of the SEO.
48:15.628 --> 48:18.581
Like I think we figured out a few issues
around that.
48:17.500 --> 48:18.865
Yeah. Yeah, yeah.
48:18.581 --> 48:19.581
So it does
48:19.011 --> 48:20.011
that.
48:19.191 --> 48:20.732
But it keeps coming back to me,
48:20.732 --> 48:25.508
it's running and essentially my primary
interface to writing code at least
48:25.508 --> 48:28.651
for my pet projects has become my mobile,
48:28.651 --> 48:32.465
right? It just keeps pinging me and I keep
responding to it.
48:33.192 --> 48:34.192
Wow.
48:33.233 --> 48:34.233
And that's
48:33.703 --> 48:37.276
pretty much all. And the coordination is
happening through a task board
48:37.276 --> 48:41.829
or something like that. I don't know
whether I can call it like my micro
factory,
48:41.829 --> 48:45.342
so to speak, but ~ that's how I'm thinking
about it.
48:45.342 --> 48:46.553
And in due
48:45.992 --> 48:46.992
Interesting.
48:46.553 --> 48:50.035
course, I hope what will happen is as
teams,
48:50.035 --> 48:53.828
like let's say we are a team and we are
ideating and we are creating,
48:53.828 --> 48:58.021
pushing it out into the board and there
are agents which go keep building
48:58.021 --> 49:00.223
it and like, you know, come back with.
49:00.725 --> 49:05.659
~ reasonable amount of accuracy so that
you don't have you're not burdened
49:05.659 --> 49:11.154
by review and ~ you can push it out with
fair bit of confidence so that's that's
49:11.154 --> 49:14.788
the part the last part is the most
important thing how confident I would push
49:14.788 --> 49:17.160
it out without looking at it right it's
again
49:15.219 --> 49:17.040
Yeah. Yeah.
49:17.160 --> 49:21.864
very parallel to how CD is right in
continuous delivery do you have the
49:22.943 --> 49:26.529
How do you say? Do you believe your system
enough to say every commit goes
49:26.529 --> 49:28.602
to production without ever looking at it?
49:27.900 --> 49:29.531
Yeah. Yeah.
49:28.602 --> 49:29.884
That's the similar
49:29.884 --> 49:33.800
question I would ask in terms of where is
your confidence you're building
49:33.092 --> 49:34.963
This factory situation. Yeah.
49:33.800 --> 49:35.062
in this factory
49:35.062 --> 49:36.062
situation.
49:36.183 --> 49:37.684
Yeah, yeah, yeah, yeah. I like that.
49:37.684 --> 49:40.365
I like that. Something I didn't get to ask
Ian,
49:40.365 --> 49:42.086
but I get to ask you now because you
mentioned it.
49:42.086 --> 49:43.527
I like how you talked about like,
49:43.527 --> 49:45.457
give me a preview so that I can see it.
49:45.457 --> 49:49.819
And that reminded me of a project that I'm
doing where it's essentially
49:49.819 --> 49:53.501
a static website with you know Cloudfare ~
pages,
49:53.501 --> 49:56.672
workers, I forget which one. ~ and you
know,
49:56.672 --> 49:58.317
it that automatically builds you a
preview,
49:58.317 --> 50:00.174
so it's very easy to go and look at the
preview.
50:00.508 --> 50:02.609
But then I thought, well, with this other
project I'm doing,
50:02.609 --> 50:05.162
I don't know how easy it would be to see
this preview.
50:05.162 --> 50:09.975
Like I had a second pre-production
environment there for a little while.
50:09.975 --> 50:11.887
And I could have said, like, go ship it to
that,
50:11.887 --> 50:14.589
you know, pre-prod, and I could have
looked at it there maybe.
50:14.589 --> 50:18.052
But I don't have built out the tech to do
an ephemeral,
50:18.052 --> 50:20.944
you know, ~ instance of my application.
50:20.944 --> 50:23.646
~ and and unfortunately, you know,
50:23.646 --> 50:26.699
there there are parts of it that have
grown just complicated enough that that
50:26.699 --> 50:28.500
wouldn't be a simple automation to do.
50:28.500 --> 50:29.541
It would be possible.
50:29.957 --> 50:32.609
But it I'd guess it's a couple of days,
50:32.609 --> 50:35.862
maybe a week of sitting down with Claude
to get to the point where I could have
that
50:35.862 --> 50:38.133
in fem ephemeral, you know, preview,
50:38.133 --> 50:39.915
which maybe I oughta do. But again,
50:39.915 --> 50:45.330
it's just me, so I don't ~ but how do you
think about getting this like let's
50:45.330 --> 50:49.613
say this were an enterprise environment
and not just Kyle's little static website
50:49.653 --> 50:52.736
or Kyle's somewhat clunky larger
application.
50:52.736 --> 50:54.607
What if this is enterprise, you know,
50:54.607 --> 50:55.948
and you've got like multiple teams,
50:55.948 --> 50:56.948
multiple domains?
50:57.189 --> 51:00.041
You know, you've got this team over here
and they're working on this back
51:00.041 --> 51:03.765
of house thing, which is critical to
billing and critical to analytics
51:03.765 --> 51:06.547
and critical to like revenue recognition
and you know,
51:06.547 --> 51:07.838
because they're publicly traded.
51:07.838 --> 51:09.299
And then you've got this other part over
here,
51:09.299 --> 51:14.694
which is more focused on e-commerce and
sales and it what are we even previewing,
51:14.694 --> 51:17.616
right? Like and we we sort of change when
we change the architecture
51:17.616 --> 51:19.618
of the the e-commerce area, for instance,
51:19.618 --> 51:22.681
and in stop depending on this third party
service and introduce this
51:22.681 --> 51:24.282
new first party version of the same thing.
51:25.326 --> 51:30.368
I can only imagine that keeping that like
give me a preview running could
51:30.368 --> 51:33.689
be pretty complicated. Now I don't think
you're saying Kyle will have
51:33.930 --> 51:36.911
the factory will be aware of all of this
at once.
51:36.911 --> 51:39.092
Or maybe you are. I d maybe eventually.
51:39.092 --> 51:42.333
But but at least on like on a team by team
sort of subcomponent
51:42.333 --> 51:45.645
by subcomponent level, how do you how are
we gonna keep it all straight,
51:45.645 --> 51:48.876
I guess? You know, like because each team
is gonna have the same challenge
51:48.936 --> 51:51.397
of how do I make my thing deployable
enough?
51:51.397 --> 51:53.832
But I have my dependencies. Are they
deployable enough?
51:54.058 --> 51:56.280
to have this like safe preview
environment,
51:56.280 --> 52:03.096
it it that's really pushing a fair amount
of like rigor,
52:03.096 --> 52:06.568
engineering discipline on teams in order
to be able to maintain that.
52:06.568 --> 52:11.353
But it could be powerful. How how do you
react to where I'm going?
52:11.353 --> 52:13.094
Am I overstating it, understating it,
52:13.094 --> 52:14.756
or or do you think I'm I'm right on?
52:14.486 --> 52:15.608
It's the most,
52:14.756 --> 52:15.756
What do you think?
52:16.249 --> 52:17.951
I think it's a very relevant question.
52:17.951 --> 52:20.294
That's the hardest question also to
answer,
52:20.294 --> 52:23.097
right? ~ Some of the teams that I'm
working with,
52:23.097 --> 52:26.732
~ we're trying to push the ~ idea of like,
52:27.106 --> 52:28.417
the way teams are organized now,
52:28.417 --> 52:30.398
right? Like, you know, each team owns
certain things,
52:30.398 --> 52:32.108
maybe it's a result of Conway's law,
52:32.108 --> 52:34.059
or the way you put it, and, you know,
52:34.059 --> 52:36.801
they're organized in a certain way because
they ship certain things,
52:36.801 --> 52:38.992
right? But when you're looking at a
factory,
52:38.992 --> 52:43.474
I honestly don't know if it should
continue to be organized that way,
52:43.474 --> 52:44.594
because
52:44.029 --> 52:45.029
Right.
52:44.774 --> 52:50.177
the modularity of what gets shipped versus
like who's sitting in,
52:50.177 --> 52:51.948
okay, let's take this analogy,
52:51.948 --> 52:53.478
right? You're talking about a factory.
52:53.478 --> 52:56.710
If you have the shop floor, you also have
the executive rooms,
52:56.710 --> 52:57.710
right?
52:57.320 --> 53:00.262
Now obviously if agents are running the
shop floor,
53:00.262 --> 53:03.244
humans are not there. Humans have to be in
the executive room.
53:03.244 --> 53:05.065
The humans in the executive room,
53:05.065 --> 53:08.097
they should not ideally be in that same
bounded boxes,
53:08.097 --> 53:11.450
right? They are operating at a certain
level of abstraction,
53:11.450 --> 53:15.712
which could be both good and bad and I
don't think we fully figured that out yet.
53:15.712 --> 53:16.783
Now what's
53:16.412 --> 53:17.412
Yeah.
53:16.783 --> 53:18.825
going to happen is in terms of reviews,
53:18.825 --> 53:21.516
at least what I can tell you, what does a
preview look like
53:21.757 --> 53:26.259
for a enterprise situation is what are
your domain driven boundaries?
53:26.259 --> 53:27.259
~
53:27.010 --> 53:30.693
feature going and you're going to do an
analysis of that you probably will
53:30.693 --> 53:33.945
get a C4 diagram of what's the change
like.
53:33.945 --> 53:36.037
Are you approving of the C4 diagram?
53:36.037 --> 53:39.805
And then if you approve of the C4 diagram
then the
53:40.067 --> 53:43.278
task list and what not that gets generated
out of it.
53:43.278 --> 53:45.209
How closely does that resemble that C4?
53:45.209 --> 53:47.710
How can you keep those things aligned?
53:47.710 --> 53:48.771
Then that comes as
53:48.485 --> 53:49.485
Yeah.
53:48.771 --> 53:50.061
a preview and then it goes in.
53:50.061 --> 53:54.901
And nothing takes away from the fact that
you still have your regression suite which
53:54.901 --> 53:59.813
is deterministic, which you cannot hand
off to saying agent will verify this
53:59.813 --> 54:01.414
manually, right? Like it's as
54:01.012 --> 54:02.012
Yeah.
54:01.414 --> 54:05.155
bad as saying, I'll just depend on manual
testing to get a job done.
54:04.467 --> 54:06.982
Yeah. Possibly worse. Yep.
54:05.155 --> 54:06.676
That's not gonna happen. So
54:07.296 --> 54:09.257
you still have all
54:09.337 --> 54:13.022
your elements of software engineering and
which is why I strongly believe that
54:14.703 --> 54:16.655
the more we get into this factory
business,
54:16.655 --> 54:19.438
there's a lot more work to do in terms of
software
54:19.060 --> 54:20.060
Yeah.
54:19.438 --> 54:23.331
engineering. We're just exposing every
ugly part of our,
54:23.331 --> 54:24.562
you know, SDLC
54:24.147 --> 54:25.147
Yeah.
54:24.562 --> 54:27.255
that we have ignored and have been lazy
about.
54:27.255 --> 54:32.279
Now you cannot be, right? So ~ it's
painful process to say,
54:32.279 --> 54:37.494
I want a factory. ~ And that means you
automatically need top of
54:37.494 --> 54:38.705
the line software engineering.
54:38.705 --> 54:40.587
And... ~
54:40.859 --> 54:41.859
That means
54:40.915 --> 54:41.915
Yeah.
54:41.579 --> 54:45.162
also reorganizing your team structure as
more
54:43.548 --> 54:44.738
Yeah, yeah.
54:45.162 --> 54:49.745
around the entire product rather than
saying I am billing,
54:49.745 --> 54:51.646
I am shipping, I am fulfillment.
54:52.392 --> 54:53.392
Yeah.
54:53.307 --> 54:55.088
Maybe you're an expert in that area.
54:55.088 --> 54:59.951
You probably will still encode those
skills and the knowledge,
54:59.951 --> 55:03.923
the wikis for that. But ~ are you going to
be isolated in that team
55:03.923 --> 55:05.835
and responsible for shipping that unit?
55:06.615 --> 55:09.116
maybe yes, you still probably need like
what you said,
55:09.116 --> 55:14.969
body and ~ you may be that seal of
approval that HITL is still required
because
55:14.969 --> 55:15.969
someone has to
55:15.186 --> 55:16.186
Yeah.
55:15.520 --> 55:17.801
be taking responsibility for that part of
the harness.
55:17.801 --> 55:18.841
But the bigger
55:18.451 --> 55:19.451
Yeah.
55:18.841 --> 55:20.562
question here is again, sorry,
55:20.562 --> 55:25.204
if I'm rambling, bigger question here is
when you're building the factory,
55:25.204 --> 55:27.746
the factory itself becomes a deliverable,
55:27.746 --> 55:29.407
right? And there has to be a.
55:29.168 --> 55:30.254
Yeah, right,
55:30.411 --> 55:31.411
set of engineers
55:30.537 --> 55:31.537
right.
55:31.212 --> 55:35.728
who work on that and that's the harder
problem and I think that itself would
55:35.728 --> 55:39.022
be a couple of hours of discussion as to
what it is.
55:38.704 --> 55:39.704
Yeah.
55:39.745 --> 55:42.026
Yeah, you're right. Yeah, I'm realizing as
you speak,
55:42.026 --> 55:46.249
~ that right, when when we think about man
manufacturing is not a product.
55:46.249 --> 55:50.352
Factories are not a product. You it is not
an easy thing to go and well,
55:50.352 --> 55:52.714
you I mean you can buy a a building shell
for a factory,
55:52.714 --> 55:56.467
but that does not give you an up and
running assembly line that's functional,
55:56.467 --> 56:00.780
right? That or or or any other kind of
manufacturing arrangement that is
functional.
56:00.780 --> 56:05.033
It's just a shell. Yeah, and the creation
of a factory itself is
56:05.033 --> 56:06.584
a lot of engineering to do it.
56:06.900 --> 56:09.832
~ so right. So conversely this this idea
that,
56:09.832 --> 56:13.403
don't worry, the robots are gonna do all
the work is gonna require a whole
56:13.403 --> 56:16.385
lot of software engineering to get right
in the first place.
56:16.385 --> 56:17.385
And
56:16.709 --> 56:17.709
Correct.
56:17.226 --> 56:18.776
maybe agents will help us with that.
56:18.776 --> 56:21.328
Certainly they will. But ~ yeah,
56:21.328 --> 56:24.639
almost certainly they will. But th it's
still gonna require some some brains
56:24.639 --> 56:26.110
to get it done. Ooh, this is interesting.
56:26.110 --> 56:29.062
One final thought, and then I need to be
respectful of your time.
56:29.062 --> 56:34.735
~ and and ~ but I'm thinking back about a
team
56:35.429 --> 56:39.952
That was suddenly gonna get helped out by
~ a team of contract engineers,
56:39.952 --> 56:41.563
I think in Eastern Europe in this case,
56:41.563 --> 56:45.996
~ and it was like, ~ look at what we're
doing for you,
56:45.996 --> 56:47.087
we're giving you all this help.
56:47.087 --> 56:48.989
You know, you you said you didn't have
enough people,
56:48.989 --> 56:51.020
we got you this team of four people
suddenly,
56:51.020 --> 56:54.292
and but then as part of the the business
arrangement,
56:54.292 --> 56:58.675
~ somebody in the engineering leadership ~
on the the customer side
56:58.675 --> 57:00.877
who I was working for at the time decided,
57:00.877 --> 57:03.559
like, yeah, we better make sure that we
check all the PRs.
57:04.249 --> 57:05.600
This new team in Eastern Europe.
57:05.600 --> 57:07.039
We've got to check all their code.
57:07.039 --> 57:09.851
It's got to be an FTE. But what that team
57:09.307 --> 57:10.307
Mm-hmm.
57:09.851 --> 57:11.922
found out pretty quickly is that suddenly
all they were doing
57:11.922 --> 57:15.343
was code review because the four people in
Poland or whatever it
57:15.343 --> 57:18.605
was were creating way more code than they
could possibly keep up with on
57:18.605 --> 57:20.685
the PR side and write their own.
57:20.685 --> 57:22.896
And suddenly they didn't like their jobs
very much.
57:22.896 --> 57:24.307
They they started to feel like,
57:24.307 --> 57:26.208
you know, somebody else is getting to
write all this code.
57:26.208 --> 57:28.058
I'm just doing PRs all day. You know,
57:28.058 --> 57:32.079
and and so that's obviously an analogy to
sort of where we are now.
57:32.504 --> 57:34.096
You know, it's a different flavored
analogy,
57:34.096 --> 57:39.444
but ~ I hope we can find ways to like our
jobs better than that team did.
57:39.444 --> 57:42.288
But I think I think we will. I don't think
it's a perfect analogy.
57:42.288 --> 57:43.650
I don't know. Yeah.
57:44.391 --> 57:49.755
Yeah, I think it's always going to be that
joy of coding versus joy of building.
57:49.755 --> 57:50.936
I don't
57:50.372 --> 57:51.372
Yeah.
57:50.936 --> 57:52.938
know if that's even the right way to put
it.
57:52.938 --> 57:54.849
But ~ coding, as I
57:54.553 --> 57:55.553
Yeah.
57:54.849 --> 57:59.663
saw, while I enjoyed the craft of putting
things together by hand,
57:59.663 --> 58:03.006
it also was limiting in terms of how much
you can get done,
58:03.006 --> 58:04.066
right? And
58:03.684 --> 58:04.684
Yeah.
58:04.066 --> 58:10.171
~ especially for smaller teams and
individuals who are building stuff out,
58:10.171 --> 58:12.833
the more it was always that I don't have
enough money to...
58:13.786 --> 58:17.750
build stuff, but I have so many of these
ideas and I would like to build it.
58:17.750 --> 58:18.832
And now
58:18.434 --> 58:19.434
Yeah.
58:19.412 --> 58:23.497
if you honestly take back, step back a
little and look at it,
58:23.497 --> 58:25.359
the joy of coding is also joy of building,
58:25.359 --> 58:27.221
right? ~ It's solving
58:26.626 --> 58:27.626
Yeah.
58:27.221 --> 58:28.703
your problem. So I don't know,
58:28.703 --> 58:31.216
but yes, certainly it's not fun to be
58:29.974 --> 58:31.145
Yeah, you're right. You're right.
58:31.216 --> 58:34.990
reviewing code. So I don't.
58:34.124 --> 58:35.124
Well, and that's the
58:35.025 --> 58:38.777
difference in my analogy. The team in
Poland had gotten to take on both
58:38.777 --> 58:40.966
the building and the coding. And the team,
58:40.966 --> 58:42.809
I think it was Ohio at that point,
58:42.809 --> 58:46.011
were only doing review. And that's why
they didn't like it.
58:46.011 --> 58:47.932
But to your point, that's not what's
happening here.
58:47.932 --> 58:52.249
We are working with the agents ~ but it
we're not just PRing and they're doing
58:52.249 --> 58:54.706
all the building. Like it is true it is a
harness,
58:54.706 --> 58:56.457
right? Like we still have the reins.
58:56.457 --> 59:00.459
So ~ we are still getting that to I
personally feel like
59:00.459 --> 59:02.362
I'm experiencing that process of building.
59:02.954 --> 59:04.808
Even if I'm not experiencing the coding
anymore.
59:04.808 --> 59:07.515
Yeah, cool. That gives me hope.
59:07.515 --> 59:09.480
Awesome.
59:08.514 --> 59:09.920
That definitely
59:10.182 --> 59:11.768
gives us all hope I suppose.
59:12.533 --> 59:13.533
Yeah, yeah, yeah.
59:13.804 --> 59:16.857
So Hari, tell us one more time before we
go ~ where they can find
59:16.857 --> 59:18.669
the book and congratulations once again.
59:18.669 --> 59:21.902
I'm I'm really excited to go and read it.
59:21.902 --> 59:24.945
But but yeah, tell tell me in the audience
where we can go find it one more time.
59:25.165 --> 59:28.686
So the book is called Spec Driven and
Development and Engineering with Intent.
59:28.686 --> 59:31.447
of course, intent is what we've been
speaking about.
59:31.447 --> 59:32.447
It's live on
59:32.070 --> 59:33.070
Yeah.
59:32.397 --> 59:38.189
MEP Manning. ~ And ~ right now three
chapters are available to you.
59:38.189 --> 59:42.810
~ soon enough we'll be putting out more
chapters.
59:42.810 --> 59:44.651
You get to read it as I write it.
59:44.651 --> 59:47.111
And of course, if you have any comments to
make,
59:47.111 --> 59:52.433
do jump in on the live book forum and you
get to shape the book as it evolves.
59:53.619 --> 59:54.619
And ~
59:53.982 --> 59:54.982
Awesome.
59:54.451 --> 59:58.317
yeah, right now you have a 50 % discount
going on because it's introductory.
59:58.317 --> 01:00:00.501
So that's where the book is at.
01:00:01.519 --> 01:00:04.508
Awesome, sir. Hari, thank you so much for
talking with us today.
01:00:04.508 --> 01:00:05.508
Appreciate you having on.
01:00:06.997 --> 01:00:08.403
As always, it's a joy talking to you,
01:00:08.403 --> 01:00:10.221
Kyle. Again, glad to be back here.
01:00:10.221 --> 01:00:11.296
Thank you for the opportunity.