UDSKRIFT 🔗
WEBVTT
00:00:05.035 --> 00:00:05.456
Thank you.
00:00:20.873 --> 00:00:22.614
Welcome to another episode of Cloud
00:00:22.655 --> 00:00:23.236
Unplugged.
00:00:23.896 --> 00:00:26.378
Today we are going to be talking about
00:00:26.617 --> 00:00:28.239
AI software factories.
00:00:29.620 --> 00:00:31.361
What even is an AI software factory?
00:00:31.821 --> 00:00:33.241
Why does anyone even want one?
00:00:35.042 --> 00:00:38.085
And what does it mean to people's jobs
00:00:38.125 --> 00:00:39.386
and how they're working today with
00:00:39.426 --> 00:00:40.026
engineering?
00:00:41.807 --> 00:00:42.328
Salman,
00:00:42.968 --> 00:00:46.170
what the F is an AI software factory?
00:00:46.450 --> 00:00:47.070
What does it mean?
00:00:47.170 --> 00:00:47.371
What's
00:00:49.436 --> 00:00:49.917
John,
00:00:49.997 --> 00:00:51.738
you've heard of the Cheesecake Factory,
00:00:52.420 --> 00:00:53.820
Charlie and the Chocolate Factory.
00:00:54.542 --> 00:00:55.963
Now it's all about the AI Software
00:00:55.982 --> 00:00:56.302
Factory.
00:00:56.343 --> 00:00:58.045
Well, basically,
00:00:58.405 --> 00:01:00.947
we've got these tools that we can use
00:01:00.987 --> 00:01:01.628
to write code.
00:01:01.987 --> 00:01:04.489
We've got Claude, we've got ChatGPT.
00:01:05.271 --> 00:01:07.893
But AI Software Factory, in my opinion,
00:01:07.972 --> 00:01:10.456
is all about what would it take to
00:01:10.536 --> 00:01:13.697
give an agent or hundreds of these agents
00:01:14.338 --> 00:01:16.361
a piece of work and have it come
00:01:17.368 --> 00:01:19.569
back with tested and working software
00:01:19.629 --> 00:01:21.390
according to your requirements.
00:01:21.510 --> 00:01:23.891
And over time, it doesn't regress.
00:01:24.070 --> 00:01:26.031
And that's basically, at a high level,
00:01:26.492 --> 00:01:28.832
the idea behind the software factory,
00:01:29.192 --> 00:01:30.832
that instead of you just using an agent
00:01:30.873 --> 00:01:32.213
on your machine and just building one
00:01:32.254 --> 00:01:32.433
thing,
00:01:33.153 --> 00:01:34.635
now you have hundreds of these agents.
00:01:34.655 --> 00:01:35.515
They could be running anywhere.
00:01:35.674 --> 00:01:36.995
They could be running Kubernetes.
00:01:37.816 --> 00:01:39.156
Sorry, I have to use the K word.
00:01:39.575 --> 00:01:40.977
They could be running anywhere you like,
00:01:41.057 --> 00:01:42.537
but that is the idea behind it.
00:01:42.557 --> 00:01:44.278
So how do I take a collection of
00:01:44.477 --> 00:01:44.938
agents
00:01:45.683 --> 00:01:48.784
or whole process that we can feed in
00:01:48.805 --> 00:01:50.525
the requirements, the building,
00:01:50.805 --> 00:01:51.986
the testing, the reviewing,
00:01:52.105 --> 00:01:53.246
and even the releasing.
00:01:53.867 --> 00:01:56.268
And how does it do it safely without
00:01:56.308 --> 00:01:58.368
the need of holding hands in various
00:01:58.409 --> 00:01:59.849
places?
00:01:59.909 --> 00:02:00.069
Right.
00:02:00.090 --> 00:02:03.230
So if I'm using agents today,
00:02:03.290 --> 00:02:04.792
do I have a software factory?
00:02:05.432 --> 00:02:07.253
If I'm getting the agents to do loads
00:02:07.272 --> 00:02:08.394
of stuff on my machine,
00:02:09.834 --> 00:02:10.713
It's writing the code,
00:02:10.753 --> 00:02:13.215
it's testing a few things,
00:02:13.314 --> 00:02:15.596
some other team members doing it as well.
00:02:17.015 --> 00:02:18.377
Is that a software factory?
00:02:19.657 --> 00:02:20.456
Well,
00:02:21.798 --> 00:02:23.457
if you're producing more code and more
00:02:23.518 --> 00:02:25.038
pull requests and things like that,
00:02:26.258 --> 00:02:27.919
it's all about the progress that you're
00:02:27.939 --> 00:02:30.319
making as an organization.
00:02:31.080 --> 00:02:32.520
So yes, you're delivering software,
00:02:32.580 --> 00:02:34.701
but you're delivering fewer defects.
00:02:35.481 --> 00:02:36.701
Are you doing less rework?
00:02:38.043 --> 00:02:38.423
And how...
00:02:39.929 --> 00:02:44.331
How can you do it with more certainty?
00:02:44.632 --> 00:02:47.052
That is the purpose behind software
00:02:47.073 --> 00:02:47.353
factory.
00:02:47.372 --> 00:02:49.274
You're just running some agents that just
00:02:49.294 --> 00:02:51.975
produces an output, in my opinion,
00:02:52.036 --> 00:02:53.736
is not what we call software factory.
00:02:53.776 --> 00:02:56.498
Do you still need the humans to make
00:02:56.519 --> 00:02:59.560
decisions for you and walk them through
00:03:00.020 --> 00:03:00.520
certain things?
00:03:01.561 --> 00:03:02.942
That's not really a software factory
00:03:02.981 --> 00:03:04.163
pattern, in my opinion.
00:03:05.043 --> 00:03:06.003
What is your opinion, John?
00:03:07.598 --> 00:03:07.798
Yeah,
00:03:07.837 --> 00:03:09.659
so I was kind of doing a load
00:03:09.699 --> 00:03:13.862
of research and reading about this as
00:03:13.923 --> 00:03:14.282
well.
00:03:15.924 --> 00:03:20.026
And from what I've gathered and from where
00:03:20.046 --> 00:03:20.828
the market is,
00:03:21.889 --> 00:03:22.128
is
00:03:24.096 --> 00:03:28.620
People are using the agents assistant in
00:03:29.121 --> 00:03:33.044
writing code, which is great.
00:03:33.104 --> 00:03:34.765
So more codes getting written.
00:03:35.747 --> 00:03:37.949
I think the challenge really is on the
00:03:38.008 --> 00:03:41.171
actual full software delivery lifecycle.
00:03:41.431 --> 00:03:45.015
So not just how many lines of code
00:03:45.036 --> 00:03:45.896
can get produced.
00:03:46.700 --> 00:03:49.822
but actually, does it meet your standards?
00:03:51.201 --> 00:03:52.842
Has it gone through a proper review?
00:03:54.182 --> 00:03:55.782
Is it aligned to best practice and
00:03:55.842 --> 00:03:57.403
architectural best practices?
00:03:58.544 --> 00:03:59.683
Other tests,
00:04:00.424 --> 00:04:02.064
are there
Load full transcript
sufficient tests even for that
00:04:02.104 --> 00:04:04.506
code with terms of like test coverage?
00:04:05.045 --> 00:04:06.086
Is there integration?
00:04:06.306 --> 00:04:07.566
Is the proper good BDD?
00:04:07.605 --> 00:04:09.206
Do you have a good smoke test?
00:04:10.447 --> 00:04:12.949
front end work efficiently and being
00:04:12.989 --> 00:04:16.571
tested against some of the back end and
00:04:17.992 --> 00:04:19.793
obviously then you kind of got bugs and
00:04:19.833 --> 00:04:21.494
like you know how do bugs get dealt
00:04:21.553 --> 00:04:23.555
with can they get autonomously picked up
00:04:24.276 --> 00:04:27.177
can they get tested and validated can it
00:04:27.218 --> 00:04:30.279
deploy can it test in a sandbox so
00:04:30.379 --> 00:04:33.862
I think the aim for the agentic software
00:04:33.901 --> 00:04:36.764
factory is it's a bit like a production
00:04:36.803 --> 00:04:38.464
line there's a handoff
00:04:39.365 --> 00:04:41.987
between each thing and each handoff is
00:04:42.286 --> 00:04:43.807
almost like an agent with a set of
00:04:43.846 --> 00:04:46.387
skills with an identity somewhere.
00:04:46.447 --> 00:04:48.908
It could have access to non production.
00:04:49.428 --> 00:04:51.848
One might have access to a sandbox.
00:04:51.889 --> 00:04:53.569
The final might have access to production.
00:04:53.589 --> 00:04:54.990
It could be just read only or I
00:04:55.009 --> 00:04:56.391
guess depending on how trusting you are,
00:04:56.891 --> 00:04:58.911
maybe to deploy and an agent could
00:04:59.110 --> 00:05:00.451
technically deploy all the way through to
00:05:00.492 --> 00:05:00.711
prod.
00:05:02.353 --> 00:05:04.314
but it's there to obviously take you
00:05:04.375 --> 00:05:06.415
through the entirety of that journey into
00:05:06.456 --> 00:05:09.238
production autonomously and then check
00:05:09.259 --> 00:05:10.399
with all the checks and balances
00:05:10.439 --> 00:05:12.822
throughout that's that's how i so you you
00:05:12.841 --> 00:05:14.663
can focus on the definitions and the
00:05:14.702 --> 00:05:17.165
approvals and more what you're trying to
00:05:17.204 --> 00:05:19.947
build and not all the intricacies of the
00:05:20.088 --> 00:05:23.209
sdlc so what you defined is
00:05:24.274 --> 00:05:26.055
really just what we currently do with
00:05:26.096 --> 00:05:29.297
people instead of doing with people we do
00:05:29.416 --> 00:05:31.516
with these agents you know you talked
00:05:31.536 --> 00:05:33.817
about like having a separate agent that
00:05:33.838 --> 00:05:35.858
does the deployment you know we have those
00:05:35.918 --> 00:05:38.218
pipelines that only pipelines have access
00:05:38.259 --> 00:05:40.519
that certain people perhaps okay if we
00:05:40.540 --> 00:05:41.560
talk about continuous delivery
00:05:41.620 --> 00:05:43.180
automatically goes in right we have
00:05:43.221 --> 00:05:44.980
different various roles you have people
00:05:45.021 --> 00:05:46.802
who do the sre stuff you have people
00:05:47.182 --> 00:05:48.642
who do the development stuff you have tech
00:05:48.682 --> 00:05:50.923
leads who might be reviewing code your
00:05:50.963 --> 00:05:52.963
business who might be providing or product
00:05:53.483 --> 00:05:56.245
Or else we might be providing the
00:05:56.266 --> 00:05:57.947
requirements for what needs to be built.
00:05:58.867 --> 00:06:00.447
What we're saying now is software factory
00:06:00.689 --> 00:06:02.790
is basically the same thing,
00:06:03.689 --> 00:06:05.552
but with agents and humans involved in
00:06:06.072 --> 00:06:07.273
certain space places.
00:06:08.293 --> 00:06:10.214
Is that a good understanding of what this
00:06:10.274 --> 00:06:12.375
software factory is?
00:06:12.396 --> 00:06:12.875
Pretty much.
00:06:13.437 --> 00:06:13.757
I think,
00:06:14.877 --> 00:06:17.079
and this is a little bit maybe more
00:06:17.098 --> 00:06:17.579
controversial.
00:06:17.598 --> 00:06:18.560
I don't know if it's controversial,
00:06:18.600 --> 00:06:20.120
but I'd argue that it's
00:06:21.271 --> 00:06:28.194
it is actually better because I think with
00:06:28.553 --> 00:06:29.713
people in teams,
00:06:30.774 --> 00:06:34.074
there's a huge dependency on specific
00:06:34.134 --> 00:06:35.315
individuals in a team.
00:06:35.355 --> 00:06:38.516
So the bus factor metric,
00:06:39.576 --> 00:06:40.916
how many people could get hit by a
00:06:40.956 --> 00:06:41.117
bus?
00:06:41.516 --> 00:06:42.317
It's a bit dark,
00:06:42.357 --> 00:06:44.398
but how many people could get hit by
00:06:44.418 --> 00:06:45.478
a bus in the team?
00:06:46.557 --> 00:06:48.298
Are certain individuals carrying that
00:06:48.338 --> 00:06:48.559
team?
00:06:49.733 --> 00:06:52.415
Whereas actually what this does is it
00:06:52.475 --> 00:06:55.178
forces the discipline of actually we need
00:06:55.199 --> 00:06:58.403
to outsource some of this quality, right?
00:06:58.583 --> 00:06:58.942
Actually,
00:06:58.963 --> 00:07:01.605
we need assurances that whoever's
00:07:01.646 --> 00:07:02.627
contributing,
00:07:03.007 --> 00:07:04.589
that actually the level of review that's
00:07:04.610 --> 00:07:04.970
happening,
00:07:04.990 --> 00:07:07.172
the level of scrutiny on that is universal
00:07:07.692 --> 00:07:09.694
and equally deterministic no matter whose
00:07:09.975 --> 00:07:10.555
code it is.
00:07:11.516 --> 00:07:12.877
And the same with the test,
00:07:12.898 --> 00:07:14.119
the same with the security,
00:07:14.778 --> 00:07:15.899
or even a pen test,
00:07:16.420 --> 00:07:17.560
which you would have to go and pay
00:07:17.600 --> 00:07:20.223
to do way later in the software delivery
00:07:20.262 --> 00:07:20.923
lifecycle.
00:07:21.403 --> 00:07:22.725
Whereas now you can have an agent,
00:07:23.444 --> 00:07:23.745
you know,
00:07:23.786 --> 00:07:26.067
trying to like pen test essentially what's
00:07:26.107 --> 00:07:28.009
being built and see what security
00:07:28.028 --> 00:07:29.350
vulnerabilities might exist.
00:07:29.930 --> 00:07:32.911
So I'd actually argue that the opportunity
00:07:33.052 --> 00:07:35.593
of bringing the quality bar up
00:07:36.374 --> 00:07:39.194
is way higher because you've now have the
00:07:39.214 --> 00:07:40.776
opportunity to really think through the
00:07:40.815 --> 00:07:42.937
quality properly as a team and start to
00:07:42.976 --> 00:07:46.057
define exactly what good looks like but
00:07:46.098 --> 00:07:49.300
articulate it down into agents and skills
00:07:49.560 --> 00:07:52.180
and practice principles and standards and
00:07:52.201 --> 00:07:53.541
all the things that people might
00:07:54.800 --> 00:07:56.661
maybe just take for granted in their heads
00:07:56.982 --> 00:07:58.642
and hope that people adhere to them based
00:07:58.682 --> 00:07:59.682
on their experience,
00:08:00.384 --> 00:08:01.783
whereas now you don't rely on the
00:08:01.803 --> 00:08:03.524
assumption that the experience is driving
00:08:03.545 --> 00:08:04.185
the quality,
00:08:04.766 --> 00:08:06.666
that actually it's a deterministic set of
00:08:06.687 --> 00:08:12.209
quality gates that are now happening.
00:08:12.288 --> 00:08:12.369
Okay.
00:08:12.408 --> 00:08:13.310
Sorry, what were you going to say?
00:08:13.350 --> 00:08:13.689
No, no,
00:08:13.769 --> 00:08:14.771
I was going to say basically what you're
00:08:14.790 --> 00:08:15.370
saying is that
00:08:16.420 --> 00:08:16.939
At the moment,
00:08:17.201 --> 00:08:18.600
the bus factor that you talked about,
00:08:18.922 --> 00:08:21.523
the one benefit of basically you're saying
00:08:21.603 --> 00:08:22.584
we should write everything down,
00:08:23.103 --> 00:08:24.384
but we need to provide the agents the
00:08:24.425 --> 00:08:26.565
context of what the practices are that
00:08:26.605 --> 00:08:27.466
need to be followed,
00:08:27.887 --> 00:08:28.968
that the agents can follow.
00:08:29.048 --> 00:08:33.289
So this thing of reliance of people goes
00:08:33.330 --> 00:08:34.130
down a little bit.
00:08:34.291 --> 00:08:35.951
Not saying that it will definitely
00:08:36.011 --> 00:08:36.751
completely go away,
00:08:37.211 --> 00:08:38.413
but it goes down a little bit because
00:08:38.432 --> 00:08:39.634
some of these things that have been
00:08:39.894 --> 00:08:40.474
written already.
00:08:40.494 --> 00:08:41.835
Is that fair to say?
00:08:42.235 --> 00:08:42.916
That's what you're saying.
00:08:43.076 --> 00:08:44.056
We need to provide the context.
00:08:45.076 --> 00:08:47.157
I think there's an interesting question in
00:08:47.177 --> 00:08:50.837
this that when you do provide these agents
00:08:50.898 --> 00:08:53.519
the context and the guardrails and the
00:08:53.538 --> 00:08:56.658
skills that they should follow,
00:08:56.700 --> 00:08:58.179
we know that at the end of the
00:08:58.379 --> 00:08:58.480
day,
00:08:58.500 --> 00:09:00.960
LLMs are still predicting the next token.
00:09:01.159 --> 00:09:03.181
They still will do what they want to
00:09:03.221 --> 00:09:06.562
do and not always follow the commands that
00:09:06.581 --> 00:09:07.341
have been given to them.
00:09:08.863 --> 00:09:10.964
how in a software factory that you make
00:09:11.004 --> 00:09:13.085
sure that now instead of you working on
00:09:13.446 --> 00:09:15.326
one feature at a time you can because
00:09:15.346 --> 00:09:17.268
you have all these agents you can work
00:09:17.307 --> 00:09:19.989
on these things at scale hundred features
00:09:20.009 --> 00:09:23.171
at the same time that things are actually
00:09:23.211 --> 00:09:25.792
being built to the standard that you want
00:09:25.831 --> 00:09:28.533
them to be built how does that scale
00:09:28.573 --> 00:09:31.956
with this software factory yeah i guess
00:09:31.975 --> 00:09:35.738
that's the the tricky part of it is
00:09:35.778 --> 00:09:36.418
like it's
00:09:37.955 --> 00:09:39.837
In its principal form, obviously,
00:09:39.898 --> 00:09:44.446
what could be achieved is great.
00:09:44.846 --> 00:09:45.347
It's huge.
00:09:45.368 --> 00:09:47.971
It means maybe if you're a team of...
00:09:49.052 --> 00:09:51.134
know eight you could probably split into
00:09:51.173 --> 00:09:54.034
two teams of four and then use a
00:09:55.095 --> 00:09:58.275
a an ai software factory sets of processes
00:09:58.956 --> 00:10:01.996
um and automation um and actually achieve
00:10:02.057 --> 00:10:03.878
maybe even more than when you're a team
00:10:03.898 --> 00:10:05.738
of eight now you can have two two
00:10:05.778 --> 00:10:10.500
teams of four and but obviously if you're
00:10:10.539 --> 00:10:14.160
gonna start to outsource to a factory like
00:10:14.181 --> 00:10:14.721
with anything
00:10:15.480 --> 00:10:17.863
the quality and the assurance on whether
00:10:17.903 --> 00:10:20.845
that is meeting the standards you'd expect
00:10:21.725 --> 00:10:22.687
is a measured thing.
00:10:22.886 --> 00:10:24.048
So suddenly you're into like,
00:10:24.107 --> 00:10:27.250
how do we even measure what good looks
00:10:27.289 --> 00:10:30.572
like and the success of those outsourced
00:10:30.592 --> 00:10:32.214
skills and outsourced agents?
00:10:34.183 --> 00:10:35.544
And obviously to do that means you've got
00:10:35.565 --> 00:10:37.206
to have quite a lot in place,
00:10:37.346 --> 00:10:39.667
definitely from a metrics perspective,
00:10:40.967 --> 00:10:45.091
but also from a token usage perspective,
00:10:45.110 --> 00:10:47.131
you need to be able to trace very
00:10:47.231 --> 00:10:51.975
easily on how many tokens per request
00:10:52.014 --> 00:10:52.576
happened,
00:10:53.135 --> 00:10:56.477
how many requests per turn or how many
00:10:56.518 --> 00:10:58.740
turns per session and kind of on it
00:10:58.779 --> 00:10:59.139
goes.
00:10:59.200 -->
00:11:00.860
And obviously you then need to obviously
00:11:00.921 --> 00:11:01.761
add all of that up.
00:11:02.832 --> 00:11:05.094
to give the overall cost on whatever
00:11:05.153 --> 00:11:07.293
actually happened throughout that chain.
00:11:07.313 --> 00:11:09.095
You need to obviously follow it through
00:11:09.815 --> 00:11:12.956
agent by agent to then know and have
00:11:12.995 --> 00:11:15.116
good, what's called eval,
00:11:15.256 --> 00:11:17.057
which is like good deterministic
00:11:17.376 --> 00:11:19.658
evaluation of the agent and the skills to
00:11:19.717 --> 00:11:22.298
say, actually, ninety percent of the time,
00:11:22.318 --> 00:11:23.658
does it come up with the same answer?
00:11:23.678 --> 00:11:24.299
And actually,
00:11:24.759 --> 00:11:26.799
what models are cheapest and can we get
00:11:26.820 --> 00:11:27.620
the cost down?
00:11:27.681 --> 00:11:29.301
And those types of things,
00:11:29.321 --> 00:11:31.302
which is you need to have quite a
00:11:31.361 --> 00:11:31.761
mature
00:11:33.557 --> 00:11:36.918
know metric and and um i guess data
00:11:36.938 --> 00:11:39.798
hygiene on all of that yeah to adopt
00:11:39.879 --> 00:11:41.279
it otherwise you could spend loads of
00:11:41.299 --> 00:11:44.000
money um but this is yeah i mean
00:11:44.080 --> 00:11:47.020
the the evaluation part uh john is nothing
00:11:47.081 --> 00:11:48.801
new right we already we've been doing this
00:11:48.860 --> 00:11:51.022
in software for a long time anyway you
00:11:51.042 --> 00:11:53.062
know you have like your code quality
00:11:53.121 --> 00:11:55.341
scanning you have your ci continuous
00:11:55.381 --> 00:11:57.602
integration tests every time you like
00:11:57.643 --> 00:11:59.182
merge code or you create like a pull
00:11:59.222 --> 00:12:02.443
request we still run this correct so that
00:12:02.464 --> 00:12:02.644
that
00:12:04.015 --> 00:12:05.096
We just need to keep following it,
00:12:05.135 --> 00:12:07.057
maybe extend it a little bit further in
00:12:07.076 --> 00:12:07.956
terms of how many,
00:12:08.476 --> 00:12:10.837
perhaps what the output of the agent was.
00:12:10.898 --> 00:12:13.859
But we can lean into existing tools and
00:12:13.899 --> 00:12:16.320
the practices that we're already using and
00:12:16.360 --> 00:12:17.700
make sure that these are followed.
00:12:18.000 --> 00:12:19.221
Is there anything new that we need to
00:12:19.240 --> 00:12:21.581
do on here, John?
00:12:21.860 --> 00:12:22.501
Kind of.
00:12:22.881 --> 00:12:23.902
I'd say not.
00:12:25.005 --> 00:12:26.527
really,
00:12:26.567 --> 00:12:29.729
because that would mean that whatever is
00:12:29.808 --> 00:12:34.552
running each step of the SDLC has got
00:12:34.572 --> 00:12:35.491
to execute
00:12:37.600 --> 00:12:39.941
and then read a load of context
00:12:39.980 --> 00:12:41.402
information per chain.
00:12:41.802 --> 00:12:43.043
So basically,
00:12:43.163 --> 00:12:44.183
if you're trying to keep your cost
00:12:44.224 --> 00:12:44.583
minimal,
00:12:45.224 --> 00:12:46.684
what you don't want to do is have
00:12:47.385 --> 00:12:49.647
loads and loads of documents being
00:12:49.687 --> 00:12:52.708
produced for each step that each agent has
00:12:52.749 --> 00:12:55.029
to go and read and then suck into
00:12:55.070 --> 00:12:58.052
its context to try and figure out actually
00:12:58.072 --> 00:12:58.711
what's going on.
00:12:59.172 --> 00:13:02.014
That's not efficient, obviously.
00:13:02.053 --> 00:13:03.414
So everything needs to be
00:13:04.880 --> 00:13:08.403
very quick in terms of almost like what
00:13:09.044 --> 00:13:10.645
they call in a kind of context graph,
00:13:11.187 --> 00:13:13.067
the relationship between each thing and
00:13:13.087 --> 00:13:15.090
then actually finding the specific bit of
00:13:15.149 --> 00:13:18.232
that thing that pertains to what it's
00:13:18.273 --> 00:13:18.913
trying to do,
00:13:19.173 --> 00:13:22.436
not go and get all the context about
00:13:22.475 --> 00:13:25.259
this kind of area and then work out
00:13:25.298 --> 00:13:27.100
what bit of this context I really only
00:13:27.159 --> 00:13:27.421
need.
00:13:28.352 --> 00:13:30.855
bin off all the other information and then
00:13:30.936 --> 00:13:32.797
actually kind of continue because the cost
00:13:32.817 --> 00:13:34.940
then per session and the cost per cycle
00:13:35.460 --> 00:13:37.101
is going to be really, really high.
00:13:38.222 --> 00:13:39.124
And then obviously it means you're going
00:13:39.144 --> 00:13:41.846
to be spending loads of money on this
00:13:41.885 --> 00:13:42.687
software factory,
00:13:42.706 --> 00:13:44.889
which then you'd probably be cheaper to
00:13:44.908 --> 00:13:46.971
hire a person or whatever in the end
00:13:46.991 --> 00:13:48.513
than it would have been to have outsourced
00:13:48.592 --> 00:13:48.712
it.
00:13:48.753 --> 00:13:50.475
So, you know,
00:13:50.715 --> 00:13:52.456
I don't think that's the right...
00:13:53.222 --> 00:13:54.705
way you should approach it but it's
00:13:54.725 --> 00:13:57.787
probably what people will do i imagine
00:13:57.847 --> 00:14:01.009
because it's you know your first way you
00:14:01.030 --> 00:14:02.351
would think about it which is obviously
00:14:02.392 --> 00:14:03.873
natural i would think about it that way
00:14:03.893 --> 00:14:05.114
as well it's like great i've got all
00:14:05.153 --> 00:14:07.316
these tools i can just plug some agents
00:14:07.336 --> 00:14:09.057
in which is true it can plug into
00:14:10.038 --> 00:14:11.220
some of the tools but we just need
00:14:11.240 --> 00:14:14.243
to be mindful that you're thinking about
00:14:14.322 --> 00:14:17.225
as tool calls that are very specific
00:14:18.458 --> 00:14:20.543
know to those external tools to get a
00:14:20.602 --> 00:14:22.865
very specific bit of information out not
00:14:23.206 --> 00:14:25.890
it's
running ci and then i'm going to
00:14:25.912 --> 00:14:29.057
spaff out the ci output into an agent
00:14:29.116 --> 00:14:29.758
if you see what i mean yeah
00:14:31.980 --> 00:14:34.321
I think the trick probably, John,
00:14:34.400 --> 00:14:36.802
is in providing the right level of
00:14:36.881 --> 00:14:37.361
context.
00:14:37.442 --> 00:14:37.621
Yes,
00:14:37.642 --> 00:14:39.503
you can restrict in what the agent can
00:14:39.562 --> 00:14:39.763
see,
00:14:40.322 --> 00:14:43.245
but sometimes we might not be able to
00:14:43.325 --> 00:14:44.945
figure out what the right relationship
00:14:44.985 --> 00:14:47.546
between the different data points are.
00:14:48.287 --> 00:14:50.006
So I think just yes,
00:14:50.047 --> 00:14:52.488
we can restrict some of the information
00:14:52.508 --> 00:14:55.009
that we can provide the context that we
00:14:55.048 --> 00:14:55.889
can provide the agents.
00:14:56.629 --> 00:14:57.711
But if we restrict it too much,
00:14:57.770 --> 00:14:59.633
the agent will not be any good either.
00:14:59.653 --> 00:15:02.956
So I think this thing about defining I
00:15:02.975 --> 00:15:04.876
know people use dependency graphs for this
00:15:04.956 --> 00:15:07.438
and graph databases to define a
00:15:07.458 --> 00:15:09.640
relationship between various components in
00:15:09.660 --> 00:15:11.101
the organization, various systems.
00:15:11.143 --> 00:15:13.884
I think Uber have a good example of
00:15:13.904 --> 00:15:14.085
that.
00:15:14.105 --> 00:15:15.986
But I think that getting that relationship
00:15:16.506 --> 00:15:16.988
is important.
00:15:17.028 --> 00:15:18.128
Basically, what we're saying is
00:15:18.889 --> 00:15:20.649
A software factory is not really magic,
00:15:20.710 --> 00:15:20.909
right?
00:15:20.950 --> 00:15:22.471
You need to have all these prerequisites
00:15:23.111 --> 00:15:24.572
in place before you can even do it,
00:15:24.592 --> 00:15:26.754
because it's not all about I think the
00:15:26.793 --> 00:15:31.157
problem that people see nowadays is that
00:15:31.177 --> 00:15:32.977
people think, like, if I just write code,
00:15:33.077 --> 00:15:34.399
if it spits out code,
00:15:36.195 --> 00:15:39.839
doing ai right but the point here is
00:15:39.899 --> 00:15:42.461
with a software factory though as inside
00:15:42.501 --> 00:15:43.743
people say if it spits our code yeah
00:15:43.903 --> 00:15:47.466
i mean the software factory and what we're
00:15:47.506 --> 00:15:49.109
seeing the patterns is the software
00:15:49.129 --> 00:15:50.809
factories the whole process
00:15:51.471 --> 00:15:53.472
of SDLC, gathering your requirements,
00:15:53.732 --> 00:15:54.552
writing the software,
00:15:55.114 --> 00:15:56.595
ensuring the quality is good,
00:15:57.154 --> 00:15:58.816
having security tests from the beginning
00:15:58.956 --> 00:16:01.238
and other tests and negative scenarios and
00:16:01.258 --> 00:16:03.600
everything and doing even the deployments
00:16:03.840 --> 00:16:07.783
and also being able to figure out when
00:16:07.822 --> 00:16:09.083
things go wrong in production.
00:16:09.104 --> 00:16:11.546
There's an alert or something falls over.
00:16:11.926 --> 00:16:13.807
But don't forget, at the moment,
00:16:14.687 --> 00:16:16.529
the biggest benefits people who work on
00:16:16.570 --> 00:16:18.230
production support is that they have
00:16:18.270 --> 00:16:19.231
context of things.
00:16:19.951 --> 00:16:20.772
When things fall over,
00:16:20.792 --> 00:16:22.352
they have an idea of where to look
00:16:22.413 --> 00:16:22.613
at, right?
00:16:23.254 --> 00:16:25.333
Because when some things falls over,
00:16:25.835 --> 00:16:26.875
you need to be able to figure out,
00:16:27.034 --> 00:16:29.436
well, first of all, what was the change?
00:16:30.116 --> 00:16:32.177
Usually errors happen because there's a
00:16:32.217 --> 00:16:32.457
change.
00:16:32.477 --> 00:16:33.398
Maybe there's a deployment.
00:16:33.437 --> 00:16:35.619
Maybe somebody updated the networking
00:16:35.658 --> 00:16:35.859
rules.
00:16:35.899 --> 00:16:36.759
Something happened like that.
00:16:37.460 --> 00:16:40.541
It's the whole process beginning to end is
00:16:40.581 --> 00:16:43.101
what a software factory is.
00:16:43.201 --> 00:16:45.903
And you have to implement the process that
00:16:45.943 --> 00:16:47.644
we're doing before and even now, right?
00:16:47.663 --> 00:16:47.945
Because...
00:16:49.004 --> 00:16:50.206
The moment it starts running,
00:16:51.589 --> 00:16:53.451
things will break, things will fail.
00:16:53.812 --> 00:16:55.455
And you've done the evals,
00:16:55.595 --> 00:16:57.138
you think you've done the evals correctly
00:16:57.577 --> 00:16:58.779
of the code that you've released.
00:16:59.861 --> 00:17:01.224
But we also have to check
00:17:02.489 --> 00:17:03.490
how the evals are written.
00:17:03.590 --> 00:17:05.270
Because if you have one agent,
00:17:05.612 --> 00:17:07.393
if the same agent is writing the evals
00:17:07.733 --> 00:17:08.693
as is writing the code,
00:17:09.515 --> 00:17:11.057
it will do whatever it can to pass
00:17:11.416 --> 00:17:12.116
the test itself.
00:17:12.198 --> 00:17:13.479
It might even get rid of the test
00:17:13.999 --> 00:17:14.819
to pass the code.
00:17:15.359 --> 00:17:17.382
So I think this is where you have
00:17:17.422 --> 00:17:21.224
to have some deterministic guardrails in
00:17:21.285 --> 00:17:21.586
place,
00:17:21.625 --> 00:17:23.948
some tests that you write or something
00:17:23.968 --> 00:17:26.430
that can follow that the agent doesn't
00:17:26.470 --> 00:17:26.730
just...
00:17:27.851 --> 00:17:29.791
make up or remove things and delete things
00:17:30.291 --> 00:17:31.633
but you know like not giving it
00:17:31.673 --> 00:17:33.173
permissions to change things that you
00:17:33.193 --> 00:17:35.675
don't want it to change so that that's
00:17:35.715 --> 00:17:39.577
what in my opinion software factory is
00:17:39.617 --> 00:17:41.179
working towards and it's a new field right
00:17:41.199 --> 00:17:42.638
john because people still trying to figure
00:17:42.699 --> 00:17:44.800
out how to do it more efficiently without
00:17:44.840 --> 00:17:47.041
wasting all these tokens and providing the
00:17:47.221 --> 00:17:49.563
right right level of context and how many
00:17:49.643 --> 00:17:51.604
agents should i have what access should i
00:17:51.644 --> 00:17:53.285
give it is it where do i have
00:17:53.325 --> 00:17:55.346
the uh responsibility between
00:17:56.207 --> 00:17:57.709
various agents do i have like a reviewer
00:17:57.788 --> 00:17:59.769
agent and a code builder agent and
00:18:00.230 --> 00:18:03.173
deployment agent and you know like an sre
00:18:03.253 --> 00:18:05.895
agent and like a business analyst agent
00:18:06.957 --> 00:18:08.377
there's a lot in the air at the
00:18:08.438 --> 00:18:10.539
moment this i i feel like i feel
00:18:10.559 --> 00:18:11.661
like there's a lot in the air it
00:18:11.721 --> 00:18:15.085
reminds me it reminds me of the cloud
00:18:15.545 --> 00:18:17.987
in some ways obviously it's different from
00:18:18.086 --> 00:18:20.670
different fundamentals but you know
00:18:22.403 --> 00:18:24.585
people click opting and getting the
00:18:24.664 --> 00:18:25.865
outcome in the cloud, right?
00:18:26.446 --> 00:18:28.127
Actually, you know,
00:18:28.188 --> 00:18:29.528
doing it as infrastructure as code,
00:18:29.548 --> 00:18:30.410
it's repeatable.
00:18:30.809 --> 00:18:32.612
All those paradigms that people would
00:18:32.791 --> 00:18:33.373
profess,
00:18:34.413 --> 00:18:35.913
the outcome was the fact that something
00:18:35.933 --> 00:18:36.834
ended up in the cloud.
00:18:37.615 --> 00:18:40.755
And so you diminish it down into not
00:18:40.815 --> 00:18:42.394
necessarily understanding the value at the
00:18:42.414 --> 00:18:43.776
very beginning because you're like, well,
00:18:44.476 --> 00:18:45.016
what does it matter?
00:18:45.036 --> 00:18:46.435
Look, we've got stuff hosted,
00:18:46.855 --> 00:18:48.896
things are there, but changing it,
00:18:49.096 --> 00:18:51.017
knowing things won't break,
00:18:51.396 --> 00:18:53.376
repeatedly deploying the same thing.
00:18:53.817 --> 00:18:56.317
All those paradigms are like ones that
00:18:58.038 --> 00:18:59.358
more mature businesses understood the
00:18:59.378 --> 00:19:01.159
value of that have been in the cloud
00:19:01.179 --> 00:19:01.538
a while.
00:19:02.465 --> 00:19:05.688
but maybe less mature businesses.
00:19:05.728 --> 00:19:06.989
It took a while for them to learn,
00:19:07.028 --> 00:19:08.449
and then DevOps kind of came around.
00:19:09.289 --> 00:19:10.250
And it was the same again with
00:19:10.951 --> 00:19:11.730
over-provisioning.
00:19:11.750 --> 00:19:14.333
I've got this massive database.
00:19:14.472 --> 00:19:15.473
It doesn't really matter.
00:19:15.493 --> 00:19:15.953
You're like, yeah,
00:19:15.973 --> 00:19:19.056
but you're using one percent of that data
00:19:19.076 --> 00:19:21.196
that you didn't need to provision such a
00:19:21.217 --> 00:19:22.217
huge database.
00:19:22.397 --> 00:19:23.978
And then how do you know what you
00:19:24.018 --> 00:19:25.378
need and what you didn't need,
00:19:25.880 --> 00:19:27.701
and whether it's over-provisioned or it
00:19:27.780 --> 00:19:28.601
isn't over-provisioned?
00:19:28.621 --> 00:19:30.082
Do you need the data to know that?
00:19:31.013 --> 00:19:32.035
um and you would have had to have
00:19:32.055 --> 00:19:35.757
thought about it so you know these things
00:19:36.797 --> 00:19:38.778
are sometimes just universal and how
00:19:38.817 --> 00:19:41.119
everybody approaches them but i think a
00:19:41.160 --> 00:19:43.461
lot of lessons you can kind of take
00:19:43.480 --> 00:19:45.642
from history which is like yeah obviously
00:19:45.721 --> 00:19:47.962
there's a cost to every service you're
00:19:47.982 --> 00:19:51.744
going to use you can overuse it and
00:19:51.765 --> 00:19:53.326
you can over rely on it and you
00:19:53.346 --> 00:19:56.387
can also use it badly right because
00:19:57.204 --> 00:19:59.086
if the goal is efficiency and
00:19:59.125 --> 00:20:01.507
effectiveness then obviously you've got to
00:20:01.547 --> 00:20:03.808
think about that and there are ways to
00:20:03.868 --> 00:20:06.470
achieve it but those ways are harder you
00:20:06.509 --> 00:20:08.391
know like you're just saying how do you
00:20:09.432 --> 00:20:11.673
test the determinism of something well you
00:20:11.692 --> 00:20:13.834
probably need to look at historical data
00:20:14.334 --> 00:20:16.055
and see whether it got to the same
00:20:16.134 --> 00:20:18.155
outcome based on that historical situation
00:20:18.195 --> 00:20:18.757
repeatedly
00:20:19.436 --> 00:20:21.740
Did it solve that PR, an historical one,
00:20:22.361 --> 00:20:24.243
based on that bug in the same way?
00:20:24.403 --> 00:20:25.685
Did it get to the same outcome,
00:20:25.726 --> 00:20:26.747
ninety percent of the time,
00:20:26.807 --> 00:20:28.950
or did it keep reinventing a different
00:20:29.050 --> 00:20:30.332
issue and change something else?
00:20:30.372 --> 00:20:30.553
Right.
00:20:30.593 --> 00:20:32.976
That means the PR
00:20:33.913 --> 00:20:34.314
you know,
00:20:34.354 --> 00:20:37.215
solution book fixing agent is obviously
00:20:37.256 --> 00:20:39.076
not very good if it keeps changing it
00:20:39.616 --> 00:20:40.718
for the same situation.
00:20:40.798 --> 00:20:42.179
So, you know,
00:20:42.439 --> 00:20:44.740
it takes time to think through a
00:20:44.779 --> 00:20:46.941
methodology and an approach and how you
00:20:47.000 --> 00:20:47.862
measure success.
00:20:47.902 --> 00:20:50.423
And that's obviously where people need to
00:20:50.442 -->
00:20:53.865
be spending the time to make it really
00:20:53.904 --> 00:20:54.525
worthwhile.
00:20:54.565 --> 00:20:55.826
And if you do like everything,
00:20:55.945 --> 00:20:58.627
it can be super high value if you
00:20:58.647 --> 00:21:00.489
can make it work like the cloud is.
00:21:02.414 --> 00:21:03.996
But you also can hear horror stories,
00:21:04.036 --> 00:21:04.676
which I'm sure,
00:21:04.817 --> 00:21:05.897
which we used to hear with the cloud.
00:21:05.958 --> 00:21:07.519
Oh my God, somebody got hacked.
00:21:07.980 --> 00:21:09.040
It was insecure.
00:21:09.480 --> 00:21:10.281
Why are we using it?
00:21:10.382 --> 00:21:11.982
I knew we shouldn't have used the cloud.
00:21:12.384 --> 00:21:13.304
You know, it's such a bad idea.
00:21:13.505 --> 00:21:14.125
Do you know what I mean?
00:21:14.144 --> 00:21:16.886
It kind of feels like the same behaviors
00:21:16.906 --> 00:21:18.449
and patterns that are about to happen.
00:21:19.730 --> 00:21:20.250
Absolutely.
00:21:20.270 --> 00:21:23.292
And I like that you mentioned the cloud
00:21:23.313 --> 00:21:25.294
adoption, what happened in the past.
00:21:25.433 --> 00:21:27.835
Don't forget after figuring out
00:21:28.214 --> 00:21:30.395
infrastructure as code and so you can have
00:21:30.435 --> 00:21:32.376
repeated patterns and you ended up with
00:21:32.416 --> 00:21:32.778
DevOps.
00:21:33.238 --> 00:21:35.138
And then there was FinOps, right?
00:21:35.219 --> 00:21:36.819
Making sure that you're not wasting.
00:21:37.240 --> 00:21:39.260
You talked about over provisioning of
00:21:39.300 --> 00:21:39.560
things.
00:21:40.060 --> 00:21:42.241
And we have all these tools and companies
00:21:42.261 --> 00:21:43.961
that came with, you know, Kubernetes.
00:21:44.382 --> 00:21:46.202
We right size your pods just so you
00:21:46.222 --> 00:21:47.803
don't, you know, waste your resources.
00:21:48.303 --> 00:21:49.762
And in EKS,
00:21:50.223 --> 00:21:52.364
Amazon coming up with autoscalers that
00:21:52.784 --> 00:21:54.243
make sure that your pods are sized
00:21:54.284 --> 00:21:55.724
correctly and put in the right nodes.
00:21:55.744 --> 00:21:57.305
You don't waste, you know,
00:21:58.005 --> 00:21:59.685
like nodes because nodes cost you money.
00:22:00.605 --> 00:22:02.847
I think a similar thing we'll see with
00:22:03.147 --> 00:22:03.989
token ops, right?
00:22:04.048 --> 00:22:06.990
So taking the right model, token ops,
00:22:07.151 --> 00:22:08.992
that's what we're going to call it from
00:22:09.053 --> 00:22:10.773
now on.
00:22:10.794 --> 00:22:12.095
Well, but actually,
00:22:12.335 --> 00:22:14.277
I guarantee it will be a thing.
00:22:15.397 --> 00:22:16.999
I mean, it will definitely be a thing,
00:22:17.038 --> 00:22:17.720
token ops, yeah.
00:22:18.217 --> 00:22:19.718
Because, you know, like, John, yes,
00:22:19.758 --> 00:22:21.318
you get billing information, right?
00:22:21.338 --> 00:22:21.778
So, yeah,
00:22:22.078 --> 00:22:25.601
you do get billing fine as a whole
00:22:25.681 --> 00:22:27.682
department or as a whole organization.
00:22:27.721 --> 00:22:30.223
But Finals is more than just that,
00:22:30.284 --> 00:22:33.085
that you can go down to an application
00:22:33.125 --> 00:22:33.465
and say,
00:22:33.526 --> 00:22:34.846
this is how much it costs to run
00:22:34.865 --> 00:22:36.887
this app because these are the resources
00:22:36.907 --> 00:22:38.367
that are being run per minute.
00:22:39.528 --> 00:22:41.769
and it's the same right because because if
00:22:41.789 --> 00:22:44.551
we're extending the agents from not just
00:22:44.592 --> 00:22:46.752
development also the agents that are
00:22:46.813 --> 00:22:48.994
running they're doing the uh the
00:22:49.255 --> 00:22:50.434
requirement gathering they're doing the
00:22:50.474 --> 00:22:51.996
research they're doing the deployments
00:22:52.036 --> 00:22:54.258
they're doing the sre all needs to be
00:22:54.298 --> 00:22:57.019
linked somehow all the tokens that they're
00:22:57.038 --> 00:22:59.361
consuming needs to be put on a bill
00:22:59.520 --> 00:23:01.701
that people realize that yeah now you know
00:23:01.741 --> 00:23:03.623
i can i can ship hundred more features
00:23:03.663 --> 00:23:05.023
a month but at what cost
00:23:06.375 --> 00:23:07.955
Yeah, exactly.
00:23:07.976 --> 00:23:08.297
Well, of course,
00:23:08.317 --> 00:23:10.936
it's the same thing as cloud builds, John.
00:23:11.317 --> 00:23:12.057
You've been in many,
00:23:12.297 --> 00:23:13.817
many organizations where people are like,
00:23:13.857 --> 00:23:16.238
oh, yeah, we did a lift and shift,
00:23:16.337 --> 00:23:18.759
and it cost way more than what we
00:23:18.778 --> 00:23:20.959
were paying on-premises because we ended
00:23:20.999 --> 00:23:21.618
up doing the same thing.
00:23:21.638 --> 00:23:23.900
And the thing is,
00:23:24.240 --> 00:23:26.259
it is a little bit dangerous in a
00:23:26.380 --> 00:23:29.240
number of things because the output is
00:23:29.300 --> 00:23:29.861
very quick.
00:23:30.000 --> 00:23:31.381
You run this agent locally,
00:23:31.421 --> 00:23:33.102
and we're like, yeah, this is beautiful.
00:23:33.241 --> 00:23:33.842
It builds it.
00:23:34.587 --> 00:23:34.867
Great.
00:23:35.387 --> 00:23:36.108
Let's start using it.
00:23:36.148 --> 00:23:36.949
Let's go all in.
00:23:37.569 --> 00:23:39.550
But have you thought about the security?
00:23:39.770 --> 00:23:41.551
Have you thought about how you're going to
00:23:41.571 --> 00:23:43.813
trace when something gets changed and you
00:23:43.853 --> 00:23:46.473
have no idea how it changed and when
00:23:46.513 --> 00:23:47.034
it broke?
00:23:47.273 --> 00:23:48.714
It's been broken in production for months,
00:23:48.734 --> 00:23:50.155
but nobody realizes because you haven't
00:23:50.215 --> 00:23:51.997
used that feature for a little while.
00:23:52.017 -->
00:23:52.997
And then now you need to try and
00:23:53.037 --> 00:23:54.498
figure out when it broke.
00:23:54.857 --> 00:23:55.759
Can you trace that back?
00:23:56.598 --> 00:23:57.659
And how much did it cost you in
00:23:57.679 --> 00:23:58.539
the end?
00:23:58.599 --> 00:23:59.700
I mean,
00:23:59.940 --> 00:24:02.201
if an agent is going to burn a
00:24:02.240 --> 00:24:04.241
thousand dollars or pounds in token every
00:24:04.281 --> 00:24:06.502
time it has to triage in a bug,
00:24:06.522 --> 00:24:10.443
I'd rather not use that or be able
00:24:10.483 --> 00:24:11.664
to know that this is how much it's
00:24:11.704 --> 00:24:12.144
costing.
00:24:12.444 --> 00:24:13.786
Just like Fenlops dashboard,
00:24:13.806 --> 00:24:15.546
you log in and you see, oh,
00:24:15.967 --> 00:24:18.207
this is my trajectory of the budget.
00:24:18.326 --> 00:24:19.667
This is my alerts that have been set
00:24:19.708 --> 00:24:19.788
up.
00:24:20.167 --> 00:24:21.368
All of that needs to happen, right?
00:24:22.449 --> 00:24:24.569
this agent for this organization or this
00:24:24.609 --> 00:24:26.769
department is causing costing this much
00:24:26.829 --> 00:24:29.371
and we think you're gonna burn all your
00:24:29.391 --> 00:24:31.872
budget in april instead of the end of
00:24:31.912 --> 00:24:34.251
the year that all of that i think
00:24:34.291 --> 00:24:37.432
is still immature john um it is but
00:24:37.452 --> 00:24:39.094
it's a choice isn't it i guess though
00:24:39.114 --> 00:24:40.294
you know like with everything it's
00:24:41.128 --> 00:24:43.412
Maturity and immaturity is a choice.
00:24:44.113 --> 00:24:46.756
You can invest time and money to mature
00:24:46.797 --> 00:24:51.342
yourself and you can try and take
00:24:51.383 --> 00:24:53.384
inspiration from others like Uber did
00:24:53.424 --> 00:24:54.346
their big talk.
00:24:56.519 --> 00:24:58.960
And obviously they spent lots of money,
00:24:59.000 --> 00:25:00.161
but then they also managed to get the
00:25:00.201 --> 00:25:02.101
cost per session down by like fifty two
00:25:02.121 --> 00:25:04.561
percent by basically optimizing.
00:25:04.582 --> 00:25:06.242
That's how they got to the context graph
00:25:06.262 --> 00:25:08.544
was to make sure actually the lookups are
00:25:08.564 --> 00:25:10.384
really efficient and effective because it
00:25:10.403 --> 00:25:11.525
was like I said,
00:25:11.545 --> 00:25:12.884
it was trying to extract so much
00:25:12.924 --> 00:25:14.425
information all the time to figure
00:25:14.445 --> 00:25:15.026
something out.
00:25:15.867 --> 00:25:17.229
when actually it could just look it up
00:25:17.269 --> 00:25:18.970
really quickly in the context graph about
00:25:19.009 --> 00:25:21.211
who owns what and where to find that
00:25:21.251 --> 00:25:21.771
data.
00:25:22.613 --> 00:25:25.355
And obviously the pull request volume is
00:25:25.394 --> 00:25:25.935
going up,
00:25:26.915 --> 00:25:30.138
but obviously the level of AI assisted
00:25:30.179 --> 00:25:32.339
pull request being merged and then tracing
00:25:32.359 --> 00:25:33.381
the number of bugs,
00:25:34.041 --> 00:25:36.303
how many of those needs evaluating?
00:25:36.343 --> 00:25:39.685
Because without an AI software factory
00:25:40.547 --> 00:25:42.407
where you're just literally writing
00:25:44.714 --> 00:25:46.537
general code, I think the bugs,
00:25:46.676 --> 00:25:48.097
there was some organization where the bugs
00:25:48.137 --> 00:25:49.479
went up something like, I don't know,
00:25:49.538 --> 00:25:51.259
fifty something percent or something
00:25:51.299 --> 00:25:54.863
insane because no one's really reviewing
00:25:54.883 --> 00:25:56.564
them because it's too much code to review.
00:25:56.644 --> 00:25:59.686
So therefore you now do need a reviewer
00:25:59.987 --> 00:26:02.709
that is more automatic that can reduce the
00:26:02.749 --> 00:26:03.970
amount you've got to review.
00:26:03.990 --> 00:26:06.672
So you have some assurance on what's going
00:26:06.771 --> 00:26:08.394
on so that actually you can carry on
00:26:08.574 --> 00:26:11.036
using the AI assistance for development.
00:26:11.755 --> 00:26:14.336
Because you can't have a bottleneck on the
00:26:14.356 --> 00:26:14.938
other side,
00:26:14.978 --> 00:26:16.439
which is a reviewing bottleneck,
00:26:17.640 --> 00:26:18.840
without trying to think about the
00:26:18.881 --> 00:26:19.922
automation on the other side.
00:26:19.942 --> 00:26:21.603
So I do think the stats speak for
00:26:21.643 --> 00:26:22.243
themselves.
00:26:24.065 --> 00:26:25.246
You could achieve a lot more,
00:26:25.626 --> 00:26:27.147
as we know.
00:26:27.228 --> 00:26:29.108
But yeah,
00:26:29.130 --> 00:26:30.951
you need to put the investment in around
00:26:30.990 --> 00:26:32.632
everything as a decision around getting
00:26:32.652 --> 00:26:33.232
the quality,
00:26:33.252 --> 00:26:34.314
which is no different from people.
00:26:35.239 --> 00:26:36.859
You can get good people, bad people.
00:26:36.921 --> 00:26:38.961
Maybe you can't be bothered to do a
00:26:39.001 --> 00:26:41.785
test in your hiring process to evaluate
00:26:41.805 --> 00:26:42.285
the person.
00:26:42.305 --> 00:26:43.385
So you just hire them anyway.
00:26:43.806 --> 00:26:45.627
And then you realise actually didn't
00:26:45.647 --> 00:26:47.308
really know what they were doing and
00:26:47.329 --> 00:26:49.771
they've put loads of bugs in or some
00:26:49.872 --> 00:26:51.212
of that.
00:26:51.252 --> 00:26:51.973
No different.
00:26:52.394 --> 00:26:54.395
You've got to evaluate people as much as
00:26:54.415 --> 00:26:55.896
you've got to evaluate the agents.
00:26:56.037 --> 00:26:57.898
And you need to have ways of mechanisms
00:26:58.259 --> 00:26:58.960
and approaches
00:26:59.559 --> 00:27:01.787
to reducing the risk, you know,
00:27:02.067 --> 00:27:03.551
and assessing that
the things that are
00:27:03.571 --> 00:27:07.142
going on actually are high quality as
00:27:07.182 --> 00:27:07.403
well.
00:27:07.462 --> 00:27:07.804
So yeah.
00:27:08.915 --> 00:27:10.557
No, I mean, that's absolutely fair, right?
00:27:10.636 --> 00:27:13.458
You're talking about having processes in
00:27:13.518 --> 00:27:13.778
place.
00:27:13.837 --> 00:27:15.679
And at the moment in software development
00:27:15.699 --> 00:27:16.339
lifecycle,
00:27:17.000 --> 00:27:19.000
those processes are when somebody writes a
00:27:19.040 --> 00:27:21.561
piece of code, you have pre-commit checks,
00:27:21.602 --> 00:27:23.563
you have unit tests, you have reviews.
00:27:25.284 --> 00:27:27.484
Some are automated, some are manual still.
00:27:27.885 --> 00:27:29.226
You have tests that run, right?
00:27:29.286 --> 00:27:32.867
So all of that needs to exist that
00:27:33.048 --> 00:27:36.029
the agents can go through and review.
00:27:37.421 --> 00:27:38.642
and just have those things in place.
00:27:38.662 --> 00:27:40.282
So as an organization, then, John,
00:27:40.343 --> 00:27:42.683
if I were, which I'm not,
00:27:42.743 --> 00:27:45.325
but if I were a CEO of a
00:27:45.384 --> 00:27:45.965
company, John,
00:27:46.665 --> 00:27:48.366
what are the top three things I need
00:27:48.386 --> 00:27:49.047
to focus on?
00:27:49.968 --> 00:27:51.628
Because there's benefits to software AI
00:27:51.709 --> 00:27:52.288
factory, right?
00:27:52.308 --> 00:27:53.289
That's what people are hearing,
00:27:53.670 --> 00:27:55.049
that you can move much faster,
00:27:55.391 --> 00:27:56.651
you can deliver more features,
00:27:57.031 --> 00:27:58.511
you can get to market quicker because
00:27:58.531 --> 00:28:00.492
that's what the software AI factory is
00:28:00.512 --> 00:28:00.973
showing you.
00:28:01.634 --> 00:28:02.835
What are the three things
00:28:03.757 --> 00:28:06.959
I should consider before I can say,
00:28:06.979 --> 00:28:07.740
you know what,
00:28:07.759 --> 00:28:08.780
I'm on the right trajectory.
00:28:09.500 --> 00:28:10.101
Because don't forget,
00:28:10.161 --> 00:28:11.362
people don't want to lose out.
00:28:11.962 --> 00:28:12.942
People don't want to lose out.
00:28:12.982 --> 00:28:15.164
And we talked about maturity is not there.
00:28:15.184 --> 00:28:16.125
But at the same time,
00:28:16.965 --> 00:28:18.527
people don't want to be behind the AI
00:28:18.567 --> 00:28:20.067
space and the race itself.
00:28:20.608 --> 00:28:22.410
So what are the three things as an
00:28:22.470 --> 00:28:25.271
organization that people in leadership
00:28:25.872 --> 00:28:29.335
should be asking people who report to them
00:28:29.654 --> 00:28:32.356
that the right processes are in place?
00:28:33.819 --> 00:28:36.300
Yeah, I don't know from a CEO level,
00:28:36.381 --> 00:28:37.582
it's a trickier one,
00:28:37.622 --> 00:28:40.564
but I think obviously you're going to need
00:28:41.824 -->
00:28:43.224
to look a little bit inward.
00:28:43.365 --> 00:28:46.967
So if you're very, very senior,
00:28:47.907 --> 00:28:49.868
if I have quite a lot of operational
00:28:49.929 --> 00:28:51.769
processes that are quite manual,
00:28:52.878 --> 00:28:55.881
if a human being can't even get access
00:28:55.901 --> 00:28:57.801
to an environment automatically to go and
00:28:57.842 --> 00:29:00.763
test something then an agent can't anyway
00:29:00.923 --> 00:29:02.506
so you've already got your answer there so
00:29:02.546 --> 00:29:05.307
it's like you you haven't automated
00:29:05.366 --> 00:29:07.909
anything anyway so therefore you're gonna
00:29:07.949 --> 00:29:09.809
have to automate that before anything can
00:29:09.849 --> 00:29:13.212
take advantage of it automatically so if
00:29:13.573 --> 00:29:15.173
you can look inwards and ask those
00:29:15.213 --> 00:29:18.296
questions then you've got work to do on
00:29:18.695 --> 00:29:20.718
you know your automation processes
00:29:20.778 --> 00:29:22.318
internally and making sure that
00:29:24.519 --> 00:29:26.320
People can get an environment quickly to
00:29:26.401 --> 00:29:28.061
test and get access to it without having
00:29:28.102 --> 00:29:28.362
to wait.
00:29:29.041 --> 00:29:31.864
I think if you fix those things and
00:29:31.884 --> 00:29:35.125
you have a good mature platform that means
00:29:35.145 --> 00:29:36.866
that things are automatically given and
00:29:36.926 --> 00:29:40.788
can be allocated to individuals and safely
00:29:42.048 --> 00:29:43.430
and things are governed and there's
00:29:43.490 --> 00:29:46.352
policies so you can trust that the
00:29:46.392 --> 00:29:47.892
policies can't be broken because they're
00:29:47.912 --> 00:29:48.472
enforced.
00:29:49.315 --> 00:29:51.836
the other one is to put something in
00:29:51.935 --> 00:29:54.696
front of the inference so in front of
00:29:54.737 --> 00:29:57.738
the model that can track every request
00:29:57.758 --> 00:29:59.699
that's kind of coming through and so that
00:29:59.739 --> 00:30:01.740
means you can then attribute cost and
00:30:01.779 --> 00:30:04.361
tokens and give meaning to every request
00:30:04.961 --> 00:30:06.622
so therefore you can kind of decorate
00:30:07.221 --> 00:30:09.103
the information and the metadata so that
00:30:09.123 --> 00:30:11.464
you can do really nice attribution,
00:30:11.505 --> 00:30:14.326
an agent, a person, a team, a project,
00:30:15.667 --> 00:30:17.989
and therefore you can start to correlate
00:30:18.028 --> 00:30:19.150
some of the value over time.
00:30:19.170 --> 00:30:20.351
That will start to give you the FinOps
00:30:20.411 --> 00:30:20.730
view.
00:30:21.211 --> 00:30:24.213
And the third thing is then the wider
00:30:24.233 --> 00:30:27.316
thing is you're comparing it against what
00:30:27.336 --> 00:30:29.037
you've got, right?
00:30:29.057 --> 00:30:33.039
When you start to adopt agentic kind of
00:30:33.079 --> 00:30:35.801
workflows and the software factory,
00:30:36.686 --> 00:30:37.988
You've got to measure whether the
00:30:38.048 --> 00:30:39.650
alternative was more efficient and
00:30:39.670 --> 00:30:42.932
effective than your current.
00:30:42.972 --> 00:30:43.613
So therefore,
00:30:43.854 --> 00:30:45.555
you need to put the measurements in place.
00:30:46.036 --> 00:30:47.596
You need to know if it's going to
00:30:49.038 --> 00:30:49.939
introduce more bugs,
00:30:49.959 --> 00:30:51.359
not just write faster code,
00:30:51.380 --> 00:30:55.243
but actually has the reliability reduced
00:30:55.763 --> 00:30:56.684
since we adopted
00:30:57.625 --> 00:31:00.027
as an ai software factory have we had
00:31:00.086 --> 00:31:02.969
more down time do we have more support
00:31:03.429 --> 00:31:05.851
issues being raised than we usually
00:31:05.891 --> 00:31:08.352
normally have and so you need to look
00:31:08.412 --> 00:31:10.932
at like all of the data points to
00:31:10.992 --> 00:31:13.515
be mindful of you know making sure that
00:31:13.535 --> 00:31:16.615
you're not introducing uh problems by
00:31:16.675 --> 00:31:18.257
going that way and actually what you
00:31:18.277 --> 00:31:19.518
should think should be trending down
00:31:20.546 --> 00:31:21.286
not trending up.
00:31:21.945 --> 00:31:24.047
So you'd hope that actually, you know,
00:31:24.086 --> 00:31:26.708
your security vulnerabilities are reducing
00:31:27.248 --> 00:31:28.689
the time it takes you to fix an
00:31:28.749 --> 00:31:30.788
incident should be coming down because
00:31:30.808 --> 00:31:32.750
you've got agents doing it, you know,
00:31:32.849 --> 00:31:34.711
so you should be looking at positive
00:31:34.750 --> 00:31:36.230
signals, not negative ones.
00:31:36.330 --> 00:31:37.751
And if it's going the wrong direction,
00:31:37.771 --> 00:31:39.771
then you, if you're doing something wrong,
00:31:39.832 --> 00:31:40.813
I would say at that point,
00:31:40.853 --> 00:31:44.193
anybody to like pause and then have a
00:31:44.233 --> 00:31:45.814
look about your approach and what you're
00:31:45.834 --> 00:31:46.054
doing.
00:31:46.494 --> 00:31:47.795
Um, that'd be my top three.
00:31:49.439 --> 00:31:53.201
So basically you're saying that we should
00:31:53.241 --> 00:31:57.765
have perhaps like a maturity grid of where
00:31:57.785 --> 00:31:59.386
you are currently with these things.
00:32:00.326 --> 00:32:01.248
You talk about costs,
00:32:01.268 --> 00:32:02.429
you talk about the processes,
00:32:02.449 --> 00:32:05.810
you talk about quality, perhaps security.
00:32:08.153 --> 00:32:09.713
It's the same as what you do currently,
00:32:09.773 --> 00:32:09.993
right?
00:32:10.075 --> 00:32:12.675
You have the DevOps maturity matrix with
00:32:12.695 --> 00:32:13.696
things currently, right?
00:32:13.977 --> 00:32:15.097
Where you are with things,
00:32:15.157 --> 00:32:17.259
how quickly you can
turn around your...
00:32:18.916 --> 00:32:20.662
security issues how quickly you can
00:32:20.701 --> 00:32:22.686
release your features to production is the
00:32:22.727 --> 00:32:25.674
same thing right like how do i do
00:32:25.734 --> 00:32:27.480
i have the cost that i can
00:32:29.113 --> 00:32:32.676
compare the cost with the outcome based on
00:32:32.896 --> 00:32:34.077
how much would it cost with people,
00:32:34.238 --> 00:32:35.460
that feature that gets released, right?
00:32:35.980 --> 00:32:36.401
Security,
00:32:36.421 --> 00:32:40.345
do I just get like a kill switch
00:32:40.785 --> 00:32:42.366
that the agent can invoke itself when
00:32:42.426 --> 00:32:43.186
something goes really,
00:32:43.887 --> 00:32:46.411
really wrong and it switches off correctly
00:32:46.851 --> 00:32:48.853
rather than just I have like an audit
00:32:48.873 --> 00:32:49.993
trail that just audits it.
00:32:50.694 --> 00:32:54.136
So I think I do get your three
00:32:54.176 --> 00:32:54.537
points.
00:32:54.718 --> 00:32:56.117
And I think the main thing for people
00:32:56.137 --> 00:32:58.099
to do is perhaps have this.
00:32:58.119 --> 00:32:58.619
Maybe, John,
00:32:58.640 --> 00:33:01.321
you can publish your maturity grid for a
00:33:01.402 --> 00:33:04.304
software factory that people can use to
00:33:04.364 --> 00:33:06.045
mark themselves against and see where they
00:33:06.085 --> 00:33:10.047
are currently and then put in a plan
00:33:10.067 --> 00:33:11.407
for how they can get to the point
00:33:11.428 --> 00:33:14.029
where the things are self governing.
00:33:15.851 --> 00:33:16.371
It's coming.
00:33:17.152 --> 00:33:18.212
Whether people like it or not,
00:33:18.613 --> 00:33:19.192
it is coming.
00:33:19.272 --> 00:33:21.595
It's going to happen.
00:33:22.335 --> 00:33:23.355
It's already here, right?
00:33:23.736 --> 00:33:25.518
I know you're already using agents on your
00:33:25.538 --> 00:33:25.897
machine,
00:33:25.978 --> 00:33:28.859
but people would want to use agents in
00:33:28.940 --> 00:33:30.260
hundreds of thousands, right?
00:33:30.300 --> 00:33:31.942
So you just talk to Slack,
00:33:32.001 --> 00:33:34.463
submit a job or in Teams or whatever,
00:33:35.003 --> 00:33:37.905
that job gets submitted to a cloud agent
00:33:37.925 --> 00:33:39.487
that's running in a container in
00:33:39.527 --> 00:33:40.008
Kubernetes.
00:33:40.067 --> 00:33:41.588
Kubernetes always keeps coming back no
00:33:41.608 --> 00:33:42.528
matter what you say, right?
00:33:42.548 --> 00:33:44.471
So running in Kubernetes somewhere,
00:33:45.230 --> 00:33:46.971
It's going to happen and you can run
00:33:47.050 --> 00:33:48.071
hundreds of these agents,
00:33:48.491 --> 00:33:49.971
thousands of these agents rather than
00:33:50.751 --> 00:33:52.211
nobody will have agents on their machines,
00:33:52.271 --> 00:33:52.471
right?
00:33:52.491 --> 00:33:54.252
Because you're just talking to them,
00:33:54.593 --> 00:33:56.073
submitting it, that's doing something,
00:33:56.093 --> 00:33:57.873
coming back with an interface,
00:33:57.913 --> 00:33:58.472
talking to it.
00:33:59.114 --> 00:34:01.773
So it's an exciting area for sure.
00:34:01.834 --> 00:34:04.974
I do like the things how we're moving
00:34:04.994 --> 00:34:06.615
towards, but at the same time,
00:34:06.634 --> 00:34:07.815
people trying to figure out how to do
00:34:07.835 --> 00:34:08.614
it more efficiently.
00:34:08.695 --> 00:34:09.715
And all the things, John,
00:34:09.735 --> 00:34:11.155
that you mentioned about the context
00:34:11.195 --> 00:34:11.536
graph,
00:34:12.295 --> 00:34:15.117
about having the quality gates in place,
00:34:15.876 --> 00:34:17.498
they're becoming even more important
00:34:17.518 --> 00:34:20.719
because you can now at the AI slope
00:34:21.378 --> 00:34:22.818
at a faster rate than you could with
00:34:22.898 --> 00:34:23.320
one agent, right?
00:34:23.440 --> 00:34:23.519
Yeah.
00:34:23.559 --> 00:34:26.000
And you make all those mistakes at a
00:34:26.059 --> 00:34:28.201
faster rate than you could before.
00:34:28.320 --> 00:34:30.181
Yeah, that's so true.
00:34:30.862 --> 00:34:31.521
Yeah, exactly.
00:34:32.242 --> 00:34:33.163
We will share then,
00:34:33.623 --> 00:34:35.523
I guess we can kind of wrap that
00:34:35.543 --> 00:34:36.864
up if anyone wants to know more,
00:34:36.923 --> 00:34:38.264
obviously can reach out to us,
00:34:38.324 --> 00:34:39.905
but we'll share then that maturity
00:34:41.123 --> 00:34:42.766
kind of assessment that you can kind of
00:34:42.786 --> 00:34:43.246
just do.
00:34:43.346 --> 00:34:45.829
It's very quick and easy on where you
00:34:45.869 --> 00:34:47.210
think you are on the spectrum and then
00:34:47.231 --> 00:34:48.632
where the investment might need to be made
00:34:48.672 --> 00:34:53.396
before you go hell leather on the AI
00:34:53.436 --> 00:34:55.940
software factory so you don't let it
00:34:55.960 --> 00:34:56.179
loose.
00:34:56.987 --> 00:34:58.588
But cool.
00:34:59.168 --> 00:35:00.590
You do also have a blog coming, John.
00:35:00.771 --> 00:35:01.570
You want to plug that in?
00:35:01.670 --> 00:35:02.271
I do.
00:35:02.371 --> 00:35:04.172
I do have a blog.
00:35:04.353 --> 00:35:04.793
Please do.
00:35:05.494 --> 00:35:06.253
We can share that.
00:35:06.293 --> 00:35:06.454
Yeah,
00:35:06.474 --> 00:35:08.295
we'll share about what it is and some
00:35:08.655 --> 00:35:10.217
learnings from other companies that are
00:35:11.117 --> 00:35:12.498
very high engineering-calibered
00:35:12.538 --> 00:35:16.161
organizations, Uber, Stripe, et cetera.
00:35:16.181 --> 00:35:19.222
They are pioneers of engineering,
00:35:19.262 --> 00:35:22.045
and obviously they are lucky enough to be
00:35:22.105 --> 00:35:24.025
able to implement this type of stuff
00:35:24.045 --> 00:35:25.487
because they've got the maturity in place.
00:35:25.527 --> 00:35:26.347
But then you can see,
00:35:27.108 --> 00:35:29.088
what they've done their learnings and and
00:35:29.148 --> 00:35:30.909
lessons that they've kind of had some of
00:35:30.929 --> 00:35:32.030
the data's in there as well in the
00:35:32.050 --> 00:35:33.829
blog to kind of make it easy to
00:35:33.869 --> 00:35:35.831
follow and understand but yeah we'll share
00:35:35.851 --> 00:35:38.211
that as well but that's it for us
00:35:39.532 --> 00:35:40.992
we should thank you see you next time
00:35:42.032 --> 00:35:44.574
bye-bye see you see you and the agents
00:35:44.614 --> 00:35:46.635
in the factory next time we'll see you
00:35:46.655 --> 00:35:47.514
in the factory yeah