РАСШИФРОВКА 🔗
WEBVTT
00:00:00.160 --> 00:00:04.719
So we noticed that agents tend to become crazy with comments.
00:00:04.959 --> 00:00:09.119
The more you use the factory, the more time it passes, the longer the comments become.
00:00:09.519 --> 00:00:20.559
You need to shift your mindset to thinking of a software factory as something that your code, context, and data is part of.
00:00:20.800 --> 00:00:23.359
It is not a separate thing, it's not a separate entity.
00:00:23.519 --> 00:00:30.640
It's not a some sort of like magic black box that does stuff for you that you can get angry at when it doesn't work.
00:00:30.800 --> 00:00:36.240
No, your code, your context, your data is also part of that factory.
00:00:37.200 --> 00:00:43.840
The AI Native Dev is a podcast for developers and engineering leads at the cutting edge of AI energetic coding.
00:00:44.000 --> 00:00:54.240
Join your hosts, Glyfer Germany, and me, Simon Maple, every week as we chat with the most exciting voices in AI and tackle the biggest questions facing developers today.
00:00:54.719 --> 00:00:57.280
This is the AI Native Dev.
00:01:00.079 --> 00:01:02.159
Yesterday, you were writing code.
00:01:02.320 --> 00:01:04.239
Today, you're prompting agents.
00:01:04.400 --> 00:01:06.959
Tomorrow, you'll be running a software factory.
00:01:07.280 --> 00:01:10.799
Agentic coding is moving fast enough to give anyone whiplash.
00:01:11.040 --> 00:01:13.840
This June, AI DevCon London sold out.
00:01:14.079 --> 00:01:24.400
Hundreds of developers, engineering leads, and VPs packed in to hear speakers from Anthropic, OpenAI, Netflix, and GitHub cut through the hype and get to what actually works.
00:01:24.560 --> 00:01:27.280
This November, we're bringing it back to New York.
00:01:27.519 --> 00:01:31.840
Bigger, bolder, and full of people actually building the agentic future.
00:01:32.079 --> 00:01:41.519
Three days of real talks, real tools, real answers, whether you're directing the agents, building the loops themselves, leading the teams, or setting the strategy.
00:01:41.840 --> 00:01:47.680
As a podcast listener, you can get 15% off your ticket using promo code POD50.
00:01:48.799 --> 00:01:52.879
We have so much planned, and we can't wait to have you join us in New York.
00:01:57.519 --> 00:02:01.439
Hello and welcome to another episode of the AI Native Dev.
00:02:01.519 --> 00:02:08.639
And today we're going to be talking about all things agents, harnesses, and specifically loops, and in particular, feedback loops.
00:02:08.719 --> 00:02:14.879
We're going to talk about what they are, how you can create them, and how you can actually benefit from feedback loops in your organizations.
00:02:15.120 --> 00:02:18.960
Joining me today is Maria Goranova, who is actually a repeat guest.
00:02:19.039 --> 00:02:20.240
So welcome back, Maria.
00:02:20.319 --> 00:02:20.639
How are you?
00:02:20.879 --> 00:02:22.800
Thank you, Sam, and good to be back.
00:02:23.039 --> 00:02:32.719
And last time we spoke,
Load full transcript
you talked to us uh about abstraction adherence, which is how agents uh use the right libraries, which was super interesting.
00:02:32.800 --> 00:02:35.439
But things have moved on, oh my gosh, so much more.
00:02:35.680 --> 00:02:36.000
So much.
00:02:36.240 --> 00:02:42.719
And it was in this very office that I heard you speak about all things uh loops, feedback loops, and software factories.
00:02:42.879 --> 00:02:49.199
And in fact, one of the opening questions that you that you asked the entire group was what is a software factory?
00:02:49.439 --> 00:02:52.159
And I think it means different things to different people.
00:02:52.319 --> 00:02:54.879
Um I'd love to I'd love to hear from you.
00:02:55.120 --> 00:02:56.319
What does it mean to you?
00:02:56.400 --> 00:03:00.879
And maybe even, you know, how does it how did it differ from person to person?
00:03:02.319 --> 00:03:05.599
Uh the takes were very different from each other.
00:03:05.680 --> 00:03:19.599
Yeah, the responses for sure, everybody had a different concept, everybody had a different angle uh at which they were looking at it, some were thinking only about automation, other were thinking about particular designs, about particular architecture.
00:03:19.840 --> 00:03:35.680
Um to me, I think uh the best way to describe what a software factory is kind of like go through history really of how agents have changed uh in the past two or so years, um or less than that.
00:03:36.000 --> 00:03:37.759
Um yeah is a long time.
00:03:38.080 --> 00:03:39.919
No, it's not a well, it's two years, is it?
00:03:40.080 --> 00:03:41.599
Uh I don't I I don't know.
00:03:41.680 --> 00:03:46.800
It it feels like it's been decades, but actually it's been such a short time.
00:03:47.120 --> 00:03:56.159
Um and I think you know, software factory is something we have been building towards in those many decades.
00:03:56.479 --> 00:03:57.120
Yeah.
00:03:57.759 --> 00:04:09.840
Um so um to me, a software factory is something where we have repeatable workflows, we have shared execution environment, and we have feedback loops.
00:04:10.400 --> 00:04:21.439
Uh and feedback loops are ways for the factory to maintain itself, to improve itself, to make sure it's operational and it doesn't degrade into quality.
00:04:21.519 --> 00:04:35.519
Because once we go into this world of everything is done by agent, we are like we are supervising the tasks that that go into the agent sometimes, but we don't really supervise the code.
00:04:35.680 --> 00:04:43.439
We don't really go into the details of writing code or even reviewing code at times, things deteriorate very quickly.
00:04:43.759 --> 00:04:46.399
It's interesting you didn't mention ticket to PR.
00:04:46.560 --> 00:04:49.040
So someone raises an issue, a PR is created.
00:04:49.199 --> 00:04:52.959
But that's kind of like a use case of the factory rather than what the factory is, I suppose.
00:04:53.279 --> 00:05:12.399
Yeah, I think that is one single workflow, one single automated workflow in a factory, which yeah, happens to be yeah, I want to write a linear uh issue, or I want to write a Jira ticket or something of the sort, and I want this to be then implemented and the PR to be opened and even merged automatically.
00:05:12.879 --> 00:05:25.279
Um But there is so much more to this, uh especially again when it comes to maintenance, when it comes to um improvements that are not necessarily you don't need necessarily a human for them.
00:05:25.439 --> 00:05:27.519
Instead, you just churn through a lot of data.
00:05:27.680 --> 00:05:37.759
Well, you the agent, the agent churns through a lot of data, and then it can suggest improvements and implement improvements that make the factory better and keep it operational.
00:05:38.079 --> 00:05:38.560
Yeah.
00:05:38.879 --> 00:05:48.319
As part of your session, you went through a brief history of of agents and and you know the various levels that they went through over the last what 30-40 years maybe?
00:05:48.800 --> 00:05:57.199
So you start you started with V1, just an agent, no context, um, or sorry, rather, no control uh over the the their performance.
00:05:57.360 --> 00:06:00.959
Um and unless of course you own the agent's code itself.
00:06:01.279 --> 00:06:05.759
Uh what were the big bottlenecks before context with with just an agent?
00:06:06.319 --> 00:06:09.279
I think I think you just said it in the question basically.
00:06:09.439 --> 00:06:14.000
So um if we only have an agent that we don't own, right?
00:06:14.079 --> 00:06:20.000
So maybe we are using cloud code that is closed source or some other closed source agent.
00:06:20.399 --> 00:06:29.920
We don't really we can't really do anything to improve the behavior of that agent if we don't have context around it, right?
00:06:30.079 --> 00:06:40.959
If we if we just if we're just prompting it and that is all that happens, we there is no much control we can um exert over that agent.
00:06:41.279 --> 00:06:52.800
So so an agent in a f agent without context in a factory, for example, uh from an autonomy point of view, from a trust point of view, it's very unlikely to do what we want it to do, so very unlikely to actually be of any use.
00:06:53.199 --> 00:06:59.920
I think so, yes, because everybody has a different code base, everybody has different requirements, everybody has different taste.
00:07:00.480 --> 00:07:16.720
So you can't really have one agent that simply based on a prompt without anything else around it, um, can behave in a way that it's compliant, um high in quality, uh, you know, things things like that.
00:07:17.120 --> 00:07:20.879
And and and then you moved on to V2, which is agent plus context.
00:07:21.120 --> 00:07:23.600
Um what did context really unlock for us then?
00:07:23.680 --> 00:07:26.399
And I guess what what's still missing from that?
00:07:28.000 --> 00:07:34.399
So context by context here, I mean things like at the beginning it was you know, the
Claude MD.
00:07:34.480 --> 00:07:34.639
Yeah.
00:07:34.800 --> 00:07:43.120
Claude and D, then it was agents and D, then we uh we had skills, uh also hooks are also a type of context.
00:07:43.279 --> 00:07:51.759
Um, all of these sort of documents that lie around in your repo, especially the ones that are specifically for use by agents.
00:07:51.920 --> 00:07:52.560
Yeah.
00:07:52.879 --> 00:08:01.279
Um, and uh I think what that unlocked is that now we have control over our experience inside of a particular repo.
00:08:01.360 --> 00:08:02.560
You can share it with your team, right?
00:08:02.639 --> 00:08:16.480
Like you if you have Cloud MD, you can have your personal one, or you can have one that is in the repo that is shared with the team, and it kind of like defines how this team wants an agent to operate in that code base.
00:08:17.360 --> 00:08:25.040
Um, and of course, you know, with skills that becomes more sophisticated because now we you have different workflows that can be for uh called and so on.
00:08:25.600 --> 00:08:36.799
Um I think what is missing is that here still we it's very it's all very manual, um, and there is no good way of evaluating this, right?
00:08:36.879 --> 00:08:40.639
Like, how do I answer the question of is my CloreMD good?
00:08:41.200 --> 00:08:42.480
Other than vibes?
00:08:43.360 --> 00:08:47.120
There is there is no really established mechanism for doing this.
00:08:47.279 --> 00:09:00.080
And also, if I wanted to improve the CloreMD, I I would just go and manually kind of like vibe something in it, or at best ask my agent to vibe something in it based on what more vibes.
00:09:00.320 --> 00:09:02.960
It's just there there is no really principled.
00:09:03.600 --> 00:09:10.639
So I guess we have evals, perhaps, which is maybe more of an offline test, more of a static test, perhaps.
00:09:10.960 --> 00:09:15.600
Um, but then perhaps this is where harnesses kick in and loops.
00:09:15.759 --> 00:09:19.759
So you you mentioned a harness as the agent, the context, and a feedback loop.
00:09:20.159 --> 00:09:22.639
Tell us what that actually now unlocks.
00:09:23.200 --> 00:09:31.440
So a feedback loop, the way I think about it is really it's an agentic workflow, like everything else in this concept.
00:09:31.679 --> 00:09:36.879
Um but it has three distinct stages.
00:09:37.440 --> 00:09:40.799
Stage number one is about analyzing some data stream.
00:09:41.039 --> 00:09:52.240
So you provided it with some data, let's say I don't know, the comments that people have given uh to PR to PRs.
00:09:52.720 --> 00:09:58.399
So I I take all of this data and I give it to the agent and I ask the agent to analyze it.
00:09:58.639 --> 00:10:12.480
So the second step here is um finding, based on this analysis, finding problems, potential problems or potential um inefficiencies, uh just things that can be better.
00:10:12.720 --> 00:10:28.320
Um in this particular example, we are if we're looking at PR reviews, uh maybe we
see that the same PR comment is given over and over again to um to someone's PRs or the or a factory P uh factory.
00:10:28.799 --> 00:10:30.000
Like a code review or something like that?
00:10:30.080 --> 00:10:31.200
Or yeah, like a code review.
00:10:31.679 --> 00:10:32.480
It could be anything, I suppose.
00:10:32.720 --> 00:10:49.679
Yeah, it could be that there is always, for example, uh there is a particular way uh developers in this repo want uh a class to be written or documented, and every time the agent doesn't do that, and every time the developers need to comment on it, right?
00:10:50.000 --> 00:10:52.080
Something like this, just a made-up example.
00:10:52.320 --> 00:10:59.039
Um so identifying this problem in the data, and then the third stage will be suggesting a fix.
00:10:59.440 --> 00:11:06.720
And this fix could be a code fix, it could be a refactor, it could be a bug fix, or it could be context improvements.
00:11:07.039 --> 00:11:19.039
So in this case, uh that I mentioned maybe uh the repo can benefit from a doc a doc that actually describes how we want a class to be defined in that repo.
00:11:19.600 --> 00:11:23.440
And and I guess from the from the data source point of view, let's start with the data source.
00:11:23.600 --> 00:11:25.360
Um, what makes a good data source?
00:11:25.440 --> 00:11:38.320
Does it matter if it's a human data source or uh or uh you know a data source which is actually coming from a set of logs which are generated from uh an another agent or or another app or something like that?
00:11:38.960 --> 00:11:39.919
It can be anything.
00:11:40.000 --> 00:11:47.200
It doesn't have to be human generated, it could be um the agent session logs where we can see um agents.
00:11:47.440 --> 00:12:05.200
So one thing that we have seen with in uh our internal dark factory, for example, is that agents are able to analyze their own session locks and see places of friction where they lose many turns to figure out how to use a tool, for example.
00:12:05.360 --> 00:12:14.799
So that would be a potential place for improvement where the agent can now have some context that helps it not have to lose those turns.
00:12:15.440 --> 00:12:20.320
Um But I don't know if I can answer what makes a good data source.
00:12:20.399 --> 00:12:32.559
It'll have to be a bit uh It sounds like it's anything that anything that displays, concisely displays friction or an or really has a kind of like a root cause of something that can be improved.
00:12:32.799 --> 00:12:33.840
Is that is that fair?
00:12:34.159 --> 00:12:42.320
I I don't know if you can give it a definition like that, because uh yeah, like it's not obvious.
00:12:42.480 --> 00:12:48.080
Sometimes it's not obvious how a data source will with the agent session logs, I didn't know that will happen.
00:12:48.159 --> 00:12:52.399
That was kind of it was cool that it happened, and in high sight we can explain it.
00:12:53.519 --> 00:13:03.519
But um it's not that uh I thought, oh, let's introduce agent session logs because it's going to identify exactly these kind of problems.
00:13:03.840 --> 00:13:04.320
Yeah, yeah.
00:13:04.799 --> 00:13:07.600
So the data sources we should just try multiple data sources.
00:13:07.919 --> 00:13:08.559
Yes, I think so.
00:13:08.720 --> 00:13:16.000
I think it's a it's a matter of experimenting and seeing what comes out of it and and um yeah.
00:13:16.799 --> 00:13:18.720
And what judges the analysis?
00:13:18.879 --> 00:13:26.960
So you talk about the analysis, it's it's essentially trying to identify issues that repeatedly occur.
00:13:27.519 --> 00:13:29.360
Um what does that?
00:13:29.440 --> 00:13:35.120
Is it is there a is there a specific agent that we use for that, or is I know Jeff has just interestingly come out.
00:13:35.200 --> 00:13:37.039
That's possibly an option, I suppose.
00:13:37.759 --> 00:13:39.120
So it it depends.
00:13:39.279 --> 00:13:50.320
Um in this particular case, I was talking about feedback loops that kind of receive data over many, many changes, like many PRs, for example, many agent sessions, and so on.
00:13:50.960 --> 00:14:04.399
Um and for that you you just it's the feedback loop is an agentic prompt, and the agent you know works through that by looking at the data and doing certain steps and figuring out things.
00:14:05.360 --> 00:14:16.720
But we also have other kind of feedback loops that are more tightly scoped to an individual change, uh, which uh we call the inner feedback loops.
00:14:17.360 --> 00:14:42.240
Um those ones, for example, we can have something speaking of Jeff, something like verifiers, so verifiers allow you to um define a bunch of criteria that you want a change or your code to meet, and then use an LLM as a judge approach to verify if that criteria is met.
00:14:42.639 --> 00:14:46.799
And you can use Jeff for the for that, for example, or you can use another model.
00:14:47.759 --> 00:14:53.759
And why would I use that over what a code review already provides?
00:14:55.360 --> 00:15:00.399
I think it's um I wouldn't say you should use it over what code review already provides.
00:15:00.639 --> 00:15:01.440
Or as well as, rather.
00:15:01.679 --> 00:15:03.360
I would say as well as, yes.
00:15:03.519 --> 00:15:09.679
Like it it is um it is an additional dimension to it because code review is a little bit more open-ended.
00:15:09.840 --> 00:15:10.159
Okay.
00:15:10.320 --> 00:15:12.879
Um, a little bit heavier, I guess.
00:15:13.039 --> 00:15:15.120
Um even if you do it with an agent.
00:15:15.440 --> 00:15:21.120
Uh if you require human time, then it's obviously much more expensive than running an LLM as a judge.
00:15:21.600 --> 00:15:26.799
Um but even when you run an agent, like it takes a little bit of time, you know, it's open-ended.
00:15:26.960 --> 00:15:29.279
Uh verifiers are very tightly scoped.
00:15:29.519 --> 00:15:39.360
You've defined specific rules that you want your code to follow that are not
something that is easy to define with tests or with linting.
00:15:39.679 --> 00:15:41.600
Um it is a it is a form of linting.
00:15:41.679 --> 00:15:47.519
I would I would actually classify it as a deterministic in terms of the triggering of the verifier?
00:15:48.080 --> 00:15:51.840
Uh yes, we do trigger the verifier on every change.
00:15:52.080 --> 00:15:58.080
So when when a PR is up during um CI, we would we would trigger the verifiers.
00:15:58.480 --> 00:16:03.759
But the output is I wouldn't call it deterministic because it's still done by LLMs.
00:16:03.840 --> 00:16:03.919
Yeah.
00:16:04.080 --> 00:16:04.240
Yeah.
00:16:04.320 --> 00:16:05.200
Okay, got it.
00:16:05.440 --> 00:16:07.759
Um so you talk about the inner loop.
00:16:07.840 --> 00:16:11.039
Uh there's also kind of, I think there's middle loops and outer loops as well, right?
00:16:11.840 --> 00:16:14.960
Talk us through uh what the other loops are as well.
00:16:15.440 --> 00:16:16.000
Yes.
00:16:16.320 --> 00:16:27.440
So uh just to uh to clarify, so the inner loop we define loosely as something that operates on a single change to the code base.
00:16:28.159 --> 00:16:36.799
The middle loop is more like the examples I was giving before, where we have many, many changes to the code base and we get some data from them.
00:16:36.879 --> 00:16:48.480
We get the agent locks, uh the PR comments, but like from all of these changes, so that we so that the agent can, in a sense, generalize by looking at many, many examples across many sessions.
00:16:49.200 --> 00:16:53.600
Um so that that is how we again loosely defined Biddle Loop.
00:16:54.159 --> 00:16:58.159
Outer loop is more about production signals.
00:16:58.399 --> 00:16:58.720
Okay.
00:16:59.120 --> 00:17:02.399
So if you want, for example, self-heal.
00:17:02.879 --> 00:17:16.480
Uh if the service that is running the dark factory fails at any point, we want it to be able to detect that it failed, see the error, and then self-heal um by fixing that.
00:17:16.640 --> 00:17:18.480
So that would be like an outer loop.
00:17:18.720 --> 00:17:31.279
Another one would be if if we have um a certain type of customer um signals, uh, for example, I don't know, um daily active users, yeah.
00:17:31.440 --> 00:17:31.680
Right?
00:17:31.839 --> 00:17:39.200
If we see that going down, we want something to investigate what's going on and find the uh find the issue.
00:17:39.359 --> 00:17:41.119
Um so that would be another outer loop.
00:17:41.440 --> 00:17:43.759
And what are the triggers for the middle and outer loops?
00:17:43.920 --> 00:17:50.960
Uh the the inner loops is obviously the trigger is the thing that happens, maybe it's a pull request that gets raised, so the loop runs on that.
00:17:51.119 --> 00:17:55.200
Um is it just scheduled for the middle and uh outer loops?
00:17:55.599 --> 00:18:00.480
It depends what exactly you need and what your data is and and so on.
00:18:00.720 --> 00:18:04.960
Uh for most middle loops that
we have, uh the trigger is a schedule, yes.
00:18:05.039 --> 00:18:11.839
So we run them every day or every week, um, sometimes uh every month for the very heavy ones.
00:18:12.319 --> 00:18:22.079
Um and they just they will just continuously kind of monitor um the code base and make sure that everything is fine.
00:18:22.240 --> 00:18:22.559
Yeah.
00:18:22.799 --> 00:18:34.720
Um the outer loops more they're more likely to be triggered on an event, as I said, if the service is failing, or if some metric is going down that we want to go up, something like that.
00:18:35.039 --> 00:18:35.440
Cool.
00:18:35.680 --> 00:18:45.119
Um I'd love to also, I mean, the the the picture looks wonderful in terms of these loops, it's self-fixing, um, but I presume it doesn't always work perfectly.
00:18:45.279 --> 00:18:51.759
What where are the kind of areas that if someone were to create a loop for the first time, they might expect it to go wrong?
00:18:51.920 --> 00:19:04.880
Perhaps it's the the identify the identification of maybe it's the data source actually, then maybe it's the analysis or the judging on the on the data to be able to perform that that change, maybe it's the change itself.
00:19:04.960 --> 00:19:09.519
What what are the kind of areas that you would say are the hottest, the hot spots for for problems?
00:19:10.319 --> 00:19:17.839
So if you wanted to create your first feedback loop, I would say the most important thing is to be very clear with your intent.
00:19:18.240 --> 00:19:39.680
You need to be very clear what the data source is, what you expect um or what you want the agent to look for in that data source, and what do you want it to improve based on that data source uh or based on the findings from the data source?
00:19:40.319 --> 00:19:55.200
Um if you're vague, I think that is that is probably the the place that will be a hotbed for disaster because if you're vague, if you just say, Oh yeah, look at my past PRs and improve my repo.
00:19:55.839 --> 00:19:58.079
This is not this is not clear intent.
00:20:00.160 --> 00:20:03.200
It's trying to fix everything and it actually needs to fix a very specific thing.
00:20:03.440 --> 00:20:07.039
Um but also it doesn't the fixes are not going to be deep.
00:20:07.279 --> 00:20:15.039
There's not there there'll be a lack of depth and it will just make up things that maybe are true but not important, that that sort of thing.
00:20:15.359 --> 00:20:22.160
You want to create depth by really clearly explaining where uh to look for problems.
00:20:22.400 --> 00:20:38.000
For example, simple uh middle feedback loop we have is um one that uh we call comment diet, because we notice that agents uh tend to become crazy with comments.
00:20:38.319 --> 00:20:47.200
They would just write a lot, like lots and lots of comments, and it the comments become longer and longer because the next agent sees long comments and then it thinks it needs to write.
00:20:48.079 --> 00:20:50.000
It is honestly it's mad.
00:20:50.319 --> 00:20:53.519
We've had some crazy files uh lying around.
00:20:53.759 --> 00:21:01.920
So the comment diet feedback loop would go and just the data source is very simple, it's just the code base, there is no additional data source.
00:21:02.079 --> 00:21:21.279
So just look at the repo and run some scripts to kind of um count how many lines comments versus code there is, identify hotbed files that have too long of comments, that the comments are too long, uh, and then fix that.
00:21:21.440 --> 00:21:26.799
Just reduce the reduce the lines of comments in a way that doesn't degrade performance, right?
00:21:26.880 --> 00:21:28.000
So this is more concrete.
00:21:28.160 --> 00:21:38.240
Now I'm like I've I've reduced it to something very, very specific, and it's not going to touch your code, it's not going to touch your architecture, and you know, big changes like this.
00:21:38.400 --> 00:21:43.119
You know that it does just that that one thing and it's focusing all its attention to do that one thing.
00:21:43.680 --> 00:21:49.839
And then you can define a separate feedback loop to, I don't know, look at the architecture or the design or something else that you want.
00:21:50.160 --> 00:21:50.400
Yeah.
00:21:50.640 --> 00:21:56.720
So in terms of feedback loops there, would it would it be is it fair to say there's almost like a couple of feedback loops there?
00:21:56.799 --> 00:21:59.759
One one which is maybe an inner loop.
00:22:00.480 --> 00:22:04.400
That when changes are made, comments aren't too verbose.
00:22:04.880 --> 00:22:12.000
And one or rather, you know, that that that's a that's a maybe a skill um point on the inner loop.
00:22:12.319 --> 00:22:20.799
And on the on the middle loop, maybe that's more of a scheduled thing, doing an audit essentially of your code base to check that that's that that's okay and that's happening.
00:22:20.880 --> 00:22:22.799
Is that is that the right way of looking at it?
00:22:23.519 --> 00:22:28.240
Um I think for some things you can divide it this way and have both inner and middle.
00:22:28.480 --> 00:22:34.400
Um I don't know if outer will make sense here, probably not, but um yes, you can have inner and middle.
00:22:34.559 --> 00:22:42.720
It depends on whether you expect this to be a drift or not, this this kind of behavior.
00:22:42.880 --> 00:22:48.799
Because so in this particular case, we notice that the comment bloat is a drift.
00:22:48.960 --> 00:22:53.759
Yeah, the more you use the factory, the more time it passes, the longer the comments become.
00:22:54.240 --> 00:23:02.240
And yes, we can have some gates on the inner loop level um that check if the comments are too verbose and stuff like that.
00:23:02.400 --> 00:23:06.960
But I my suspicion is that even if we have those gates, it will still happen over time.
00:23:07.359 --> 00:23:20.319
And and just to find what drift is, is that is that uh is that is that a difference between what we expect to happen based in
the context and the reality in the in the code base good question.
00:23:20.559 --> 00:23:27.920
Um I would define drift as some drift from some ideal state that that we hope for.
00:23:28.400 --> 00:23:35.680
And uh drift happens in human-maintained codebases too, like even before agents, you know, even before all of this.
00:23:36.160 --> 00:23:53.599
Um when many many developers are contribu contributing to uh a code base, over time, small errors compound, and then at one point you have to do a refactor or you or you discover a big bug or something like that.
00:23:54.000 --> 00:23:58.640
Um there is uh a very good example uh for this is actually the rule of three.
00:23:58.720 --> 00:23:59.839
You know the rule of three?
00:24:00.160 --> 00:24:00.880
Tell me.
00:24:01.279 --> 00:24:06.880
Well, if you uh if you have to write code once, obviously you write it once.
00:24:07.119 --> 00:24:09.920
If you have to write it twice, you write it twice twice.
00:24:10.319 --> 00:24:14.400
But if you have to write it a third time, now is the time to introduce an abstraction.
00:24:14.559 --> 00:24:15.279
Yeah, right.
00:24:15.759 --> 00:24:20.960
So it's kind of like this like uh humans do it, agents will do the same.
00:24:21.039 --> 00:24:34.079
Like you're going to write the code once, you're gonna write it twice, and then on the third time you need to uh introduce an abstraction, but that's not the job of the agent writing the third time code, that's a job of something else, something beyond that that is maintained.
00:24:34.480 --> 00:25:09.599
That even recognizes it's been done three times, whereas the agent won't necessarily exactly so it is kind of like it's something beyond, so that's why the middle loop looks at many, many sessions, because it it needs the bigger picture, it needs to generalize and correct those drifts, correct any spaghetti codes, yeah, as I like saying, um and um kind of you know fold back the repo in a nice state, in a better state that is healthy, um, that subsequent agents will have a good time working in.
00:25:09.839 --> 00:25:10.240
Yeah.
00:25:10.559 --> 00:25:13.359
Hey everyone, hope you're enjoying the episode so far.
00:25:13.599 --> 00:25:21.759
Our team is working really hard behind the scenes to bring you the best guests so we can have the most informative conversations about agentic development.
00:25:21.920 --> 00:25:27.279
Whether that's talking about the latest tools, the most efficient workflows, or defining best practices.
00:25:27.519 --> 00:25:31.119
But for whatever reason, many of you have yet to subscribe to the channel.
00:25:31.279 --> 00:25:38.079
If you're enjoying the podcast and want us to continue to bring you the very best content, please do us a favour and hit that subscribe button.
00:25:38.240 --> 00:25:45.599
It really does make a difference and lets us continue to improve the quality of our guests and build an even better product for you.
00:25:45.839 --> 00:25:47.839
Alright, back to the episode.
00:25:48.400 --> 00:25:53.440
Let's get practical now and and and in for for people who want to get started.
00:25:53.599 --> 00:25:58.079
Um, one of the things that you kind of like mentioned is is getting the correct mindset.
00:25:58.240 --> 00:26:03.039
What's the correct mindset for creating loops and thinking about loops?
00:26:03.519 --> 00:26:14.400
You need to shift your mindset to thinking of a software factory as something that your code, context, and data is part of.
00:26:14.880 --> 00:26:25.200
It is not a separate thing, it's not a separate entity, it's not uh some sort of like magic black box that does stuff for you that you can get angry at when it doesn't work.
00:26:25.440 --> 00:26:31.440
No, your code, your context, your data is also part uh of that factory.
00:26:32.400 --> 00:26:44.079
Um just to give an example, imagine you have uh project A and project B that semantically uh do exactly the same thing, right?
00:26:44.240 --> 00:26:47.359
When you run things through them, it's exactly the same behavior.
00:26:47.680 --> 00:27:03.279
But project A is written well, like the code is well structured, has good abstractions, uh, there is lots of context available, there is you know agents and the uh skills in it, um, there is uh I don't know, hooks that configure the agent.
00:27:03.599 --> 00:27:09.039
There is like all sorts of mechanisms that um control this agency uh behavior.
00:27:09.519 --> 00:27:11.920
Um and project B has none of that.
00:27:12.079 --> 00:27:14.000
The code is looks like spaghetti.
00:27:14.079 --> 00:27:17.039
It's the same code semantically, but it looks like spaghetti.
00:27:17.839 --> 00:27:23.759
It is obvious that an agent will perform better in uh project A, right?
00:27:23.920 --> 00:27:24.880
Very much like a human.
00:27:25.039 --> 00:27:29.680
A human will also be more confused by the spaghetti code than by the well-structured code.
00:27:30.000 --> 00:27:33.039
So this is what I mean by this.
00:27:33.200 --> 00:27:37.279
Uh you have control over your code, context, data.
00:27:37.440 --> 00:27:45.440
Uh, you can use all of this as part of the factory to make the factory better, because the factory is also your context code and data.
00:27:45.839 --> 00:27:46.319
Okay, cool.
00:27:46.400 --> 00:27:51.359
And I guess I guess another thing is there are a couple of different types of loops that we talk about internally here at Tesla.
00:27:51.440 --> 00:27:54.799
We have a regular loop, and then we have a feedback loop.
00:27:54.960 --> 00:27:56.720
What's the difference between the two?
00:27:57.440 --> 00:28:00.720
Well, I guess for feedback loops, we already talked about them.
00:28:00.799 --> 00:28:12.319
It's that idea of a loop that analyzes some data source, finds some problems and fixes them, or rather um requests fix fixes for them.
00:28:12.960 --> 00:28:25.920
Um what we call a general loop in Tesla is more of a any any sort of workflow, any any agentic workflow um can
be a loop.
00:28:26.559 --> 00:28:35.039
So um we have for example loops that um define the issue to PR, what um what that looks like.
00:28:35.279 --> 00:28:44.000
So how an agent should look at an issue, read the issue, uh plan the change, implement the change, and open a PR.
00:28:44.480 --> 00:29:02.880
Uh we have loops that look at PR review and can respond when there is a comment uh on on a PR, can then trigger um um and you know, either implement the comment or maybe push back if they don't agree with the comment, that sort of thing.
00:29:03.599 --> 00:29:06.240
Um and feedback loops are type of loops.
00:29:06.799 --> 00:29:13.279
It it almost feels like a the definition of a general loop isn't actually a loop, but I think you mentioned it was a workflow.
00:29:13.440 --> 00:29:24.079
It doesn't feel like it loops back on itself, whereas a feedback loop truly does feedback, it loop it loops, you know, takes information, passes it back, and and actually does a loop.
00:29:24.480 --> 00:29:30.480
Yeah, it is a little meta, I think, because when we say yeah, it I think naming is hard.
00:29:30.640 --> 00:29:43.839
I think when we when we say just loop, even in those workflows that are kind of linear in a way, um but I like I think we are referring to the loop of the agent inside it, right?
00:29:43.920 --> 00:29:45.359
Because this agent is in a loop.
00:29:45.519 --> 00:29:52.400
This agent obviously, you know, uh loops through um many steps and tools and decisions and so on.
00:29:52.640 --> 00:29:55.599
Um so that's what we mean by loop.
00:29:55.680 --> 00:29:59.839
And the feedback loop is like, I don't know, a loopy loop, I guess.
00:30:00.319 --> 00:30:01.680
The loopiest of all the loops, yeah.
00:30:02.079 --> 00:30:02.319
Like it.
00:30:02.480 --> 00:30:03.599
Oh no, there is more loopy loops.
00:30:04.400 --> 00:30:21.440
Yeah, there is there is there is the meta feedback loops that now would uh would analyze data uh that kind of tells you if a feedback loop is doing well, and then make changes to that feedback loop in order to improve that feedback loop.
00:30:21.599 --> 00:30:24.799
So now we're managing the loops, essentially, right?
00:30:24.960 --> 00:30:25.440
Yes, yes.
00:30:25.519 --> 00:30:27.599
Now we are looping instead of a loop instead of a loop.
00:30:27.839 --> 00:30:31.119
Yeah, and I suppose that could go up any number of depths, I suppose.
00:30:31.519 --> 00:30:32.799
Yeah, yeah, it's cool, no?
00:30:32.880 --> 00:30:34.079
It's it's a it's recursive.
00:30:34.400 --> 00:30:43.680
That's really interesting actually, because you don't, you know, a feedback loop, you you you know, you want it to be correct, you want it to be making good changes, but how do you know it is actually making the right changes?
00:30:43.839 --> 00:30:46.000
So it's everything needs that feedback.
00:30:46.079 --> 00:30:47.279
Um super interesting.
00:30:47.519 --> 00:30:50.319
Yeah, and that's why so measurement is extremely important.
00:30:50.400 --> 00:31:14.559
That's something I think we haven't touched on in this discussion, but measuring um the impact of of changes is actually really um very important it is because otherwise if you're just you know you're just making changes, but if you can't observe that uh that's having a good effect, then you can't really trust that anything is happening.
00:31:14.880 --> 00:31:18.799
Yeah, and I was actually gonna ask the next question, which was around human interaction.
00:31:18.880 --> 00:31:28.079
So when do when do human like obviously with loops and and really the goal of this is to really increase that autonomy, increase that trust.
00:31:28.240 --> 00:31:30.000
When do humans get involved?
00:31:30.079 --> 00:31:37.119
And I guess I can kind of like follow on to your point there in terms of like that observing what good should be and and things like that.
00:31:37.200 --> 00:31:43.440
When when does a human actually enter that that being part of that judging process?
00:31:45.119 --> 00:31:47.920
Everywhere, Simon, and all the time.
00:31:48.160 --> 00:32:03.839
Um I think I think many people might find it a little bit daunting that we're moving towards so much automation, uh, but actually I think it's just uh an abstraction level.
00:32:04.079 --> 00:32:23.440
We are still doing so much work to uh like analyze what's happening, um request the right changes, monitor if the loops are performing well or not, coming up with new metrics that we think will um you know reflect better if something is working or not, all of these things.
00:32:23.920 --> 00:32:24.480
It never ends.
00:32:24.559 --> 00:32:28.160
I don't think the factory is something that can just be built and done.
00:32:28.319 --> 00:32:28.480
Yeah.
00:32:28.720 --> 00:32:35.839
Like with any software, it it it's especially when you make the your code base, uh, your context and your data part of it.
00:32:36.000 --> 00:32:45.279
Uh it it constantly evolves and you're part of this evolution, and uh that is where the work is rather than code implementation.
00:32:45.599 --> 00:32:46.000
Yeah.
00:32:46.400 --> 00:32:56.160
And and in terms of roles, let's get let's talk a little bit about roles for humans in the in the software development um world going forward with software factories.
00:32:56.400 --> 00:33:13.920
I guess software factories, if we have 10 repos, should we have 10 software factories, or are there shared components where you might have a shared orchestrator and and and multiple uh more specific components that are repo specific?
00:33:14.160 --> 00:33:23.759
Um how much of this how much of this rule require humans to to manage versus actually building the product?
00:33:23.839 --> 00:33:29.039
Is there going to be is there a role where a human will do both of those as part of their job, do you think?
00:33:29.200 --> 00:33:40.400
Or do you think there's more of an infra role where people manage the software factory and more of a app
development role where people manage the you know what needs to be built and how it gets built?
00:33:41.039 --> 00:33:42.640
A really good question, Simon.
00:33:42.880 --> 00:33:43.519
I really like this.
00:33:43.920 --> 00:33:47.839
These are all good questions, yeah.
00:33:47.920 --> 00:33:48.960
Why am I surprised?
00:33:49.440 --> 00:34:02.160
Um at Tesla internally, we actually, um, especially when it comes to metrics and uh improving on those metrics, we actually talk about several different personas.
00:34:03.119 --> 00:34:07.839
Uh one of the personas we call uh an operator, dark factory operator.
00:34:07.920 --> 00:34:11.679
So I think this is what you just described as like an infrarole.
00:34:11.920 --> 00:34:23.599
Um this is somebody who really cares about the uh main loops of the dark factory, um, and make sure the dark factory is operational and it can service everyone and so on.
00:34:23.840 --> 00:34:26.800
Um and then we have the steward role.
00:34:26.960 --> 00:34:30.000
And the steward role, we have many stewards at Tesla.
00:34:30.239 --> 00:34:36.320
Uh they're stewards of different areas or different projects, um, different repos, if you wish.
00:34:36.639 --> 00:34:41.920
And it's kind of like uh the new code owner, if that makes sense.
00:34:42.159 --> 00:34:46.400
So the code owner is funny though, because they didn't write the code.
00:34:46.639 --> 00:34:47.519
That's the idea, right?
00:34:47.599 --> 00:34:49.840
That we like everything goes through the factory.
00:34:50.000 --> 00:34:51.760
Uh so that's why stewards.
00:34:51.840 --> 00:34:55.920
So they know about this area, they they are kind of responsible for this code base.
00:34:56.480 --> 00:35:02.719
Uh and they make sure that the dark factory operates well in their code base.
00:35:03.280 --> 00:35:15.920
So there is um I think both roles need to think about the factory and think about this new way of development, but in a different through different lenses, if that makes sense.
00:35:16.079 --> 00:35:16.559
Yeah.
00:35:16.880 --> 00:35:23.039
Um whether this is one dark factory or many, I think I think both.
00:35:24.079 --> 00:35:43.840
I think it's both one dark factory, and also each project, each repo is its own thing, but it can go through the common orchestrate uh orchestration, but then it it can define its own, it can and it should define its own feedback loops and its own special purpose loops and so on, um, so that it works better.
00:35:44.159 --> 00:35:55.119
Yeah, I think that's really key as well, that you know, you can't just expect one repo to you know copy paste all the context and everything in terms of how to operate um and expect it to work just as well in the next one.
00:35:55.280 --> 00:36:00.480
It's everything needs to iterate and learn from the repo it's in, um, which is which is super important.
00:36:00.800 --> 00:36:01.519
Yeah, absolutely.
00:36:01.760 --> 00:36:13.280
Um Maria, what advice would you give people who haven't you know kicked off their first loop but really wants to actually build, wants to they like what they hear and they want to build a loop.
00:36:13.360 --> 00:36:15.119
What what advice would you give them?
00:36:15.679 --> 00:36:19.760
Come to AIDEF CON Brilliant.
00:36:20.480 --> 00:36:23.920
Uh yeah, no, I think that might be very helpful.
00:36:24.800 --> 00:36:37.840
Um I would say what I was referring to before in terms of how do you how do you get going with one of those feedback loops, uh be very clear with your intent.
00:36:38.000 --> 00:36:55.679
Think about what's the data source, think about what um what signals, especially signals that when repeated make sense are in um can be found uh in that data source, and then think about what can you improve based on this.
00:36:56.079 --> 00:37:10.320
In fact, and and perhaps focus on those pain points that you know what is it that your agents are tripping up on most, and almost like start building the loops around those because those are most likely the areas it can it can loop and improve and feedback the most.
00:37:10.880 --> 00:37:13.199
Absolutely, because then it can improve its own behavior.
00:37:13.360 --> 00:37:15.760
Yeah, it's it becomes uh self-improvement.
00:37:16.000 --> 00:37:16.400
Yeah.
00:37:16.639 --> 00:37:17.119
Amazing.
00:37:17.199 --> 00:37:22.719
Well, Maria, you you've been on the podcast before, you've looped right round and been on the podcast uh once again.
00:37:22.960 --> 00:37:24.480
Um thank you very, very much.
00:37:24.719 --> 00:37:27.119
Really amazing insights and appreciate you uh joining us today.
00:37:27.440 --> 00:37:28.320
Thank you, Simon.
00:37:28.480 --> 00:37:29.440
Really appreciate it.
00:37:29.760 --> 00:37:32.880
Thank you very much everyone for listening and uh see you soon on another episode.
00:37:33.039 --> 00:37:33.840
Bye for now.
00:37:34.159 --> 00:37:39.039
The AI Native Dev is brought to you by Tesla, the package manager for skills and context.
00:37:39.119 --> 00:37:42.159
Your hosts are Guy Pajani and me, Simon Maple.
00:37:42.239 --> 00:37:43.760
Our producer is Tom Dowler.
00:37:44.000 --> 00:37:47.280
The AI Native Dev is not just a podcast, it's a community.
00:37:47.440 --> 00:37:50.880
And we host monthly meetups at the Tesla offices in central London.
00:37:51.039 --> 00:37:56.079
Visit Tesla.io forward slash community to learn more, and I hope to see you there.