逐字稿 🔗
WEBVTT
00:00:00.000 --> 00:00:02.879
For me, I'll never write another line of infrastructure as code.
00:00:03.120 --> 00:00:09.599
We use Swamp to manage our own infrastructure, and we've actually changed production infrastructure four times in the last six months.
00:00:09.759 --> 00:00:10.720
Agents don't care.
00:00:10.880 --> 00:00:15.119
They're like, your cloud is here, the new cloud is here, let's do the migration.
00:00:15.279 --> 00:00:26.480
We've gone from railway to DigitalOcean app platform, we've gone to DigitalOcean Kubernetes, and now we've actually gone to like uh GCP, and every time it's been like completely flawless and completely seamless.
00:00:26.559 --> 00:00:28.399
And that includes full data migration.
00:00:28.480 --> 00:00:33.840
It's kind of amazing to watch because a year ago we wouldn't have been in that situation.
00:00:34.880 --> 00:00:41.520
The AI Native Dev is a podcast for developers and engineering leads at the cutting edge of AI and agentic coding.
00:00:41.679 --> 00:00:51.920
Join your hosts, Sky Pajani, and me, Simon Maple, every week as we chat with the most exciting voices in AI and tackle the biggest questions facing developers today.
00:00:52.320 --> 00:00:54.640
This is the AI Native Dev.
00:00:58.799 --> 00:01:01.679
AI DevCon is coming back to London.
00:01:01.840 --> 00:01:04.000
Our first London event sold out in June.
00:01:04.159 --> 00:01:15.840
After our second New York event this November, we're heading back to the brewery Moorgate in 2027 with a bigger lineup and two full days of talks from the people actually building with agents.
00:01:16.000 --> 00:01:25.680
You'll hear honest takes on where the industry's at, real stories from teams adopting AI, the wins as well as the messy bits, and where they think things are heading next.
00:01:25.840 --> 00:01:27.439
It's on June 7th and 8th.
00:01:27.519 --> 00:01:30.400
And don't tell anyone, but we've got a really great deal for you.
00:01:30.560 --> 00:01:34.159
Scan this QR code to grab an early bird ticket for just£100.
00:01:34.799 --> 00:01:36.079
See you in London.
00:01:39.519 --> 00:01:42.879
Hello everyone, welcome back to the AI Native Dev podcast.
00:01:43.120 --> 00:01:48.640
Today we're recording this actually at AIDEVCON, TESL's semi-annual conference on it.
00:01:48.799 --> 00:01:49.760
Super fun to doing it.
00:01:49.840 --> 00:01:51.519
There is a session going on in the background on it.
00:01:51.599 --> 00:01:56.400
So if you hear some clapping or cheering on it, that's just like normal occurrence here at AIDEVCO.
00:01:57.359 --> 00:02:02.719
And specifically over here, we'll do uh a short podcast with Paul Stack here from Swamp.
00:02:02.879 --> 00:02:07.840
We'll talk more in a sec, followed by his talk that will come up later in this conference on it.
00:02:08.000 --> 00:02:10.080
So, Paul, first of all, thanks for coming onto the podcast.
00:02:10.479 --> 00:02:11.280
Thanks so much for having me.
00:02:11.360 --> 00:02:12.240
It's been a pleasure.
00:02:13.360 --> 00:02:21.520
And uh so you you do a lot of things, but I think probably like a core competence is the whole world of uh infrastructure as code and the build-up.
00:02:21.759 --> 00:02:26.960
Uh, can you give us just a quick kind of background note on just a few of the highlights of your past on it?
00:02:27.039 --> 00:02:28.879
And importantly, what does Swamp do?
00:02:29.280 --> 00:02:34.080
So my past is I I helped build a lot of what Terraform is today.
00:02:34.159 --> 00:02:36.159
I was very early doors in HashiCorp.
00:02:36.240 --> 00:02:41.759
I was there for quite a few years, left HashiCorp, went to Pulumi, worked on Pulumi for four years after that.
00:02:41.919 --> 00:02:45.120
I've spent like just an extraordinary amount of my career in infrastructure as code.
00:02:45.520 --> 00:02:46.639
To define infrastructure as well.
00:02:46.719 --> 00:02:47.039
Well, yeah.
00:02:47.360 --> 00:02:49.199
You should automate some of that infrastructure creation.
00:02:50.800 --> 00:02:53.199
And then I joined a company called System Initiative.
00:02:53.280 --> 00:02:57.520
Um, and System Initiative was trying to reimagine how infrastructure as code was.
00:02:57.599 --> 00:03:09.039
It was trying to use digital twins to be able to manage it, but it just never really hit that product semen with users who wanted it to be very declarative and they wanted it to be in Git.
00:03:09.120 --> 00:03:29.680
So earlier this year we created a new company called Swamp, and we sort of thought about how we would actually bring agents into like managing infrastructure, managing like systems as a first-class citizen rather than you know what unfortunately a lot of other people had to do, which was try and retrofit their code and their system to work with agents.
00:03:29.919 --> 00:03:31.360
Yeah, yeah, interesting.
00:03:31.599 --> 00:03:32.879
Definitely a world over there on it.
00:03:32.960 --> 00:03:33.840
So we'll touch on it.
00:03:34.000 --> 00:03:39.759
Maybe let's do baby steps a little bit, but um, so there's a lot of work going on around sort of tickets to PR, right?
00:03:39.840 --> 00:03:45.840
How do you kind of go from whatever, like an intent or something into like a pull request and maybe even all the way to like auto-merging it?
00:03:45.919 --> 00:03:54.879
Uh and I think as more people uh advance down that path, there's now a more attention to the uh sort of PR to production, if you will, right?
00:03:54.960 --> 00:03:56.719
Like how do we actually get that and get that deployed?
00:03:56.800 --> 00:03:58.479
Because otherwise, you know, what have you done?
00:03:58.639 --> 00:04:05.039
Um and in that world, you know, we do use like infrastructure's code is useful because agents can generate that.
00:04:05.120 --> 00:04:19.519
But there's been a lot of kind of complaints around how infrastructure's code, maybe unlike other types of code, the the amount of like a public uh kind of you know
Load full transcript
reservoir of examples of code and all that were not good, and that the models were not as good at it.
00:04:19.759 --> 00:04:37.120
So maybe let's start with something something simple, which is like today, like how how well do agents uh, how well are they able to produce infrastructure as code and kind of help guide you, just like they create the regular code on it, manage and sort of create that sort of deployment side of it?
00:04:37.360 --> 00:04:40.480
So over the past six months specifically, they've got a lot better.
00:04:40.639 --> 00:04:48.399
They really have, okay, as the evolution of models has continued and has just got better across the board, for infrastructure as code, they have got a lot better.
00:04:48.720 --> 00:04:53.439
A lot of the pain that agents find with infrastructure as code is it's very opinionated code.
00:04:54.319 --> 00:05:16.079
It's very uh refactored in a way that suits like taste and and presentation of an individual developer, an individual team, and how modules sit on top of each other and how abstracted each module is, and it's modules upon modules upon modules upon modules, and it makes it actually a wee bit more difficult for you know for agents to be able to create that level of sophistication within your org.
00:05:16.560 --> 00:05:21.920
If you asked an agent to produce base vanilla infrastructure as code, no problem.
00:05:22.319 --> 00:05:24.879
No problem, it'll do such a good job of it.
00:05:25.279 --> 00:05:42.959
But then, of course, we know that some of the pain with infrastructure as code comes when you try and have to refactor it, you know, and it's not because infrastructure as code is bad at it, it's because it's got a layer of state underneath it that becomes more difficult to move it around and be able to like, you know, change how it's represented.
00:05:43.199 --> 00:05:52.079
So as users bring in models to do it, they have to try and find ways for those models to interact with their existing systems.
00:05:52.240 --> 00:05:56.720
And that's why they're they're not getting as good results as they've actually been like hoping for.
00:05:57.439 --> 00:06:07.040
So it's super interesting to maybe dig a bit into that state, because I think on one hand, infrastructure as code indeed is a place where, you know, we're like often sort of seen like it's very sensitive as well.
00:06:07.120 --> 00:06:10.480
You know, there's like just every line has so much weight on it.
00:06:10.639 --> 00:06:15.439
Uh, there is uh like there's a lot in what is not present in the code.
00:06:15.519 --> 00:06:20.879
Like a lot of the choices are like you're leaning into the defaults, and so that requires maybe an understanding of that default.
00:06:21.199 --> 00:06:23.759
So it's interesting that I guess that kind of represents that state, right?
00:06:23.839 --> 00:06:38.800
Like maybe echoing back, you're saying, okay, like maybe the first sort of finicky, don't make small mistakes, is one piece of it, but then beyond that, ordering is important uh because it's managing state behind the scenes and um kind of understanding what the current state is.
00:06:39.040 --> 00:06:44.560
And but but even from that, there's a lot of operations that like we we know how agents interact, right?
00:06:44.639 --> 00:06:51.120
They try, they find a pattern, they build upon that pattern again and again in order to like get us to the outcome that we care about.
00:06:51.360 --> 00:06:53.600
Sometimes infrastructure is code that's difficult, right?
00:06:53.680 --> 00:06:57.199
Because when they find a pattern and they apply it, something's out there.
00:06:57.360 --> 00:06:58.720
Something's like an actual resource.
00:06:58.879 --> 00:07:01.199
You can't sometimes you can't just change it.
00:07:01.360 --> 00:07:01.519
Right.
00:07:01.600 --> 00:07:09.920
You know, it's a it's the equivalent of like taking it down, recreating it, and there's like an entire feedback loop there that becomes really slow in that infrastructure as code loop.
00:07:10.000 --> 00:07:28.160
So, you know, if we're hoping that an agent is gonna go from idea to like a an entire Terraform script or Pulumi script or whatever infrastructure is code script that you want, and you can apply that first run, you're gonna be probably a little mistaken that that's gonna be first time, you know what I mean?
00:07:28.560 --> 00:07:28.879
Interesting.
00:07:29.120 --> 00:07:37.519
How much of that is is like a harness problem uh of just sort of I guess informing the agents uh at the right time?
00:07:37.759 --> 00:07:47.759
Because I I mean I suspect the reason that they got better uh over these last six months are part is just the reasoning capabilities of the model, and probably a part of it is just sort of the data feedback loop, right?
00:07:47.839 --> 00:07:52.959
Which is as more people started doing the work, then the labs and the models get better.
00:07:53.360 --> 00:07:57.759
But you know, how so how much of it is kind of a model problem versus a harness problem?
00:07:58.399 --> 00:08:00.240
I mean hugely.
00:08:00.480 --> 00:08:08.560
Like the amount of the number of people I know who write like vanilla terraform or vanilla plume is like really low.
00:08:08.720 --> 00:08:08.879
Yeah.
00:08:09.120 --> 00:08:20.639
They're usually using a tool that sits on top that they can like scaffold it, that they can like interact with it, they can have their level of abstractions, which sort of outputs the terraform or outputs the infrastructure with code itself.
00:08:20.720 --> 00:08:26.480
So of the same complexity, even for sort of humans, you know, that sort of like error prone nature of infrastructure as code.
00:08:26.800 --> 00:08:44.240
So, you know, people are used to that level of abstraction, and then models coming in in the mix, it sort of like adds another level of, you know, you kind of have to take almost your entire harness, that harness that you created previously, and then recreate it as
a new harness for models to be able to do it.
00:08:44.399 --> 00:08:45.440
And you can totally do it.
00:08:45.519 --> 00:08:48.720
Like, you know, Pulumi have done this really well with Pulumi Neo.
00:08:48.799 --> 00:09:00.080
They've done a fantastic job with Neo, and it's given really good results about it, but it was really based on, you know, if people were actually like using Pulumi itself and Pulumi Automation API to be able to do it.
00:09:00.320 --> 00:09:05.759
And like the in in this context, the the job of the harness is primarily to convey state.
00:09:05.919 --> 00:09:14.639
It's to it's to say, look, this is my representation, maybe that sort of digital twin or like a flavor of that, a little bit of this is the state of my production.
00:09:14.720 --> 00:09:22.000
And so you have to assess whatever piece of code that you are conveying in uh to kind of inform the agent at the right time.
00:09:22.320 --> 00:09:27.679
And that and that you know, plan apply loop is it it's slow for humans.
00:09:27.840 --> 00:09:28.000
Yeah.
00:09:28.240 --> 00:09:33.919
So the fact that it's slow for humans means it's even slower for agents than you know anything else.
00:09:34.080 --> 00:09:39.600
And and we're still in this era where people are not sure, do we give agents access?
00:09:39.759 --> 00:09:41.039
Do we not give agents access?
00:09:41.200 --> 00:09:43.120
Should they have access to run the terraform?
00:09:43.279 --> 00:09:46.480
And if they don't and they apply it, then you're gonna have to meet proxy.
00:09:46.960 --> 00:09:48.480
You know, that result back and forward.
00:09:48.639 --> 00:09:53.919
It's just it's just a combination of paper cuts that just make it more and more difficult to be able to do.
00:09:54.320 --> 00:10:00.559
So maybe like getting a little bit more controversial, do we in do we even need infrastructure as code?
00:10:00.639 --> 00:10:07.120
I mean, I think so now you're you're kind of in this mode in which uh let's imagine you know you you have some form of map of your state.
00:10:07.200 --> 00:10:09.679
So I don't think you can get away uh from that, right?
00:10:09.759 --> 00:10:12.559
Like you need to kind of understand a little bit of what's deployed in production.
00:10:12.720 --> 00:10:24.080
But once you have that, I I think we we kind of created infrastructure as code iteratively because you know, people like because human labor was expensive, you know, this expensive and error prone.
00:10:24.159 --> 00:10:28.480
You'd sort of SSH into the machine and modify a bunch of resources and come back on it.
00:10:28.559 --> 00:10:34.320
And but you know, now kind of human labor is not human labor, it's AI labor, is cheap.
00:10:34.480 --> 00:10:43.279
Uh and you can get fairly you know bright ones, and you can you can still create a harness that will sort of nudge them to say, no, no, no, this resource is sacred.
00:10:43.519 --> 00:10:51.840
What is the value that we get out of that sort of intermediate
kind of representation of like infrastructure as code in Palumi or in Terraform or in whatever others?
00:10:52.399 --> 00:10:53.679
Like for me, I wouldn't choose.
00:10:53.759 --> 00:10:56.799
Like for me, I'll never write another line of infrastructure as code.
00:10:57.200 --> 00:11:05.600
Okay, I like we use Swamp to manage our own infrastructure, and we've actually changed production infrastructure four times in the last six months.
00:11:05.840 --> 00:11:06.080
Okay?
00:11:06.399 --> 00:11:11.840
And I mean like live migrations, so fast, so easy, because of exactly what you said.
00:11:12.000 --> 00:11:21.759
You have a map of what your existing system is, and you then have an idea of what the intent of where the new map needs to be, and you can actually use an agent to go from A to B.
00:11:21.840 --> 00:11:24.639
And it's incredibly easy, it's incredibly straightforward.
00:11:24.799 --> 00:11:33.279
It knows the outcome, it knows what it has to get to, it knows where it has to go, and it has all it is almost doing is translating A to B in order to get there.
00:11:33.440 --> 00:11:46.159
Now, to do that in infrastructure as code, to do a cut like an entire cloud provider migration in infrastructure as code is painful, it's really painful because there's many layers of abstraction in infrastructure as code, right?
00:11:46.240 --> 00:11:53.360
So you have the SDK of the cloud provider, which you have to know, then you have the Terraform equivalent or the Pulumi equivalent abstraction on top.
00:11:53.440 --> 00:11:56.879
Um, like we've referred to that in the past as the 200% problem, right?
00:11:56.960 --> 00:12:00.000
You kind of need to know both layers to be able to interact with it.
00:12:00.240 --> 00:12:01.279
Agents don't care.
00:12:01.519 --> 00:12:03.200
Agents do not care about that.
00:12:03.279 --> 00:12:07.440
They're like, your cloud is here, the new cloud is here, let's do the migration.
00:12:07.679 --> 00:12:11.279
It's A to B, and it's it's done it really well live.
00:12:11.440 --> 00:12:16.320
And we've done this as we've grown and our system has grown, we've gone from railway.
00:12:16.480 --> 00:12:41.440
Railway to DigitalOcean app uh platform, we've gone to DigitalOcean um uh Kubernetes, and now we've actually gone to like uh GCP, and every time it's been like completely flawless and completely seamless, and that includes full data migration, like that's click house servers, that's you know, entire buckets, and it's just doing it, and it's it's it's kind of amazing to watch because a year ago we wouldn't have been in that situation.
00:12:42.000 --> 00:12:51.360
So fascinating and and super interesting, and I can see because these are again, these they're sort of like human intelligence uh heavy uh exercises, right?
00:12:51.519 --> 00:12:53.120
You know, which you need to build those on it.
00:12:53.279 --> 00:12:55.679
What about the the kind of the systems operations?
00:12:55.840 --> 00:13:02.159
Like, okay, that is like for migrations that they're complicated, but then there's also the I just committed a new piece of code.
00:13:02.399 --> 00:13:13.919
Does it make sense for an agent to deploy that for me, or should that be uh like some whatever Terraform piece that would we are AI first, like we are literally AI first.
00:13:14.159 --> 00:13:18.799
We don't do any manual configuration of any systems, any servers, anything.
00:13:18.960 --> 00:13:26.320
At this moment in time, when a pull request actually when an issue is picked up, our software factory goes the whole way through.
00:13:26.399 --> 00:13:33.279
And not only does it run the software factory, it runs the test factory, it runs a release factory, it runs the entire thing the whole way through to production.
00:13:33.440 --> 00:13:41.360
And that's our agents at different levels, different agent perspectives and personas actually doing the right pieces in the factory to be able to take care of it.
00:13:41.519 --> 00:13:52.399
So for us, it's like it it takes care of the deployments, it takes care of like the changes to the load balancers that's needed, it takes care of the deployments to like click house, it does the whole thing end-to-end.
00:13:53.039 --> 00:14:00.879
So um I mean it it's interesting, but I think like you and I'm very much on board with having kind of an agent on the outside.
00:14:01.039 --> 00:14:05.360
Um but I guess still kind of my question stands when it comes to the to the inside, right?
00:14:05.440 --> 00:14:08.080
Because like agents can do a lot of things, but like software is still useful.
00:14:08.240 --> 00:14:11.519
Like you you don't want, I mean, some software should go away, right?
00:14:11.600 --> 00:14:15.440
And should just be an agent that delivers the software, like the code is is useless.
00:14:15.600 --> 00:14:15.759
Yeah.
00:14:15.919 --> 00:14:19.039
Um in this case, like is is the code actually useless?
00:14:19.120 --> 00:14:26.960
Like when you when you take that piece of code that you've just merged into your main branch and you need to deploy that, like replace the binary of it.
00:14:27.360 --> 00:14:29.440
I I guess what would you use?
00:14:29.519 --> 00:14:43.200
Is it like literally the agent will sort of go off into the copy-paste, or is it like a uh or or or do you do you still use snippets of code and they're just like the agent is allowed to change them, but otherwise it is running a piece?
00:14:43.759 --> 00:14:49.600
So we would use Swamp for this, and what Swamp allows us to do is to create like reusable behaviors, right?
00:14:49.759 --> 00:15:00.480
Behaviors are like uh pieces of intent that are encoded with they have schemas and they have like a workflow and they have like a you know a specific data structure that is like adding gates to the process.
00:15:00.720 --> 00:15:05.600
So when something goes into our factory, it is running Swamp behaviors the whole way along.
00:15:05.679 --> 00:15:05.759
Right.
00:15:06.000 --> 00:15:14.559
So Swamp is actually controlling with the agent to be able to do all of the pieces of the puzzle, including taking servers offline, putting them back online.
00:15:14.799 --> 00:15:18.639
You know, it has the full access to do it the entire way through the pipeline.
00:15:18.720 --> 00:15:24.960
So for us, it's not that it's particularly code, it is an actual agent harness that is doing it the whole way.
00:15:25.279 --> 00:15:25.600
Got it.
00:15:25.679 --> 00:15:30.320
Okay, so interesting, and indeed we didn't dig too much into swamp on it, and people should check it out and and try it.
00:15:30.480 --> 00:15:33.279
But those are indeed kind of defined behaviors and they're ingentic.
00:15:34.159 --> 00:15:37.120
Maybe let's talk a little bit about the blast radius aspect of it.
00:15:37.200 --> 00:15:44.639
So, like another reason that operations is sort of like ops teams are afraid of uh of uh agents is that the blast radius is big.
00:15:44.720 --> 00:15:48.559
You make a mistake, you might take the system down, drop the database, do whatever you want.
00:15:48.639 --> 00:16:00.879
Uh, and so they kind of refer to these sort of layers of infrastructure automations as layers of higher sensitivity, where they want kind of a human approver uh to say, like, I don't want any statistics over here.
00:16:00.960 --> 00:16:13.519
You know, like I want uh I want an approval by a human, which I will say like is clearly uh at the best a short-term solution because you you like any any bottleneck, any human bottleneck will eventually go away.
00:16:13.840 --> 00:16:15.440
But so like how do you how do you think about that?
00:16:15.600 --> 00:16:21.200
Do you feel you know we have like with swamp or maybe with sort of other sort of like good scaffolding of the agent?
00:16:21.360 --> 00:16:31.679
You get sufficient determinism that people just need to get with the program, uh, or or is this more about the future, like okay, maybe today you should approve things because we're not quite there yet.
00:16:31.840 --> 00:16:35.279
But um it's just just know that you're not future-proofing.
00:16:35.519 --> 00:16:38.000
So that blast radius is incredibly important, right?
00:16:38.080 --> 00:16:46.879
Those guardrails, like those um real like systems that stop the agent going offline and like really having that deterministic behavior.
00:16:47.039 --> 00:16:54.399
It's like it's why we built what we built, because of the fact that we were and we seen other people just having that pain and that structure.
00:16:54.559 --> 00:17:06.000
So, like for us, it's very much encoding the exact workflow step, and the agent knows just to run that exact workflow step, and it doesn't have to think A to B in order, because it's probabilistic, right?
00:17:06.079 --> 00:17:08.960
It doesn't have to think how can I go from here to like production.
00:17:09.039 --> 00:17:17.759
It goes, I just need to run this exact like script, this exact piece of
code, this exact like um deployment of the API behavior, and it actually does it.
00:17:18.000 --> 00:17:23.440
The other interesting part with the operations world is people are worried about giving an agent credentials, right?
00:17:23.599 --> 00:17:27.599
We've seen CV Mageddon the last couple of weeks in October.
00:17:27.759 --> 00:17:33.519
It's been horrendous for so many people, and it's actually like made people worry even more, right?
00:17:33.599 --> 00:17:37.759
Because if you give an agent a key, then you know, whatever can happen.
00:17:37.839 --> 00:17:50.799
So it's it's like about making sure that like your agents don't just have raw access, don't just have environment variables in the system, they need to have the encoded vault structures, like you know, all of that needs to be wrapped away.
00:17:50.880 --> 00:18:00.319
So an agent doesn't even need to understand what the keys are, it needs to know that when it runs an automation that the system itself is going to be able to pick up the keys inside the automation.
00:18:00.480 --> 00:18:10.720
And if you can wrap those pieces away together, then you'll start to find that you know things become a little easier and a little less blast-radisy, and like you'll be able to control it more.
00:18:11.039 --> 00:18:13.759
Hey everyone, hope you're enjoying the episode so far.
00:18:14.079 --> 00:18:22.160
Our team is working really hard behind the scenes to bring you the best guests so we can have the most informative conversations about agentic development.
00:18:22.319 --> 00:18:27.759
Whether that's talking about the latest tools, the most efficient workflows, or defining best practices.
00:18:27.920 --> 00:18:31.519
But for whatever reason, many of you have yet to subscribe to the channel.
00:18:31.680 --> 00:18:38.480
If you're enjoying the podcast and want us to continue to bring you the very best content, please do us a favor and hit that subscribe button.
00:18:38.640 --> 00:18:46.000
It really does make a difference and lets us continue to improve the quality of our guests and build an even better product for you.
00:18:46.240 --> 00:18:48.000
Alright, back to the episode.
00:18:48.319 --> 00:18:49.119
So so interesting.
00:18:49.200 --> 00:18:55.759
So I guess kind of describing this and trying to kind of keep this from just being swamp, although clearly that should be sort of one of the solutions that assesses it.
00:18:55.839 --> 00:19:03.519
Uh, I think your point is when you're managing uh production right now, really what you should have is a system where it's sort of an agent on top.
00:19:03.599 --> 00:19:08.880
Uh and you do want that agent to be in a very well-managed harness.
00:19:08.960 --> 00:19:29.200
Uh, and that harness you know probably will use some pieces of code, maybe sometimes infrastructure's code, uh, and sometimes just whatever scripts or sort of you know program programmable code that we created that uh will will will basically apply determinism and contain access and such at different times.
00:19:29.359 --> 00:19:47.839
Uh but I think what's important is that the agent is at is is sort of on the top because then it can it can kind of resolve problems, it can adjust, it can adapt, and it will have tools, and maybe, you know, in some cases you'll say, look, I'm commandeering judgment from you, like over here, and that's the harness's job.
00:19:47.920 --> 00:19:55.039
It's like over here you're gonna run this tool because I did not give you the keys to be able to apply this, so you really can't do anything else.
00:19:55.279 --> 00:19:59.599
And like in the ops world, we have been riddled with toil for so many years.
00:19:59.759 --> 00:20:05.519
Just the mundane things that we hate doing, but it's the things that we have to do to keep our systems running.
00:20:05.839 --> 00:20:08.480
Agents are not bored by toil.
00:20:08.640 --> 00:20:13.279
Agents love toil, they will do those jobs again and again and again and again and again.
00:20:13.839 --> 00:20:18.880
And it is it's incredibly satisfying because when you tell it to do it and it does it again, it's amazing.
00:20:19.200 --> 00:20:23.279
So, as an ops person, now we can actually turn that to our advantage, right?
00:20:23.359 --> 00:20:37.839
We can exactly what you said, we can still maintain that control, but we can let the agent go and do all of the pieces for us, and we're the ones in control that say, yes, now is the time it goes into production, or this is what we have to do next.
00:20:38.000 --> 00:20:44.000
And if we can continue to hold that control, we're the ones who know the system more than anyone else.
00:20:44.240 --> 00:20:54.240
You can tell an agent, you can give an agent access to the system, but it doesn't have all of the like the detail, it doesn't have the entire systems thinking that goes with it, or the understanding of the system.
00:20:54.400 --> 00:21:02.799
So you still need to have an operator who is able to take control and take it over at any point and go, you know, we have to make sure that this doesn't go off the rails.
00:21:02.880 --> 00:21:03.920
And that's like a really important thing.
00:21:04.319 --> 00:21:07.920
Their job is to build basically the sort of these instructions to absolutely correct.
00:21:08.079 --> 00:21:22.319
So it's like almost becomes a like ops harness engineering, and you know, like you hear like terms like harness engineering, you hear terms like agentic engineering and stuff like that, but it's real like ops system engineering in AI to be able to control that in a lot of people.
00:21:22.559 --> 00:21:25.279
It's more about the choice of expertise that you are embedding into the harness.
00:21:25.759 --> 00:21:32.559
And like I I've been saying for like forever now, it's like people are like, oh, you know, agents are gonna take our job away and they're gonna do that.
00:21:32.720 --> 00:21:34.160
I actually think quite the opposite.
00:21:34.319 --> 00:21:47.920
If
you're like if you understand your system inside out, back to front, and you really like understand the pieces of it and the constraints of it, an agent is a force multiplier for what you can do, and you can actually achieve so more in your or so much more in your org.
00:21:48.240 --> 00:21:49.680
Yeah, fully, fully agree with that.
00:21:50.000 --> 00:21:51.920
So maybe one last question here.
00:21:52.160 --> 00:21:54.079
Uh taking it into software factories.
00:21:54.160 --> 00:21:59.359
Uh so actually, like a lot of what you described is very aligned to TESL's sort of software factory approach, right?
00:21:59.440 --> 00:22:03.839
We have A Tesla JSON file that defines skills.
00:22:04.000 --> 00:22:05.920
It's always the agents at the top.
00:22:06.000 --> 00:22:09.599
So like this behavior or whatever that may be is always encoded as a skill.
00:22:09.759 --> 00:22:11.200
I say skill, but really I mean plugin.
00:22:11.519 --> 00:22:14.000
It has like some hooks and it has some tools in it.
00:22:14.160 --> 00:22:17.119
It's just skill is the sort of the term that's been out there.
00:22:17.279 --> 00:22:22.559
And so you take that, you define an environment, suddenly, you know, a skill plus an environment is an action.
00:22:22.720 --> 00:22:26.480
You take an action, you add a trigger or a schedule or something like that.
00:22:26.559 --> 00:22:27.599
That's a type of trigger.
00:22:27.759 --> 00:22:30.240
You know, suddenly together you have a loop, you're sort of running a thing.
00:22:30.400 --> 00:22:34.960
And so it's very much, I guess we define our software factory in this sort of declarative way.
00:22:35.359 --> 00:22:56.880
And we sometimes refer to it, I don't know if we have at the moment like the ops credentials, the sort of really code design, but we refer to this as infrastructure as code for software factories and the definitions, contrasting it from uh what feels like sometimes software factories look like, which is the ops days of old, which is just a jumble of scripts uh that you might kind of cobble together and they will manage your system.
00:22:57.039 --> 00:23:03.440
They're kind of doing similar things, you know, when you build it just naively with the uh with the agent.
00:23:04.000 --> 00:23:09.359
Like how how do you think about infrastructure as code uh for software factories?
00:23:09.599 --> 00:23:13.039
Like if you think about this, this is basically a new layer of ops, right?
00:23:13.279 --> 00:23:21.279
It's a new um uh a new type of system that you are deploying that is operating on it.
00:23:21.680 --> 00:23:27.839
I guess how do you think about the role of infrastructure as code for managing software factories to begin with, right?
00:23:27.920 --> 00:23:31.200
Or or maybe it's not infrastructure as code, because we just talked about that.
00:23:31.279 --> 00:23:34.400
Uh I don't know what what is your term really for these behaviors.
00:23:34.799 --> 00:23:36.000
Like we call them behaviors, yeah.
00:23:36.319 --> 00:23:48.960
So like for me, there's always been this disconnect between software teams and ops teams, and they kind of like keep a you know, separation of duties and how like these pieces like are far apart.
00:23:49.359 --> 00:24:01.279
In the world of agents now, the closer that these scripts, these infrastructure management scripts that come closer to the software, like it's gonna give a much better like understanding of what's happening.
00:24:01.519 --> 00:24:12.559
So embedding them as part of that flow, that factory, and like we hear the term software factory, and it's kind of like it's almost misleading because it's not just about software.
00:24:12.640 --> 00:24:21.039
Um it's it's actually more about the encompassing software, it's not just the actual code itself, it's about how the code goes into production and how it runs in production, right?
00:24:21.200 --> 00:24:36.799
And so the closer like these two like systems come together and the more embedded they are, the more that that factory is actually you know fine-tuned to be able to do it because you don't have to have that handoff between teams that we've used to have for a long period of time.
00:24:36.880 --> 00:24:39.119
Yeah, because the feedback loop is so short.
00:24:39.279 --> 00:24:57.039
Like the agent should actually be able to understand how it can determine and how it can change the infrastructure's code or whatever it is that it's actually using to deploy and manage the infrastructure to be able to do it, rather than saying, right, I'm finished here now, here's an entire set of context that you need to pass on to the next team.
00:24:57.200 --> 00:25:04.720
Because as soon as you start passing that context around as a layer that's not controlled, it becomes really difficult to maintain.
00:25:04.799 --> 00:25:06.960
And there's definitely going to be something lost in translation.
00:25:07.279 --> 00:25:16.640
It's interesting, it's like it makes me smile a little bit thinking that uh you know, to achieve true DevOps and kind of mix the two teams, we needed to take the human out of the equation.
00:25:17.279 --> 00:25:27.119
To sort of get the agents to do actual dev and ops, and then you will actually have uh the two kind of parts, and frankly, much more of software development in the same case.
00:25:27.519 --> 00:25:38.559
I blogged like maybe two months ago about how the teams that are most efficient with agents and delivering the best with agents at this time are the teams who are really far down their journey of continuous delivery.
00:25:38.799 --> 00:25:41.599
And exactly what you're saying with continuous delivery, right?
00:25:42.000 --> 00:25:56.079
Those teams who had to put those structures in place, those processes, those workflows, those ways of working, like for delivering their systems, you apply that same process now to like engineering an AI, and you just get such a smooth transition.
00:25:56.160 --> 00:25:57.359
So, exactly what
you're saying.
00:25:57.680 --> 00:26:05.599
Yeah, so I I guess that's uh that's a good point to maybe end a little bit the uh the um uh the podcast part of the uh conversation.
00:26:05.759 --> 00:26:09.440
And we're actually gonna go on to show your talk over here.
00:26:09.599 --> 00:26:12.799
So, can I ask you to just explain in a word, you know, what is it?
00:26:12.880 --> 00:26:16.160
It's called uh from uh from page to dossier, right?
00:26:16.319 --> 00:26:20.880
Uh tell us a little bit about what it is, you know, to entice people to stick around and listen to the next up.
00:26:21.279 --> 00:26:28.960
So we have this idea of lights on, lights off factories, um, where you know an agent can be, you know, as part of it or a human can be part of it, what's happening.
00:26:29.039 --> 00:26:43.279
And like I've I've been, for a while now, I've been professing to people that you know having a human as the human touch and like understanding what's going on in the system, be able to make decisions is really important, but that you can actually put an agent in charge of doing the research.
00:26:43.599 --> 00:26:47.359
Ops people have been woken up at two o'clock in the morning forever.
00:26:47.599 --> 00:26:52.720
First thing they have to do when they wake up is they have to go and run this runbook, that run book, get this data, that data.
00:26:53.200 --> 00:26:55.039
An agent can do all of that for us.
00:26:55.519 --> 00:27:02.160
It can present us with a dossier of exactly what's happening in the system, and then we as humans we can take over and we can understand where it is.
00:27:02.240 --> 00:27:08.720
So it's actually a sneak peek into how you would actually build a factory like that using smaller models.
00:27:08.799 --> 00:27:11.759
I've been using JEV as an example about how we can actually control.
00:27:12.400 --> 00:27:13.440
These are just decisions, yeah.
00:27:13.759 --> 00:27:20.079
Just decision engines, exactly, about how we can actually control is this a correct alert, is it not, how it routes through the system.
00:27:20.240 --> 00:27:25.920
So it's a it's just a different way to thinking about how ops can actually be empowered to actually build systems.
00:27:26.240 --> 00:27:26.960
Yeah, got it.
00:27:27.039 --> 00:27:29.440
And I love the play on words from a page to a dossier.
00:27:29.599 --> 00:27:30.880
So love that on it.
00:27:30.960 --> 00:27:32.240
So definitely tune in for that.
00:27:32.400 --> 00:27:35.039
And with that, let's switch over to the talk.
00:27:35.440 --> 00:27:36.000
Good morning.
00:27:36.160 --> 00:27:36.880
Everyone hear me?
00:27:37.039 --> 00:27:37.519
Beautiful.
00:27:37.680 --> 00:27:42.240
Um okay, so this is a one-hour talk that's gonna be condensed into 30 minutes.
00:27:42.400 --> 00:27:46.000
So this could go a number of different ways, but we'll get through a lot of different things here.
00:27:46.160 --> 00:27:52.240
Uh, this talk is called From Page to Dossier, and it's about using agents to your advantage when you're an operator.
00:27:52.400 --> 00:27:57.279
Any operators, systems admins, SREs, platform engineers?
00:27:57.440 --> 00:27:59.359
I'm sure we all dread this.
00:27:59.519 --> 00:27:59.680
Okay?
00:28:00.480 --> 00:28:07.440
It happens to me so often, I've been paged so many different times, and it's incredibly frustrating.
00:28:07.599 --> 00:28:17.119
And when we wake up, we have to go through a checklist of things that happens in order to achieve, like, and figure out what's going on.
00:28:17.440 --> 00:28:20.319
So we like have to wake up, open your eyes.
00:28:20.400 --> 00:28:21.839
Like it's just it's just painful.
00:28:21.920 --> 00:28:29.279
You gotta look at your logs, you gotta try and determine what's going on, you gotta look at dependencies, maybe it's an upstream issue that you're having.
00:28:29.440 --> 00:28:34.160
It's just a set of paper cuts that just make things pain.
00:28:34.240 --> 00:28:44.559
Okay, that 30 that first 30 minutes of context gathering is incredibly important to help you make a decision that what you do next.
00:28:44.720 --> 00:28:44.960
Okay.
00:28:45.440 --> 00:29:02.559
I'm sure operators, software engineers, people who have been on call before have been in the situation where you're trying to determine what the hell is going on, there's multiple pages happening, there's multiple pieces of information going on, and you try and figure out what is happening.
00:29:02.720 --> 00:29:22.880
Now it's not we we have to context gather really well, not because we're bad engineers, not because we're bad platform people, not because we're bad operations people, because at three o'clock in the morning or five in the morning, whenever you're paged out of hours, the last thing you're thinking of is what are the steps that you actually have to do.
00:29:23.359 --> 00:29:26.960
What if that 30 minutes context gathering was actually two?
00:29:27.440 --> 00:29:28.480
What if it was zero?
00:29:28.640 --> 00:29:30.960
What if something had already done it for you?
00:29:31.759 --> 00:29:40.240
What we're in in this period of time in ops world when we have that level of checklist of do this, do this, do this, do this, and do this, this is called toil.
00:29:40.400 --> 00:29:40.640
Okay?
00:29:41.039 --> 00:29:55.200
It's manual, repetitive, no value, like that scales, you know, it's just as it's just a segment of pieces that we have to do to check the boxes to help us understand what's next.
00:29:55.279 --> 00:29:55.519
Okay.
00:29:56.079 --> 00:30:02.480
One thing that's really important is an agent can do every single one of these steps.
00:30:02.720 --> 00:30:03.039
Okay.
00:30:05.119 --> 00:30:07.200
Agents love toil.
00:30:07.519 --> 00:30:07.839
Okay.
00:30:08.400 --> 00:30:15.200
Agents don't care if you ask them to do the same task 500 times in a row.
00:30:15.440 --> 00:30:24.799
If I ask one of my engineers to do the same task three times in a row, they will grumble
and moan and like try and cut corners and they will make mistakes.
00:30:24.960 --> 00:30:30.319
Okay, it's not because they're bad engineers, it's we are not programmed in that way.
00:30:30.640 --> 00:30:37.200
We are programmed to make things as fast and efficient as possible, and we will try and undercut the system to do that.
00:30:37.359 --> 00:30:40.319
I have worked in environments where there have been run books.
00:30:40.480 --> 00:30:43.839
Okay, it still makes me shudder a little bit to this day.
00:30:43.920 --> 00:30:47.440
And those run books are sets of guides that you have to follow.
00:30:47.599 --> 00:30:53.839
And I know that when those run books were written, they're immediately out of date because they're very rarely kept up to date.
00:30:53.920 --> 00:30:56.240
And I know that I can skip specific steps.
00:30:56.400 --> 00:31:06.559
I know I don't need to do step six because I'm not configuring that part of the system, and I can jump straight to step seven, and then I get to step 23, I'm like, ah shit, I meant to do step six.
00:31:06.720 --> 00:31:09.839
So, you know, it's very much a case of agents can wrap this away.
00:31:10.000 --> 00:31:16.480
Um agents never push back on us to say, this is boring, I told you not to do it.
00:31:16.640 --> 00:31:17.680
Well, Claude sometimes does.
00:31:17.839 --> 00:31:23.200
Like Claude sometimes tells us that, like, and it also tells us that we're awesome, and it also, yeah, all those different things.
00:31:23.279 --> 00:31:26.400
But of course, it's that's just the fluff that comes with it.
00:31:26.480 --> 00:31:28.880
That's not the agent itself, that's just what what happens.
00:31:29.119 --> 00:31:32.160
Agent will do the boring part the same way every time.
00:31:32.319 --> 00:31:34.400
Okay, it is sh sure.
00:31:34.559 --> 00:31:43.680
How it actually does it internally is a probabilistic system, but if you tell it to do step, A, step, B, step, C, and step C, it will do step, A, step, B, step, C, and step C.
00:31:43.839 --> 00:31:48.079
How it does each step could be slightly different just because of the nature of an LLM.
00:31:48.799 --> 00:31:59.599
We are very much in a case of um, you know, we can start to piece tools together to allow us to take away this toil.
00:31:59.759 --> 00:32:03.440
Okay, vendors are already shipping tools like this.
00:32:03.519 --> 00:32:10.640
Okay, Honeycomb have Canvas, Dash Zero are like an up-and-coming uh vendor in the monitoring space.
00:32:10.720 --> 00:32:13.279
They're not up and coming, they're like an incredibly established company.
00:32:13.359 --> 00:32:21.759
But like they're you know, they're they're spending a bunch of time rethinking how you can actually embed operations practices with AI in order to do it.
00:32:21.839 --> 00:32:29.680
And you know, these vendors know their platform better than anyone else, and they will create incredible tools that will allow you to do this.
00:32:29.920 --> 00:32:40.880
You know, every piece of the run book, if you have a third-party vendor, they will give you the tools to do it, and you can create the process that brings it together.
00:32:41.680 --> 00:32:54.559
Now when you have a run book, when you have like deterministic steps that you have to go through in order to help determine what's happening, you can create a factory, okay?
00:32:54.799 --> 00:32:57.680
Please excuse that I use the term factory, okay?
00:32:57.920 --> 00:32:58.720
One, I hate it.
00:32:58.799 --> 00:33:00.880
I absolutely hate it with a passion.
00:33:01.039 --> 00:33:06.480
Everything seems to have become a factory, and it's just a crap way to describe it.
00:33:06.559 --> 00:33:08.799
Okay, it's like, I mean, we're not in manufacturing.
00:33:08.880 --> 00:33:17.200
It is sure, it is a process of pushing inputs into a system, the system acts upon them, and you get things that come out the other side.
00:33:17.359 --> 00:33:22.880
If anybody else gives me a better term for it, I promise you I will change my slides because I hate the term factory.
00:33:23.039 --> 00:33:31.119
Okay, but at each level and each step in the factory, you can actually automate and you can bring it together.
00:33:31.599 --> 00:33:33.200
Now, if you think about it, right?
00:33:33.359 --> 00:33:46.799
We talked about you need to talk to logs, you need to talk to upstream dependencies, you need to check your deployments, you need to check this system and that system to help you pick up a picture of what's actually happening when a page goes on.
00:33:47.039 --> 00:33:50.160
If you think about it, all of these tools are all there.
00:33:50.319 --> 00:33:59.759
Okay, so you have metrics tools, you have traces, you have logs, you have your deploy pipeline, you have your config changes, you have your upstream status pages, you have your chat history, your your run books.
00:34:00.079 --> 00:34:02.640
All of that information is available to you.
00:34:02.799 --> 00:34:08.880
So what you can do is you can start to build a process that connects them and brings them together.
00:34:09.920 --> 00:34:17.039
There is terminology around factories, okay, where you have a lights out factory or a lights on factory.
00:34:17.199 --> 00:34:25.119
A lights on factory is a human is involved at multiple steps along the way and is helping the factory decide what it's doing next.
00:34:25.280 --> 00:34:36.239
As the human is keeping control of what's happening, and you largely, largely, not always, do this when there is work that you need human judgment and human taste for what you're actually doing.
00:34:36.320 --> 00:34:45.920
Okay, but there is also the concept of a lights out factory where an agent will just rip through a set of systems and it will give you the output the other side.
00:34:46.239 --> 00:34:49.599
We created a lights out factory for our alerts.
00:34:49.920 --> 00:34:50.239
Okay?
00:34:50.800 --> 00:34:56.639
Now, please take this that I'm not telling you that you have
to go and put together a lights out factory for your alert.
00:34:56.880 --> 00:34:58.320
You absolutely do not.
00:34:58.559 --> 00:35:02.559
You build the systems that work within your organization and your environment.
00:35:02.800 --> 00:35:09.360
We are just sharing a little experience of what we've actually built and the tooling that we've put together to allow people to be able to do it.
00:35:09.519 --> 00:35:11.280
I'm gonna stop talking about the abstract here.
00:35:11.360 --> 00:35:14.960
I'm gonna actually show you the details of what is going on.
00:35:15.280 --> 00:35:18.559
Okay, so we start with alerts.
00:35:18.880 --> 00:35:28.079
An alert comes in from our system, it either comes from GCP, it comes from a tool called Axiom, which is another metrics tool, or it comes from our Grafana system.
00:35:28.239 --> 00:35:34.000
Okay, every alert is pushed as a webhook into our platform that we build and maintain.
00:35:34.559 --> 00:35:39.199
After that, we actually put that alert through a verification system.
00:35:39.360 --> 00:35:52.960
Okay, I don't want you attacking my verification system or my alerting system, so I verify that it is a correct alert with the correct HMAC signature that comes in that matches the parts of the system that we actually know.
00:35:53.199 --> 00:35:56.639
When it is designed that it is okay, it continues.
00:35:57.119 --> 00:36:01.599
If the naughty people on the internet try and attack things, we will try.
00:36:01.760 --> 00:36:07.920
I'm not saying we will succeed, we will try our best and try and determine that it that is hooked off.
00:36:08.239 --> 00:36:10.960
When we go in, we then start to de-dupe things.
00:36:11.119 --> 00:36:12.960
Have we seen this alert before?
00:36:13.199 --> 00:36:31.039
We take a hash ID of what actually comes in, and if there are multiple alerts that come in of the same type, because we all get alert storms, 500 alerts fire for the same thing, and as you're working on it, PagerDuty or whatever tool you use is continuing to throw other alerts in the background.
00:36:31.119 --> 00:36:34.800
So we actually dedupe it and we hash it at that point to help us understand.
00:36:34.960 --> 00:36:36.239
We then correlate it.
00:36:36.400 --> 00:36:36.719
Okay.
00:36:37.599 --> 00:36:39.760
Is this an alert that we've seen before?
00:36:40.000 --> 00:36:44.400
Is this something that has recently resolved we have to reopen it?
00:36:44.480 --> 00:36:50.719
Is there actually correlation by what is happening in the system that allows us to drive it forward from there?
00:36:50.880 --> 00:36:54.000
And if there are, we close things out.
00:36:54.159 --> 00:36:54.480
Right?
00:36:54.559 --> 00:36:58.880
There is like resolutions that we can say, this one is good, we can close it.
00:36:59.039 --> 00:37:02.000
This one is a jupe of what's happening, we can close it.
00:37:02.239 --> 00:37:07.280
This one is related to another system that is currently being worked on, that is an open issue, we can close it.
00:37:07.440 --> 00:37:09.119
Notice what I say here.
00:37:09.519 --> 00:37:11.599
It is closed by code.
00:37:12.079 --> 00:37:14.320
It is not closed by an agent.
00:37:14.559 --> 00:37:20.480
An agent is never able to make a decision about when an alert should be closed.
00:37:20.719 --> 00:37:24.239
We have written deterministic systems that allow us to do that.
00:37:24.400 --> 00:37:24.719
Okay.
00:37:25.440 --> 00:37:28.239
I have not written a deterministic system to do it.
00:37:28.400 --> 00:37:32.079
I have instructed an agent to write a deterministic piece of code that can actually do it.
00:37:32.159 --> 00:37:39.360
Okay, it is a it is a piece of code that has logic, and that logic is attached to business rules within the org that allow us to do it.
00:37:39.599 --> 00:37:41.760
From there, we go into a screen.
00:37:41.920 --> 00:37:43.440
Okay, is it actionable?
00:37:43.599 --> 00:37:44.719
Is it not actionable?
00:37:44.880 --> 00:37:48.800
You know, and like this is where you can start to mix in a little more.
00:37:48.960 --> 00:37:49.920
We use JEV.
00:37:50.239 --> 00:37:51.440
Anyone heard of JEV?
00:37:51.760 --> 00:37:55.199
It's a small language model, like the decision-based model.
00:37:55.440 --> 00:38:00.960
Um we got in the early beta in September, they gave me$5 credits.
00:38:01.360 --> 00:38:07.840
I heard the funny meme on the internet saying when I die, I'll be able to hand the rest of my JEV credits to my children.
00:38:08.079 --> 00:38:13.519
Like I cannot spend my JEV credits because it is such a cheap model to run and it's incredible.
00:38:13.760 --> 00:38:19.280
We actually use JEV a lot, and I'll talk about it a little bit later about what it actually did for us.
00:38:19.440 --> 00:38:28.079
But that that initial understanding of it's actionable, dismissed, or recorded or whatever is so good in a decision tree like this.
00:38:28.800 --> 00:38:33.199
Then once it passes the JEV test, it goes into a real triage loop.
00:38:33.360 --> 00:38:35.920
Triage loop then is actually our deterministic workflow.
00:38:36.079 --> 00:38:45.679
Talk to the run books, talk to the upstream systems, check the logs, check the dependencies, all of these different things, and that is a workflow itself that runs and goes through an agent to do it.
00:38:45.760 --> 00:38:54.079
And then we validate, we apply the triage of it, we actually check the actionable outcomes, we understand what's happening, and lastly, we get a ping.
00:38:54.159 --> 00:39:04.000
Okay, we ping in multiple places, we ping on Discord because we use Discord for everything, not Slack, because we're trying to be uh hipster and cool, trying to be done with like the gaming kids.
00:39:04.159 --> 00:39:08.079
And uh, secondly, we use like of course like pings like to people's phone.
00:39:08.320 --> 00:39:14.000
Now, the signal only reaches a person on call when it has traversed the entire tree.
00:39:14.159 --> 00:39:18.480
Okay, that does take approximately five minutes.
00:39:18.719 --> 00:39:27.360
Okay, this is actually a real alert that fired at 2153 UTC on Halloween night.
00:39:27.599 --> 00:39:39.280
Okay, and it was a certificate renewal was overdue in our Grafana system, and immediately we could see the correlation and the cause and effect of exactly what happened.
00:39:40.239 --> 00:39:45.440
What's actually incredibly funny is that was that did ping somebody.
00:39:45.920 --> 00:39:50.719
Same alert, because we haven't fixed it, it's like if it was a small system, it doesn't really matter too much.
00:39:50.880 --> 00:39:52.400
Same alert fired this morning.
00:39:52.559 --> 00:39:57.840
Okay, I checked on it specifically before my talk at 1148 UTC.
00:39:58.320 --> 00:40:02.960
Okay, um, that's like five hours ago, six hours ago, whatever it is.
00:40:03.119 --> 00:40:07.440
And we can actually see it says yes, it is the same two certificates.
00:40:07.599 --> 00:40:11.360
It's now 4.6 days past renewal, not 24 hours.
00:40:11.679 --> 00:40:14.480
The um the ingest basically says it's a duplicate.
00:40:14.639 --> 00:40:22.480
Claude basically says don't worry about it, attach it to the open incident, and then we basically say, no new incident, no new page, don't wake anybody up.
00:40:22.639 --> 00:40:33.039
Okay, now that's fascinating because what it actually did was it created a very specific set of context that it passed back to the agent.
00:40:33.199 --> 00:40:36.239
By the way, all of these slides I will publish and you can have access to them, okay?
00:40:36.320 --> 00:40:39.599
So don't be worrying about trying to like read the like the minute details.
00:40:39.840 --> 00:40:43.119
Basically, it had all of the context that was needed to be able to do this.
00:40:43.199 --> 00:40:48.400
It had the category, the title, the summary, the rationale, the recommended actions, all of these different pieces.
00:40:48.639 --> 00:40:55.920
And what happens now is we've built this loop, we've built this deterministic system that allows us to be able to do this.
00:40:56.079 --> 00:41:03.599
I showed you the top half of the factory, the lights out factory, but then there's also a bottom half of the factory.
00:41:03.760 --> 00:41:08.320
The bottom half of the factory is when something arrives in Discord, we as operators can query it.
00:41:08.400 --> 00:41:09.599
We can ask questions.
00:41:09.760 --> 00:41:14.480
And when we ask questions, that introduces new context into the system.
00:41:14.800 --> 00:41:23.119
It goes off, it queries, it prepares, it goes and does another round of analysis, and then it attaches the answers to what's happening.
00:41:23.280 --> 00:41:35.039
And we end up with a system that looks a little like this, where we have a trigger, where we have a concurrency model, where we have an ingestion engine, where we have a triage, and then we have apply triage.
00:41:35.440 --> 00:41:50.079
So we've kind of like sat down and thought about what it is we do as operators and how we can actually change that into a system where we are the last people in the queue that we are then make the decision about how we remediate a problem.
00:41:51.199 --> 00:41:54.079
Don't let that LLM drive at 3 a.m., okay?
00:41:54.320 --> 00:41:59.679
Because you're gonna go approve, approve, approve, shit.
00:41:59.840 --> 00:42:00.960
That's what's gonna happen.
00:42:01.119 --> 00:42:01.360
Okay?
00:42:01.920 --> 00:42:05.039
But we only built the part that's ours, okay?
00:42:05.199 --> 00:42:18.320
When I say that, it's like every step in our system is actually a behavior, it is a model, it is a workflow that comes together to allow us to bring and stitch this behavior together using our own toolswamp.
00:42:18.400 --> 00:42:18.639
Okay.
00:42:19.039 --> 00:42:29.920
Now you get a schema that looks like this, because it's type schemas, and you do things like this where you actually have like steps in the graph, looks very similar to a GitHub action.
00:42:30.000 --> 00:42:31.840
Okay, if you've ever seen a GitHub action.
00:42:32.000 --> 00:42:35.119
Okay, and then as I told you earlier, we use Jev.
00:42:35.440 --> 00:42:38.000
Now, Jev was a new addition into our pipeline.
00:42:38.079 --> 00:42:41.440
Okay, we were doing all of this initial triage with Claude.
00:42:41.599 --> 00:42:52.000
Okay, we're anthropic users, other models are available, but we are just an anthropic place where we try and just use Opus and like uh Sonnet for like what's happening.
00:42:52.239 --> 00:43:09.119
When we swapped in to using Jev, we actually got 91% cost improvement in our system because one claude call is the same cost as approximately 1365 GEV calls.
00:43:09.280 --> 00:43:17.679
Okay, now that's kind of fascinating because now you can see why JEV is so fast, so small, so like cheap to run.
00:43:17.840 --> 00:43:22.000
But what it is, it was very much a case of the system became faster.
00:43:22.159 --> 00:43:29.519
We didn't need to use large frontier models to actually do um very simple basic classification of what happened.
00:43:29.599 --> 00:43:37.760
We then deferred to using those large frontier models and intelligent models after like the classification understand that happened.
00:43:37.920 --> 00:43:39.920
And it was a massive reduction in our costs.
00:43:40.000 --> 00:43:44.400
And of course, we're in the era now where we can really control those costs and understand.
00:43:44.480 --> 00:43:50.400
But effectively, what it's doing under the hood is JEV's like running the equivalent of like a query.
00:43:50.639 --> 00:43:56.880
Okay, it's running the equivalent of a query, and then it's basically giving us a probability of is this something we need to do?
00:43:57.039 --> 00:43:58.960
Is this something that we've seen before?
00:43:59.199 --> 00:43:59.519
Do we
00:44:00.159 --> 00:44:00.480
Close it.
00:44:00.719 --> 00:44:04.239
It's like a very simple yesno style like decision tree.
00:44:04.400 --> 00:44:08.320
But what we end up with is a decision, like a dossier that looks like this.
00:44:08.559 --> 00:44:08.880
Okay?
00:44:09.519 --> 00:44:11.920
And yes, we did call our bot Swamp Ops.
00:44:12.559 --> 00:44:20.320
But we end up with like a very structured level of information that comes back every single time in the same format.
00:44:20.559 --> 00:44:32.960
It has a title, it has an understanding of what's happened, it has the category, it has which signals that come in, which webhooks are actually attached to it, how many times it's actually fired, when it last updated, etc.
00:44:33.119 --> 00:44:33.440
etc.
00:44:33.760 --> 00:44:36.480
But we build dashboards on the fly.
00:44:36.800 --> 00:44:47.679
Okay, dashboards for me are a thing of the past because we used to create dashboards and operations, and those dashboards would live forever, and you would have to maintain them, and they were just an absolute pain to maintain.
00:44:47.840 --> 00:44:52.800
Now we build them on the fly specifically about the data we're we're actually trying to report.
00:44:52.960 --> 00:44:55.119
But you see at the bottom we have a little ask button.
00:44:55.280 --> 00:45:12.079
You click the ask button, you get a little window that pops up, and as an operator, you can start to query and talk with the LLM, which will help you be able to do it rather than saying go to this system, go to that system, go to this, wherever, it does it centrally and it records that conversation as part of that incident.
00:45:12.639 --> 00:45:14.559
The ask flow is kind of cool here.
00:45:14.719 --> 00:45:19.599
Um so basically, you go to an ask question.
00:45:19.840 --> 00:45:21.519
Do you have permission to do it?
00:45:21.760 --> 00:45:28.719
If you do, you ask the question, you then dispatch it, it starts doing some queries and some start-come code in.
00:45:28.880 --> 00:45:33.760
It actually goes off in a read-only mode to our our um ops system.
00:45:34.400 --> 00:45:39.679
It then prepares, like you know, all of the payload that it has to bring back in order to do it.
00:45:39.840 --> 00:45:46.079
It analyzes the output with the LLM and it gives us the answer for how we push it in through the system.
00:45:46.320 --> 00:45:47.599
It's very, very cool.
00:45:47.840 --> 00:45:50.480
We're in charge of the decision at that point in time.
00:45:50.719 --> 00:46:00.239
This is an example, this is a real example of an investigation of when a human has asked a question on that same incident.
00:46:00.320 --> 00:46:05.760
And what they said is uh can you see why the last rotation failed?
00:46:06.000 --> 00:46:12.480
And we can see its findings, we can see its evidence, we can see its likely cause, and we can see the recommended next steps.
00:46:12.639 --> 00:46:15.599
Notice that the agent, again, the slides will be there so you can see it.
00:46:15.760 --> 00:46:22.239
Notice the agent hasn't said, I've gone and fixed it, it's asking me to make the
decision for the next steps.
00:46:22.400 --> 00:46:23.840
So if we go back to 3 a.m.
00:46:24.000 --> 00:46:27.119
and it happens, only signal pages you.
00:46:28.719 --> 00:46:29.360
No noise.
00:46:29.519 --> 00:46:37.440
No noise gets through, you don't want noise to get through because the more noise that we get through in a system, the less we pay attention to that.
00:46:37.679 --> 00:46:54.480
We get our alert and our dashboard and our links, we can query it, we can look for regular uh for correlation and incidents and occurrences, and like we can take actions, we can take actions based on what the agent actually gives us and helps us understand.
00:46:54.960 --> 00:46:57.760
Why is it the same shape every time?
00:46:58.079 --> 00:47:03.039
One, it's not always the same person on call, not always the same person woken up.
00:47:03.199 --> 00:47:11.519
But if we understand and we give the information in the same way, the same structure, the same understanding, then it's very easy to read it and understand what's happening.
00:47:11.599 --> 00:47:31.360
So a sleepy engineer who's on autopilot knows exactly what information to look at, and it knows what to um the steps that it actually has to take next rather than trying to determine which system do we think this is from, which application could it be, is it a downstream dependency?
00:47:31.440 --> 00:47:33.519
All of that is taken care of away from us.
00:47:33.760 --> 00:47:35.679
The agent never makes a call here.
00:47:36.079 --> 00:47:38.480
Never once does an I take that back.
00:47:38.639 --> 00:47:39.360
That's not true.
00:47:39.519 --> 00:47:50.079
An agent makes the call about if it's seen the incident before or if it's um an incident that is currently already um being investigated and it's a dupe coming in.
00:47:50.159 --> 00:47:56.239
That is the only time the agent will make a decision on our behalf about the findings and what we actually have to do.
00:47:56.400 --> 00:48:00.559
We as humans are the right people to make the decision.
00:48:00.960 --> 00:48:14.400
In 1979, an IBM training slide had the following words on it: a computer can never be held accountable, therefore, a computer must never make a management decision.
00:48:14.559 --> 00:48:20.239
Okay, if we treat LLMs in this respect, we will get fantastic outputs.
00:48:20.400 --> 00:48:27.360
Okay, never, ever, ever, ever, ever defer your taste and your expertise to an LLM.
00:48:27.519 --> 00:48:27.760
Okay.
00:48:28.239 --> 00:48:33.039
Eric's talk this morning could not have been a better introduction to this topic.
00:48:33.119 --> 00:48:42.880
Okay, they are very, very, very good at what's happening, and he said, like, we are at the era now where we won't make money on code, but you can make money around products with code.
00:48:42.960 --> 00:48:49.360
And what happens there is that you bring taste, you bring expertise, and you bring domain knowledge into what's happening.
00:48:49.599 --> 00:48:58.000
That
same theory can be applied at any point of using LLMs across the sp the spectrum, and operations is an incredibly important aspect of that.
00:48:58.239 --> 00:49:01.199
Our agent never ever touches the controls.
00:49:01.360 --> 00:49:11.599
Okay, it's never gonna go silence or ping it up the chain, ignore it, let the next person, the agent picks it up because an agent doesn't care about toil.
00:49:11.760 --> 00:49:20.400
An agent will run that same step again and again and again, and it'll follow that same flow that you actually build and you that you understand.
00:49:20.960 --> 00:49:25.360
What a human brings into the system is that judgment.
00:49:26.000 --> 00:49:27.840
Do we roll back now?
00:49:28.559 --> 00:49:31.119
Is it something else that we have to think about?
00:49:31.360 --> 00:49:34.400
Do I need to wake another engineer who knows this better?
00:49:34.639 --> 00:49:37.119
An LLM won't be able to give you any of that.
00:49:37.280 --> 00:49:46.880
Okay, spend the judgment in the areas that matter for you as an organization and you as a team rather than deferring that away and making the potentially wrong decision.
00:49:47.119 --> 00:49:48.320
I want to be very clear here.
00:49:48.559 --> 00:49:52.400
LLMs are not here to take our jobs.
00:49:53.039 --> 00:49:54.000
They're absolutely not.
00:49:54.159 --> 00:49:57.119
I've been telling this for the last nine months to people.
00:49:57.840 --> 00:50:03.920
LLMs are there to be a force multiplier for what we actually want to be able to do and what we can do.
00:50:04.239 --> 00:50:15.679
But we as people who have systems knowledge, systems thinking, architecture, taste, all of these things work incredibly well with agents to give you really great outcomes.
00:50:15.920 --> 00:50:21.119
If you defer and you don't trust an agent output, that's totally fine.
00:50:21.280 --> 00:50:21.920
You know what I mean?
00:50:22.000 --> 00:50:25.599
Nobody's telling you you gotta burn away all your process and just throw everything to an agent.
00:50:25.840 --> 00:50:30.079
You are still in full control of what can happen, and this is a great way of being able to do it.
00:50:30.320 --> 00:50:39.920
You can run this same playbook yourself, let an agent run it at the same time, and see how you can actually close you are.
00:50:42.239 --> 00:50:42.719
Muse?
00:50:43.280 --> 00:50:45.119
Like as in like a personal agent?
00:50:46.719 --> 00:50:50.800
Um I I I I I I have words.
00:50:51.519 --> 00:50:54.320
I I'm not a personal agents fan at this moment in time.
00:50:54.480 --> 00:50:58.159
Um I don't know why I'm not a personal agents fan at this moment in time.
00:50:58.320 --> 00:51:08.719
I think it was probably because when I first looked at OpenClaw, it just looked such a pain in the ass to set up, and I've just skipped that whole personal agents thing, and I kinda just set up workflows that allow me to do it myself.
00:51:08.880 --> 00:51:11.199
But you're right, like Muse can handle a
bunch of this stuff as well.
00:51:11.360 --> 00:51:12.079
Totally could.
00:51:12.239 --> 00:51:14.559
I I wouldn't trust my data with them, but that's okay.
00:51:14.639 --> 00:51:15.760
That's that's a your decision.
00:51:15.920 --> 00:51:17.760
That's a that's a your decision, people.
00:51:18.320 --> 00:51:33.599
And but what actually happens here is that because an LLM is actually helping us and going through the steps and understanding, you're able to gain understanding and review and sharpen what is going on at every moment in time.
00:51:33.920 --> 00:51:40.719
This is genuinely how many times we've changed this flow in the last couple of like the last two months.
00:51:40.880 --> 00:51:41.199
Right?
00:51:41.360 --> 00:51:58.079
Every time we find something new, every time we have a learning out of an incident, all of that feedback is pushed back into the system to strengthen it, to make it better, so that the next time we actually can get better decisions, better understanding, and get a better outcome from what we have.
00:51:58.719 --> 00:52:00.480
How do you get started?
00:52:00.719 --> 00:52:10.800
I mean, if you if you're interested in this type of area with regards to agents and operations, first thing you gotta do is you gotta take out some of that institutional knowledge out of people's heads.
00:52:10.960 --> 00:52:21.199
Okay, we all have people in our team who we go to when something happens, when shit hits the fan, we go and say, Hey, like, can you help me out?
00:52:21.360 --> 00:52:23.119
Can you show me this part of the system?
00:52:23.280 --> 00:52:24.639
We gotta start bringing those people out.
00:52:24.719 --> 00:52:28.239
I was actually talking to Ahmed um earlier um about how we do that.
00:52:28.400 --> 00:52:31.039
We do that specifically at our company because we interview people.
00:52:31.119 --> 00:52:42.480
Okay, when we're helping customers, we go and we sit with the customers, we interview them, we pull that data out of them, and then you're able to turn that into actually you know a reusable system that like that you can encode.
00:52:42.639 --> 00:52:44.079
It's there, it's like available.
00:52:44.239 --> 00:52:50.079
And the more you do that, the more you pull that institutional knowledge out, the better that you actually get the outcomes of what's happening.
00:52:50.320 --> 00:52:58.800
You can then turn each step in that flow into a workflow, into a behavior that is encoded and it can run and it can run on its own.
00:52:58.960 --> 00:53:05.760
You can like fix that shape of the output that you want, you know, that makes it easy for people to understand what's happening.
00:53:05.920 --> 00:53:16.800
You can let the LLM read it, and then every time you do a post-mortem in your system, you would add a feel to the end of the post-mortem about how you can strengthen that system and make that system better.
00:53:16.880 --> 00:53:18.400
Okay, you have to be able to feed it.
00:53:18.559 --> 00:53:20.079
For us now, a page at 3 a.m.
00:53:20.320 --> 00:53:21.519
kind of looks like this.
00:53:22.000 --> 00:53:24.000
We sometimes get paged.
00:53:24.239 --> 00:53:31.280
I mean, I think the number of times we've been paged in the last two months is maybe twice.
00:53:31.599 --> 00:53:32.320
You know what I mean?
00:53:32.400 --> 00:53:36.320
And yes, one of them was Halloween night, but you know, that sometimes happened.
00:53:36.400 --> 00:53:37.599
It's only a certificates issue.
00:53:37.679 --> 00:53:42.800
It was very it's it's not that it fully expired, it was classed as a P2, so it wasn't a drop, everything and fix it.
00:53:42.880 --> 00:53:49.280
So, you know, our judgment now is about spending on the problem, fixing this system strength in this system to make this better.
00:53:49.519 --> 00:53:52.079
You are not being replaced in any way, shape, or form.
00:53:52.239 --> 00:54:00.480
You're kind of arriving with that first 30 minutes done, armed with a bunch of material, a bunch of um information that can help you make a decision.
00:54:00.639 --> 00:54:07.519
If you scan the QR code, you'll the slides will be on there in maybe 30 minutes, um, so you'll be able to see it.
00:54:07.840 --> 00:54:13.440
I I never know if people are comfortable asking questions or not, so I always prepare my own.
00:54:13.599 --> 00:54:20.639
I come armed with questions because every time I give this talk, people do have questions, and then I'm able to feed those questions back in.
00:54:21.039 --> 00:54:23.840
Look, I'm even like taking my own advice of what's happening, okay?
00:54:24.079 --> 00:54:27.119
What happens if the agent gets the root cause wrong?
00:54:27.280 --> 00:54:39.920
The agent's not telling you what the root cause is, the agent is giving you hints and ideas of where it can be, areas where you can spend your time and focus to sharpen up the resolution and like what's happening, okay?
00:54:40.079 --> 00:54:43.440
It's evidence, it's not a conclusion.
00:54:43.760 --> 00:54:46.639
You, as people, are part of making that conclusion.
00:54:46.880 --> 00:54:51.440
Um, why not just use honeycomb like Canvas or Dash Zero or stuff like that?
00:54:51.599 --> 00:54:52.079
Use them.
00:54:52.159 --> 00:54:52.719
Absolutely.
00:54:52.880 --> 00:54:54.400
If you have these tools, use them.
00:54:54.639 --> 00:54:57.760
It just so happens that our flow is a little different.
00:54:57.920 --> 00:55:01.519
It's not specifically a honeycomb-shaped problem or a dash-shaped problem.
00:55:01.599 --> 00:55:08.159
It is a system that brings together a bunch of you know different areas and telemetry and all these different areas to be able to do it.
00:55:08.239 --> 00:55:10.239
So we built the system that fits us.
00:55:10.400 --> 00:55:12.880
We are in the era of personalized software.
00:55:13.039 --> 00:55:15.760
As you quite rightly heard this morning in Eric's talk.
00:55:16.000 --> 00:55:21.440
We should be able to build the tools that
work for our teams and our organizations and our workflows.
00:55:21.679 --> 00:55:23.599
What about alert storms?
00:55:24.079 --> 00:55:25.360
Serialize them.
00:55:25.599 --> 00:55:31.440
Don't make the system take hundreds of issues at the same time if you're not able to cope with a hundred issues at the same time.
00:55:31.599 --> 00:55:32.079
That's up to you.
00:55:32.239 --> 00:55:37.599
Fingerprint them, dejup them, have correlation windows, have resolutions, all of these different things that make happen.
00:55:37.840 --> 00:55:39.440
What about prompt injection?
00:55:39.599 --> 00:55:41.920
Surely somebody's going to prompt injection you are.
00:55:42.079 --> 00:55:42.960
Yeah, they will.
00:55:43.199 --> 00:55:46.159
Ultimately, there's always somebody smarter, and they will get through.
00:55:46.320 --> 00:56:01.920
So, what we have to try and do is we have to try and decouple what the alert text and what the label is and mark them as untrusted data and not instructions for the LLM, and then our LLM output is actually goes through a very specific schema.
00:56:02.079 --> 00:56:07.360
You have to think about building these systems, like you have to think about building software systems.
00:56:08.239 --> 00:56:14.639
CVEs, attack, surface, everything is getting incredibly, incredibly sophisticated.
00:56:14.800 --> 00:56:19.119
I mean, October was probably the worst month on record for CVEs.
00:56:19.280 --> 00:56:20.719
It's gonna get worse.
00:56:20.960 --> 00:56:22.159
It's gonna get worse.
00:56:22.320 --> 00:56:27.280
I just read before this talk that there's an uh Node.js vulnerability in the newest version of Node.js today.
00:56:27.360 --> 00:56:31.280
So there's gonna be patchmaged in this afternoon, okay, when that comes out like later today.
00:56:31.440 --> 00:56:35.199
It is literally every single day now, you're gonna have to deal with this.
00:56:35.280 --> 00:56:41.840
So the best thing that you can do is get ahead of it and start to put the systems in that can help you actually put those uh pieces in place.
00:56:42.079 --> 00:56:48.880
Isn't this just giving an access, or excuse me, isn't giving an access, um an agent access to production dangerous?
00:56:49.039 --> 00:56:50.239
Absolutely it's dangerous.
00:56:50.320 --> 00:56:52.239
Don't ever give them rewrite access, please.
00:56:52.400 --> 00:56:53.119
Jesus no.
00:56:53.360 --> 00:57:02.079
Give them just read access, give them the minimal position, the minimal credentials that they need to be able to do the job that you need them to.
00:57:02.320 --> 00:57:05.679
Don't just share an API key to be able to do something.
00:57:05.920 --> 00:57:08.320
You are setting yourself up for pain.
00:57:08.480 --> 00:57:11.840
Uh, won't all call engineers lose the skill of investigating?
00:57:12.000 --> 00:57:12.480
Nope.
00:57:12.719 --> 00:57:13.440
They will not.
00:57:13.599 --> 00:57:17.920
They still have to make the judgments, they still have to bring all the data
together, they still have to do all this.
00:57:18.079 --> 00:57:19.440
What does it cost to run?
00:57:19.599 --> 00:57:21.280
It's really cheap, I told you.
00:57:21.440 --> 00:57:23.039
I can't spend Jeb credits.
00:57:23.199 --> 00:57:31.360
I mean, I wish I I don't wish I had more alerts, but I wish I could spend more JE credits because in two months I've spent 23 cents.
00:57:31.519 --> 00:57:31.840
Okay?
00:57:32.159 --> 00:57:34.239
Like, that's not a lot of credits.
00:57:34.400 --> 00:57:39.119
Um, you know, there's any uh anybody any other questions that I can add to my questions bank for next time?
00:57:42.320 --> 00:57:48.000
So you said a couple times that LMs are not here to replace us, which is a great sentiment.
00:57:48.079 --> 00:57:50.000
UmAI is a charter.
00:57:50.400 --> 00:57:58.159
It says that they are trying to create automation for quote most economically valuable work.
00:57:58.800 --> 00:58:00.719
What do you think is doing all these big signals?
00:58:01.280 --> 00:58:08.320
I mean, please take it with what I say with all of the respect to all of these people.
00:58:08.639 --> 00:58:09.760
They are a business.
00:58:10.079 --> 00:58:17.920
They're an incredibly profitable business, they're a marketing machine, they are going to continue to try and sell software for what's actually happening.
00:58:18.079 --> 00:58:22.079
Okay, they are not the only frontier model that you can use.
00:58:22.239 --> 00:58:31.280
You can use local models, you can use all sorts of, you know, there's a lot of uh sovereign models that are coming out of Europe to be able to help like move away from this.
00:58:31.440 --> 00:58:38.159
You've got to take statements like that from big large companies with a pinch of salt, in my opinion.
00:58:38.320 --> 00:58:42.800
I've heard other frontier model companies saying code is solved, okay?
00:58:42.960 --> 00:58:44.639
There's no need for it anymore.
00:58:45.039 --> 00:58:47.599
Sure, taste is not solved.
00:58:48.079 --> 00:58:51.519
You know, like developer expertise is not solved.
00:58:51.679 --> 00:58:54.320
Like an LLM can't bring any of that judgment.
00:58:54.400 --> 00:59:02.320
It'll just continue building code on top of code, on top of code, and it'll do brilliant, and you'll have a tool that does every single thing in the world, everything.
00:59:02.639 --> 00:59:04.320
But it might not be for you.
00:59:04.559 --> 00:59:14.639
So in my in my opinion, I think we are very, very, very, very far away from like code, like LLMs replacing all of us.
00:59:14.800 --> 00:59:33.360
I think unfortunately, there are companies who will like uh undergo like a lot of change because of restructuring of agents in the in the org, but I I still continue to believe that engineering will exist in X number of years' time because we are people who have a lot of thinking and a lot of expertise.
00:59:33.519 --> 00:59:40.960
And once you couple that with an agent who can rip through like what you need to do in a short period of time, you can have incredible results off the back of it.
00:59:41.199 --> 00:59:42.639
I hope that answers your question.
00:59:49.199 --> 00:59:49.519
Hello?
00:59:50.239 --> 00:59:52.000
Yeah, first of all, thanks for the talk.
00:59:52.159 --> 00:59:57.280
Uh I immediately took a couple of pictures of the slides and sent them to our DevOps team and our data team.
00:59:57.519 --> 01:00:00.480
Um I'm wondering like what's next?
01:00:00.719 --> 01:00:06.800
What gaps do you see that you're working on, or what workflow do you want to do a similar flow with next?
01:00:11.039 --> 01:00:14.639
We're basically pulling like all of the pieces of the puzzle for our organization.
01:00:14.800 --> 01:00:19.199
We use everything of that we build our tool, builds our entire organization.
01:00:19.280 --> 01:00:29.199
Okay, we build our software with our tool, we build our um CI pipeline with our tool, we build our release pipeline with our tool, we build our ops pipeline with our tool, we build our CMRM pipeline for our tool.
01:00:29.280 --> 01:00:35.679
We're basically pulling at all the pieces and we're trying to like break the status quo of the old tools that we need to use.
01:00:35.920 --> 01:00:45.679
We broke away from using GitHub issues, and we did that specifically because not because GitHub issues were bad or that they're suffering an incredibly bad uptime.
01:00:45.920 --> 01:00:47.920
That is a completely separate issue.
01:00:48.079 --> 01:01:01.039
It was much more about the fact that we were building, we build agentic software and we have a very like specific agentic flow, and inserting agents into a flow that was designed for humans was a wee bit more difficult than there.
01:01:01.119 --> 01:01:03.360
So I can talk to you more about some of the things that we do.
01:01:03.440 --> 01:01:04.639
So, folks, thank you so much.
01:01:04.719 --> 01:01:05.519
Enjoy the rest of your day.
01:01:05.599 --> 01:01:06.880
I really appreciate you coming out.
01:01:12.000 --> 01:01:13.519
Hope you enjoyed Paul's talk.
01:01:13.599 --> 01:01:20.480
If you want to check out other excellent talks from DevCon, you can click the link in the show notes or scan the QR code.
01:01:20.559 --> 01:01:29.360
Uh we have some amazing talks in DevCon that will have concluded by the time you listen to this episode from the top labs, from thought leaders Lex Tiege, and others.
01:01:29.519 --> 01:01:37.679
Paul, thanks for coming here again on the podcast and to the AI Native Dev again, and I'll see you next time.
01:01:38.400 --> 01:01:43.280
The AI Native Dev is brought to you by Tesla, the package manager for skills and context.
01:01:43.360 --> 01:01:46.320
Your hosts are Guy Pajani and me, Simon Maple.
01:01:46.400 --> 01:01:48.000
Our producer is Tom Dowler.
01:01:48.239 --> 01:01:51.519
The AI Native Dev is not just a podcast, it's a community.
01:01:51.599 --> 01:01:55.119
And we host monthly meetups at the Tesla offices in central London.
01:01:55.280 --> 01:02:00.239
Visit Tesla.io forward slash community to learn more, and I hope to see you there.