Sam Newman on Software Factories: Why AI Can't Fix Your Production Incidents (Yet) の全文書き起こし

situation where you've gone a bit too far or somebody else made decision to go too far, how am I going to help you make sure the system is actually resilient? So it's like My hope is this book is like a rubber ring you give to a someone who's drowning. It's like, okay, this is it. This is the bo speaker-1: I mean I I I feel that this is something that gets also more and more important with with AI. So this is probably something that we are going to talk about in the in the next hour as well. But first I thought when you said that the idea of resilience is in this case to like not make stuff go wrong, it's more like make sure that stuff goes right. I really think that is a fundamental difference we might wanna explore a little bit more before we switch into the AI topic because I feel that also for the AI topic this is very much true, this different view on the things, you know. speaker-0: And this is kind of the other reason I wrote the book is that there's a lot of there's a whole field of study around resilience engineering that comes out of the world of sort of safety critical systems. So think things like, you know, power station control systems, air traffic control systems, fly by wire aircraft, but even into things like, you know, operating theaters and how hospitals run, right? These are all safety critical systems. There's a whole field of study in that world because the Implications of getting those things wrong is that people die. And out of that is a lot of lessons and knowledge, but it's often can be quite academic and hard to process. And so I first came across that stuff maybe about 15, 20 years ago, but I really didn't understand it. And I was like, can I make it more approachable? And so I've been trying, I've been backing, you know, headbutting myself, you know, the these papers don't make sense to me. And the thing that kind of helped click a lot of this, this is something I talk about in the first chapter, is that You've got kind of traditional safety management and then you've got what we now refer to as resilience engineering. So safety management, traditional kind of approaches towards making sure that things are safe, was very much focused on let's make sure nothing breaks. Mm-hmm. Right. So that's like the, you know, the in the Simpsons, there's that sign, you know, how many days since there was last an accident? Right. That's a focus on let's make sure there's nothing, nothing ever goes wrong. And that's not a bad idea. It's not wrong to not want things to go wrong, but it's not by itself sufficient. speaker-1: problem is that in reality a lot of stuff goes wrong, right? And and even like being faster with AI and being like more distributed and so on, there's more factors that make stuff go wrong, right? So speaker-0: Absolutely. Yeah. I mean it's partly a numbers game. It's partly a things outside your control game as well. I mean, looking at it from a numbers point of view, just as a sort of a theoretical exercise, imagine you've got a component in your system and on any given day, that component has a one in one hundred chance of failing. It's quite high of as a failure rate for like a piece of hardware, right? But just to go with it. So on any given day, you've got a one in one hundred chance that component has broken. So if you've only got one component in your entire system. Putting effort into making sure that that failure rate goes to one in a thousand may actually make sense because if it does fail, the whole system goes down, right? So you you kind of you treat it a bit differently. Now, if you've got a hundred components, the chance on any given day of a component failing is getting close to one. It's not exactly one, but it's getting close to one. So it's getting close to certain. So once you start getting into that world, you're like, well. I can keep trying to reduce the chances of individual components failing, or I can now switch my mindset to how can I make sure the system still operates, even if things fail. Mm-hmm. And one of the best sort of physical manifestations I saw of this was getting to see some of the old Google servers. you can see these in the Science Museum, I think, in the UK. I they're still there. And I think there was a rack of them in the computer museum in Silicon Valley. And I got to see them on the Google campus back in 2007. But these were like so you you think about a traditional server in a rack, right? You see kind of glowing lights and all these nice panels. The old Google servers are just bare motherboards wrapped directly, no case on top. and actually turns out it's fine for airflow, so that's not a big issue. Yeah. when you look at them, you realize some interesting things. So firstly, you see the h the power supplies hanging out the back of the case. It's not built in, it's hanging out the back of the case. And it's secured with Velcro. The hard drives in the case are also secured with Velcro because the things with movable parts, the things that are most likely going to fail in the server, power supplies, spinning patter hard drives, those that have the highest failure rates amongst the components. And so Google had realized we've got so many computers, one of them's going to fail anyway. So Let's firstly build our software in such a way that we can keep operating even if we have a computer fail. Tick. And secondly, let's put some effort into making sure that when a failure does occur, we can get that machine up and running quickly. And so literally it with a computer got fails, the software doesn't really care. I mean this is we're back in the days of MapReduce, right? You'd slide a tray out, you'd find, okay, that's the hard drive, unzip the Velcro, throw it in a bin. put another hard drive in, tape on the Velcro, put it in, it pixie boot up and it away you go. And so there's kind of two levels of this. There's a guy called David Woods who wrote quite a seminal paper on resilience engineering, kind of the

four the four concepts of resilience engineering. And he took two of those concepts he talks about are kind of robustness and rebound. So robustness is the ability for the system to keep operating even where things go wrong. So that's like the computer failing. And then rebound is your ability to recover when something does fail. And that's the example of the computer coming, getting up and running. Right. So to and so you can kind of approach that from two levels. So they they optimized around not caring if an individual computer failed and making it really easy to get an individual computer up and running again. And I'm not saying that's right for everybody, obviously. if you've only got one computer, it absolutely makes sense for you to consider things like redundant power supply supplies and, you know, redundant rap local networking and multiple nicks and all these other things that you might do to kind make one machine more resilient, but increasingly our systems don't rely on a single machine anyway. So it it changes how we think a little bit about about keeping our systems up and running. speaker-1: I think we should explore this rebound idea a little more because I I feel that this is super interesting now with with the with the AI way of working with AI and using AI for building systems as well. because there is this idea of AI is going to introduce like less quality and so on, so the side that is making sure the system does not do any errors might become less good. Maybe you have some ideas about that as well. But on the other side, we could maybe also be much better now on the rebound side because we have AI, so we can react much more quickly using AI. So I I think we sh maybe we can what's what's your idea? What's your take on that? speaker-0: Yeah. We can peel apart some different parts to that. Yeah. So firstly at a very l base practical level, if you're relying on models from the foundation companies, if you're kind of using anthropic or open AI models, their quality of service is quite poor compared to normal companies. Very poor. So at a base level, they don't always run. They you you you do have connectivity issues. You do have kind of I mean they are trying to improve that, both of those companies are trying to improve that. But fundamentally, you get nothing like the quality of service that you would expect from a normal managed cloud resource. And so at the base level, that's why a lot of companies will either run their foundation models from Amazon on AWS, like with Bedrock, or they'll go to Microsoft or they'll go to Google, either as the primary place they run those foundation models or to fallback. And so I spoke to a few companies that basically use them As a fallback. so that's kind of just the basic level, right? We can't, they're not great. Th there's a second issue in there though, which is more about the quality of the responses we get back. And that's so I don't really talk in the book about functional correctness

in the resiliency book because I feel there's other places we can talk about that. But that is an issue fundamentally with the use of any LLM enabled products because the non determinism means that working out if it's doing the right thing becomes sort of more challenging. I'm sure people have heard this, but just in case, you know, if I have a deterministic bit of code, given the same inputs, I get the same outputs. Therefore it's quite easy to test and verify that it's going to do the right thing in a given situation. That our LLM based agents are non-deterministic. Given the same input, they will give us different output. And that makes it quite hard to test in a traditional way. So actually verifying that these things are behaving correctly is is difficult. So, this is where you get into the world of evals and having to build evals basically into your production systems. And it also increases the importance of observability around your agentic systems. There's another little hidden bit that comes back down to how much we can trust the model providers here, is that we definitely see examples where models just start degrading in their quality for no apparent reason. So they're not steady state. They're not even inconsistent within a narrow band of inconsistency, right? You do get times where they just seem to be a bit dumber for some reason. And that's not great. So having some checks around that i is kind of really, really important. And this is also why, you know, there's other aspects that come into it from a you know, from a resiliency point of view, but basically breaking, chunking these things up into smaller pieces is is is worthwhile. And, you know, if you're taking a a really big view to resiliency, right? You're thinking in resiliency in the longer term. You've also got to be thinking about multi-model and multi-vendor, because the extent to which we can guarantee that these vendors will be around in the shape the way they are in the next five years, next 10 years, I think the likelihood is low given what everything that's going on. So what can you do to create opportunities in your architecture to replace this with other things, alternative vendors, alternative models? So I think at the at the moment now, you know, I would strongly recommend that you at least have some experimentation around mixing models from different providers, be it open weights models or different foundation models, just so it's you make sure you've got a a different direction to go into if one of those main providers kind of goes goes away. So that's kind of the big picture. speaker-1: I think that that's more on not the rebound part, right? That's more on the like how to do the how to build it in a secure way in the first place, right? With the e valts and so on. And I mean the the the specific questions I have here is if we we can focus a few more minutes on that, but I feel that also what you said, the like switching a model is not only like the technical part of just like routing it through a different server or something, but if it's a completely different model, the responses will not only be of different quality, but they also will be completely different, right? So the same skill or the same prompt will not result into the same output and not in a sense of okay, it's a little bit bad quality. It could be like completely different, right? What this model is doing. speaker-0: And and this is this though for me is like so there's an analogy here around multi cloud and like should you go multi cloud? And I tend to say no, don't bother because it's too much work and it's too much hassle. Yeah. And that's based on the risk factors around is a cloud vendor really gonna massively jack up its prices compared to the competition? It's unlikely. Is a cloud vendor gonna vanish? It's unlikely. Yeah. I don't think either of those two two r things apply to the main model vendors at the moment. Yeah. Could the prices fluctuate wildly? Absolutely. Could one of them vanish? Absolutely. Right. So the risk factors are different. But you're totally right that they are less utility than the other pieces. And so my approach to that is not to say for a given workload, run it both on say both anthropic and open AI. But if we imagine our genetic workflow as a series of steps, right? A series of small discrete units, maybe you try using one of those units, maybe all of your units you're running off anthropic models. You pick one of them and say, actually, this bit here, we want to try using this component of our workflow running out of an open weights model from a different provider. So you're not hedging your risk on an individual bit of workflow, but you're broadening your skill sets and your awareness. And you are you're giving yourself some institutional knowledge that means we know a bit more about this open weights model or we know a bit more about this alternative provider. of these models. And so that that's more about, I mean, that's also like what difference between one solution being multi-cloud and an organization as a whole doing some stuff with AWS and doing some stuff with Zure. So I think there's a distinction there. and I'm more bullish like about doing that with models than I am necessarily multi-cloud. So I think the trade-offs are different. Coming back though to the the kind of last part of your question, which is about the rebound, and I think the thing you talked about is okay, does it help us get up and running quicker? And then there is definitely some arguments there. And I think there's not in the automated sense though. So I really want to be re caution people, right? So kind of rebound is like the system's ability to recover in the wake of something that we could foresee. And rebound in Woods' context is typically speaking about the things where our system just sort of works itself out for itself, right? So this is like the like you know, Kubernetes podfails. Kubernetes spins are pop back

up. So a lot of rebound is about automated things. Automated remediation with AI is a long way off. And it's not something I would suggest doing. Because again, we have this challenge that we've got the trustworthiness of the models. Trustworthiness means also that non-determinism. I'm not sure we can afford, like, I at the moment, automated remediations. work quite well when they're deterministic and for a narrow band of things. Where I think AI does help us a lot is the moment the human has to get involved in working out what the hell's going on. And that's where if you've got good signals coming out of your production systems, so you've got good observability, then I think AI can help an awful lot. So things like I mean, you know, things like so I I was chatting to Martha Lambert from instant.io recently is talking about how their kind of AI SRE stuff works. And like one of the things it does is it sees something that's a bit wrong and it and it says, look, this looks a bit wrong. So it creates like a little incident for you. And then you're like, I should have a look at that. You start looking at it. And this can obviously you but also it could be like an SLO got breached or an SLO got breached. Okay, we're going to trigger an alert. And behind the scenes, the AI thing's going, okay, well, let's see what's going on. Okay, well, this looks interesting. It sends you some information. It goes, and then while you're looking into it. I've also found these things. This could be related. there was a code change recently. And so what it starts doing is kind of delivering information to you to help you then understand could these things be the cause? And so it's, you know, what they're working. I mean, obviously I think gathering the information, I think, is quite straightforward and lots of people do that. What Insident IO have been working on a lot is can we get the AI to determine that there's an incident in the first place. And they've been working on that now for about three years, four years now, I think. And it's not trivial work either. So we're getting smarter at spotting there's a problem. We're already quite good at helping you when there is an issue. I think the thing that I still believe is a pipe dream is is the automated remediation part of that for most things. I think most organizations are pretty bad at doing automated remediation with deterministic code, let alone non-deterministic code. And there's a whole bunch of reasons for that. But like one of the prime aspects around why I don't think AI is useful for automated remediation at the moment is because when you look at instant response, instant response isn't a clear linear process. It's quite a messy exploitative process. It requires a lot of intuition. It requires a lot of exploration going off in different directions. And human beings have a poor enough understanding about instant response, let alone expecting and the thing is, this is the problem with AI. Like, if we want to get AI to do things for us, we have to be able to tell it what we want it to do. And we also have to be able to put things in place to make sure that we know AI is doing things correctly. And I think human beings have a bad enough understanding about instant analysis that we've got no chance of doing that with AI. So maybe five, ten years, or maybe around really trivial cases. But I do think right now, you know, around gathering that data and also around like things like post-mortem processes as well, like the post-incident reviews, they could be really helpful, right? One of the big things you need to do as part of a post-incident review is you need a timeline for what happened when, who did what. And, you know, you still rely on human testimony for that, getting interviews as as well as part of that. But what a lot of these some of these tools can do now is they'll just gather a lot of the raw data and give you that kind of draft timeline. And I say again, few enough companies do proper post instant reviews that having something which at least can get you started, I think, is really beneficial as well. speaker-1: Yeah, thank you. I I think so as well. With the rebound, I had one more idea and I'm kind of interested in what what you think about this. So if you if you use an LLM for coding, for example, I mean we are doing most most of that not in the product, but more for building any kind of product. And the the agents, they do a lot of rebound all the time, right? They the model says call this tool, and then maybe the model tells it in a Incorrect way, and then there's an error, right? Because the CLI command was wrong or the tool call was wrong. And then the harness and the model itself together, they are able to rebound because they just try it again, right? They say, okay, this was the wrong call. Maybe you can try another call to do it, and then they do multiple iterations, and then they are able to do it correctly. That was more the way that I thought about that. This is also kind of helpful to the ways we now can. can do the rebound that we couldn't do before because in a deterministic system that it would have wouldn't have been possible, right? It would have done the same wrong call all the time. Or am I thinking something incorrect? speaker-0: Well, I think the analogy there is interesting. It's more about how we handle the process and where we do it. I mean, there's a like in the first two chapters, well, so it's actually chapter three and chapter four of my book, of chapter three is all about like timeouts and chapter four is all about retries, right? So if you think about it, we already have to do this in deterministic systems. The difference is that we have to process what we're gonna do. So when I call a server endpoint, like so I'm saying, Okay, I need to know how much this thing costs, right? So I make a call. If I don't get a response, what do I do? Mm-hmm. Now, it's a little bit different with say a harness where you're going

just trying to get some information. There might be other places I can go for the information, but I know this is the place where the costs are. I can't get that. Okay, so I'm gonna retry. I'll try again. Okay, how often do I retry? Now, the important thing about something like a retry is retries bring systems down all the time. So if I just don't know how the retries are gonna be done, that's Massive issues because if I'm retrying too much, I'm causing resource saturation, which brings systems down. So we have to think carefully. Okay, I'm gonna retry, but I'm only gonna retry twice. And I'm gonna have a gap between them. So we have to think about this stuff. And you know, you can use things like Resilience for J, for example, that help you implement these retry policies. But we're having to think about it. But that's actually because we want to know what's going to happen. We want to know what's going to happen because we want to be able to predict as much as we possibly can what our system is going to do. And then you've got the kicker. And this is another the what happens when you've done the retries and you still don't know how much something costs, what do you do then? At that point, you don't want to say, it will just work it out, right? It's probably not a good answer. You you know, at that point, that's when you go talk to a business stakeholder and run them through the scenario and Look, okay, so in this situation, if we couldn't work out what something costs at that moment, what do you want us to do as a business stakeholder? How do you want the system to behave, or a user or whoever? I don't think we wanna leave it up to guesswork as to what the it might decide to do in that moment. I think at those situations we want a degree of specificity. I think the difference is if we think about it in the context of harnesses and coding, is what we're generating there is something which can then be further validated to make sure that it's done the right thing. Yeah. Whereas if we're making a decision in a production system about should we do A or B, that's a bit different to I've created some code based on spec. Now we can go through the, you know, the iterations, the running the test to make sure that I've done a good job. The stakes of getting that wrong are quite low. And I think we can afford to let the harness kind of work itself out. The stakes of getting it wrong in production can be bad for an individual user and calamitous if it causes like a production issue. So I think I'm not sure that we can take the things that harnesses are able to do and assume that they would work in the same way in our kind of production running systems. I think there's a the the stakes are are a little bit different and the cost of getting things wrong were a little bit different. speaker-1: Yeah, I I feel that this is a very good distinction because from from the work that we do with with our clients and and companies, often AI is treated very similarly if I

want to build a production system that uses AI and if I'm using AI to build a production system. yeah. Either with or without AI in it. And it's exactly what you are saying there is, right? That this is you need to look very differently at both of these scenarios. Yeah, and and I think there's also now this this bridge that you mentioned in the beginning of how does that also affect the organizational side of things, right? Like it's the resilience is not only about the system and the technology, but also about the organization that uses it. And that also might be very different for both cases, right? speaker-0: Yeah. I think the first thing is you're just you're spot on with that distinction. I think it's a really important distinction to make, is that develop you know, we can create great products that don't have any AI in it in them with AI, right? We can have an AI enabled software delivery process that creates great software and great products. That d it sells not have any AI in it. We can also use AI to create software that has AI in it, right? And that's also can be fantastic. But then you add in a bunch of other concerns that need to be considered. The resiliency stuff I touched on earlier, the security side of things as well, like considering things like the lethal trifecta. And I think it's kind of important to distinguish between those those two cases. So I think you're you're spot on. I think in terms of going kind of beyond that robustness and rebound, you touched on into some of the organizational and sort of human factors. Around resiliency. This comes back to Woods' four model, the four concepts. There's four concepts he gives us the robustness and rebound, which we've kind of touched on. The other two that come out of that paper is sustained adaptability and graceful extensibility. These are quite verbose, like hard to understand terms, because again, the original paper's not designed for us to read. It's designed for other people to read, right? So when he talks about graceful extensibility, he talks about our system's ability to deal with the unknown. Primarily when we try and make our system robust, we have to have foresight. We have to know what could go wrong and build things into our system to deal with the known potential problems we might face. Like we might have a timeout, we might have an error, so we know we need to retry. So those are things we know about in the future, and we can build those into our systems. What about the things we don't expect? How well do we deal with the unexpected? I once had a network partition caused by rabbits eating network cable. I would have been crazy to think. ahead of time. What about rabbits? I had another one where an SSD caught fire and it took us five days to be told that that's what had happened at an SSD inside of SAN. Right. You can't foresee those things. So then the question comes is okay, how do we continue making sure that as many things go right as possible when things we don't expect happen? And that is primarily all about people. It's not only about people, but it's mostly about people and organizational culture. Because if you're in an organization that's quite hierarchical, where you've got a very rigid and structured role description, right? You're put in a box and these are the things that you are expected to do and these are the things you're allowed to do. Often those constraints that are placed upon you are based on how the organization operates, based on past knowledge. So then when something surprising happens, you'll find all kinds of constraints that are going to stop you. Doing different things to deal with surprising circumstance. That's why when you often see organizations which claim to be resilient, they're actually brittle because they have no ability to change. They have no ability to react. And so that's kind of an example where organizational dynamics can come into play. So it's like surprise, and it's like, well, how well are we gonna deal with this? What are we gonna do? How are we gonna act in that moment? And you can get better at that. Right. There are lots of great exercises and things like game day exercises, for example, that can help you get better at dealing with surprise as an individual. But you also need a a kind of corporate culture and set of controls that are flexible enough to give you the wiggle room to deal with that surprise as well. And then the last part of that was sustained adaptability. And that is our system actually changing over time. So again, you know. I have a system that does really well right now, but the world changes around us, right? We get new technology that comes into play, new types of adversaries out there, new types of users. Our customers want different things, our business stakeholders want different things. The world is always shifting. Is our system also continuing to adapt based on what we learn? And so it's resilient over the longer term. And that's that sustained adaptability piece. Now, for that to happen, we need to create opportunities to reflect. When things go wrong. So having good po post-incident reviews, but also creating the opportunity to reflect when things are going well and taking information in from outside. Psychological safety plays a huge role in sustained adaptability. Because if you don't feel safe to raise an issue, that's a problem. Or you feel like admitting you made a mistake is going to get you immediately fired. You're going to cover it up. And so an environment like that. all this valuable information and learning isn't coming forward. And that's gonna undermine your ability to adapt and change the systems. This is kind of where a lot of the the people and organizational dynamics very much come into play. speaker-1: Yeah, very, very interesting. I have a little bit of a I don't know. It's like maybe we can manage that now or maybe we can't. I have a little bit of a experiment. So yeah. So we are talking a lot about the

these concepts and ideas around like this is like a production running system and we need to operate it and and so on. And in my current thinking or what I see a lot at the moment is that instead of Talking about production running systems, you could also talk about any kind of organization that builds something as being this production system, right? Like it's an organization that builds something. And you could also think about this being an AI system like a factory or something that will build you some sort of so I know it's like a little bit of a mind. I don't know, it's like no yeah, for maybe for the viewers, yeah, for the listeners, right? You know, it's like you need to kind of go a little bit around the corner and the meta and you know what I mean? And now we need to rethink what does that mean now for resilience. Yes. Our system and the system that the system builds. So speaker-0: There's this lovely term called socio technical systems. Mm-hmm. And this is a term that describes any system where you've got people and technology working together. It's actually quite an old term. It comes from like studies done in coal mining, believe it or not. And the point I make in the book is that our a distributed system is a socio technical one. It is the people and the technology working together. So it you have to take that holistic view. And so it does include, I mean, although a lot of my book is focused on like the kind of things you do in production systems, you're you're it if you look at the system as a whole, you absolutely have to look at all of those parts. Those four concepts I mentioned, you know, rebound, robustness, graceful, extensibility, and sustained adaptability, they come from a paper that look at resiliency of systems in general, including biological systems, voting systems, air traffic control systems. So you can absolutely see how those concepts, those four concepts apply in those in those different places. Mm-hmm. Like let's take a software factory, because I know you you guys work on it quite a bit. speaker-1: And sorry to interrupt you there for for just 10 seconds. Because I feel that this software factory is for me it's like a twist in the head because you build a system that that like you described, this is socio technical system. But that system now is a software factory that will again need to build a system that needs to be resilient and so on. Like I'm maybe it's like maybe you can try to help me unpack this in my mind because this is always super hard for me to comprehend. speaker-0: We should probably talk, I mean, maybe talk about what like we mean by a kind of a software factory. Good idea. Which is yeah, which is kind of you know, we're we're having this discussion at the moment, which is how involved with the at the code level are we as human beings, right? Do we write the code by ourselves? Do we get AI to write the code? And then if we're getting AI to write the code, do we need to read the code? So a lot of

work is being done now by people spending more time doing kind of spec-driven development. So we'll write a spec that defines what they want. And then the question is. Do they just use it to generate the code that they then maintain? Or do they only work at the spec level? And if you get to a point where you're just working purely at the spec level and you're never looking at the code anymore, then what you kind of do is you build an apparatus around how that spec is executed. So you start building the ability to kind of create the code and validate the behavior of the system that you've now changed. Huge part of a software factory and spec-driven development, if you want to push it to those extremes, is Getting really good at saying what good looks like. And this will come I think this might solve some of your challenges here. In terms of whether PP have speaker-1: But this apparatus is now also a system that needs to be resilient, you know? It needs to like be correct and it also needs to be able to rebound. And that is something that I feel is super interesting because that system again needs to have an idea about the resilience of the system it builds or it comes out of the specs. speaker-0: Yes. Again, I think we can put this at two points. So first is like like like I and this is something that people realized early on with CI. CI is a production system. Because if my CI server isn't running, I can't roll software out. That's a big problem. So it's a production system, right? So from that point of view, we have to treat the apparatus around our software factory in the same way. We we need and that's why I had a chat with charity made spats a couple of days ago and we're talking about the you know, we see observability as something that is really important in production. It's just as important to get observability out of your The process by which you create the software. So you should have your observability in that world too, right? So that's really important part of it. So 100% it's a production system. The next thing you went on to say, which I think is also completely valid, is if we're expecting the software factory to create the software that then runs in production, that software factory has to be given the knowledge to understand how to create software that's going to behave the way we want the software to create, to run. So to give make it really concrete, we might have SLOs. So service level objectives around acceptable uptime and latencies and things like that. So we might say we need, I don't know, 99.9% availability with with a 99%ile latency of you know 300 milliseconds or less for order processing. Okay, that's an example SLO we might have. So we can have that in production. The question then is that's what we want. How can we then instruct or guide the software factory to know how to meet those demands. Yeah. I don't know that we know the answer to that. speaker-1: Yes, we took a I don't know the answer. I have no idea. speaker-0: No, and I don't think we do either. But that's also why I think this is the problem I think where where some of the people are taking the software factory ideas. They take it to logical cluster. I'm gonna build my whole system of software factories. Let's wind back a little bit and think about this concept of software factories really cribbing off the idea of what we call dark factories in the physical world. So a dark factory in the physical world is an automated facility where robots, typically industrial robots, And automated other types of automation are constructing components. Human beings have constructed that facility. They've helped build it. They've defined how it's supposed to operate. They have the ability to monitor it. But by and large, they don't have to get involved because it just works. And things go in, things come out. And they're often called dark factories because the lights aren't on. It's quite fun trying to search for pictures of dark factories online because they're all AI because the real dark factories are dark. You can't photograph them. Yeah. They don't build finished customer products by and large. They mostly build intermediate components. speaker-1: And I feel that they are deterministic, aren't they? Because they are set up it's like a super deterministic, very, very deterministic process, right? speaker-0: Ver absolutely. Big but they're deterministic. They have validation built in. They have their own inbound quality checks. And they're also building small components. They're not building entire, they're not coming out with you don't you don't send in iron ore and you know, and bauxite out comes a car. That's not how that works. Right. It's not a factorio. Instead, they probably could build things that then go into other things. So take that part of it for the moment. Things that build into other things. We don't necessarily need a software factor that creates an entire system. We're nowhere near doing that really yet. There's a couple of companies I'm talking to I can't talk about, which might be close, but by and large, that's not we're at. So, but we can use them in certain areas. So let like this is what I'm trying to encourage some of my clients to do at the moment. Because this whole thing is scary. It's like don't think about your put the way you I think we we still have this idea that we have a one-size-fits-all process by which, by how we go from an idea To running production code. And I think what we're now looking at is that we have different ways of thinking. So imagine your system not as a big box, but now imagine it as a set of modules, whatever that modular architecture looks like. Some modules, some bits of functionality are really important, ultra critical, and you're really worried if anything goes wrong with them. There's also bits of mod There's also modules in your code base right now that you use that you never looked at, that you didn't check, that you just pulled in. And you

put them in with your system and it's fine. You do that right now, but I bet you've got millions of lines of code like that. Think about the third party libraries you put in. Sure. Right. So somebody else wrote that, you have an API, you pull it in, it's fine. So we've got software factories. We don't really think they can create systems that have a better, bigger, widespread understanding about resiliency. But what if we had them construct some of the modules? So have rather than this is our whole system, let's have them do two, three, four modules and plop them in. And then we as the human beings can then still say, okay, so if I've got a a service boundary, this service boundary has to have these SLOs and everything else. I could still have the factory kind of create the innards of that. And so a lot of that, and that's freeing me, the human being, to think about the higher level concerns, how all this stuff is stitched together. And I think that is an on-ramp for the software factory idea. Avoid some of the challenges I think you're quite sensibly outlining here, which is I I don't think I think if we give you can't one-shot these things anyway, right? I mean if you're trying to one-shot your way to a large-scale distributed system, you're nuts. But I don't think you can say, here's the code, here's the SLO, go do it. I I mean, look, it's technically possible it could do it. It would cost you so much money because it would imply that your software factory is able to spin up kind of production low test scenarios and it could work it out all for itself, right? And no one's got time for that. But I do Think though that rather than pushing for the like that massive, like maybe far off end goal that might never be achievable, let's talk about what's practical today. Take a big system, break it down, take one bit that's maybe not as critical, try doing some specs around that one component. If it works well for you, great. If it doesn't work well for you, well you can go back doing things how you always did. speaker-1: I think that is a super interesting idea. And one thing that now for me is the next step in in this thought experiment, you could say, is how does that now affect the organization, you know? Like how does it affect our way of currently creating software or or features? speaker-0: I'm not sure it does. So let's imagine you've got Sam's ideal organization already, which you may or may not do. But you have teams who have responsibility for part of your business domain. Right. That team is responsible for X, that team is responsible for Y, that team is responsible for Z. They work closely with their business stakeholders, with their product owners to understand what's required. And then that team works out the best way to achieve that goal. Team X over there realize we should just get a third-party system in here to solve that particular issue. We're gonna go buy a system, we're gonna use a SaaS product. Team over here says we're gonna build some software to do this. Then they go, what's the best way for us to build software? we could use software factory here. So I think actually what you can do is say if you've got an organization like that where you have teams who have responsibility and accountability for their product owners. speaker-1: End to end, right? Like a like a stream aligned team if you are team topologies. speaker-0: Hundred percent. And then of course, using the team topologies language, if you see more people wanting to build software factories here and there and everywhere, then what you're gonna need is enabling teams that support you in that. Yeah. So having a a a platform capabilities that make spec driven development, for example, easier. A friend of mine, Ben's got he's working on a book at the moment about building a gentic platforms that are sort of around this. So for me it's more like We know the problem space. We know the risks. We know what our product owner needs. We should make decisions if we're doing this or not. And then we can take advantage of the new tooling around us. I think that's a healthy place for it to be. I think the problem is we've got the opposite happening in a lot of organizations where we've got like corporate mandates. You must do X. This is what you're gonna do. Good example of this would be Meta, right? Who recently said, okay, we're going all in. Everyone's got to use AI to Do all their software development. This may or may not have any something to do with the fact that they spent billions and billions and billions and billions on GPUs and don't know what to do with them. But that's part of one side. So they went from sort of ten to twenty person teams to like four to five person teams using all this AI stuff to create software. And they found that the amount of code they were creating increased by two hundred and twenty five percent. The actual stuff they shifted, the actual uplift in terms of delivery was about twenty to thirty percent. Not nothing. Production instance up by 40 to 50%. Time firefighting up by 70%. And so you have this top-down mandate, go do this stuff. It didn't work out so well. And my feeling is if you've got an organization where these teams understand what their product owner needs, understand their responsibilities to the wider organization, make these things available to them, encourage them, support them in doing this, and say, I want to see more of you doing it. But not all teams are the same, because not all code is the same. And I think we have to also recognize that piece of it, which is, I think a lot of the problem is people think all code is the same in the same way as they think somehow that programming is typing. It's not, right? So different parts of your product are gonna have different risk profiles. They can have different degrees of difficulty in the work that they're doing, they can have different requirements from the business. And so I think if we can approach this. In a bit of a

more nuanced way, then I think it gives us the opportunity to experiment with these really great new ideas and to see what works and adapt ourselves as we bring these things in, rather than just trying to force wholesale massive changes, which end up being amazingly expensive and often I think unfortunately ends up like often tainting the ideas in the first place. We tried that, it didn't work. No, you tried doing it in a really bad way. And so I that's what I'm trying to encourage people to do is to kind of chunk it up a little bit more. Don't get me wrong, Ben. A lot of this is about me really still wanting to push modular architectures because I think they're a great idea, right? We still didn't do them despite the ideas being around since the nineteen sixties, nineteen seventies, right? But fundamentally, once you see your production system as a modular system, this stuff becomes easy. speaker-1: Yeah, I I mean I'm I'm totally on your side there. I I really I really like how you described that and it it's really helpful. The thing is that you you're also absolutely right that you need to deliberately like introduce these new ideas where it makes sense and not like everywhere. And and you are also absolutely right. Like code is like you need to differentiate there a lot between the modules and like on so many levels, like it's insane. But often when I see organizations that are very, let's say, agile already, like in a in a positive and good way, like you describe it, like end-to-end, streamlined teams and so on, that they now they do it also deliberate and very good because now they have the they already have a structure in the organization that makes it possible for them to pick up on these new ideas also in a very deliberate and nuanced and good way. And then there are other organizations that are not there yet or that say that they are there yet. And they now man so that's like a downwards and an upwards spiral. I don't know what you think about that or how do you like speaker-0: I think you get a lot of companies that think they're better than they are. And therefore they think they can go faster than they can. And I think that's always going to be the case. I don't know what you do about that, other than to say, measure both qualitative and quantitative. By all means track, are people happy? Are people enjoying their work? Are you seeing increased levels of burnout? Quite a few studies are showing increased levels of burnout with the use of AI software, people working longer hours. So you've got to look at that because ultimately your people are your biggest asset. Also looking at the quantitative measurements. Are we getting more soft features into the hands of our users? And the next piece is really important. Are the users enjoying those features? I'm sure you've come across a few products that have had AI shoehorned into them. And it's just like, what the hell is this for? And so I think if you're collecting that information, it gives you the ability to take some of your own personal investments. out of this. This is the way we're doing it. If you get that information, it gives you the ability to to to try and change your mind going different directions. So a great I was at a conference called ShipIcon in Dublin last week. And there was a talk by I'm gonna get the speaker's name up because I don't want to get her name wrong. is it a speech by talk by Jennifer Cox who sort of works in cybersecurity. She did a really great talk called Ship Fast Enough to Care. And it was well, it was that wasn't the talk. I told her she should change the name of talk to ship fast enough to care, but it's something she said at the end. And she was talking about that AI gives us this idea that we can go much, much, much faster. But that actually we should examine is speed the most important thing. and we should show up faster by all means, but only as much as we can care. Care about our people, care about the product we're building. care about how it impacts our users. Because if we're going faster than that, we're going to have a problem. And I think that's where we're at at the moment is that it's like we've been given this really powerful sports car and everyone's jumping in and driving off, but no one's yet thinking, should we be going this fast? Should we be going this direction? You're also seeing this kind of problem of round cognitive debt, which I think the Margaret Ann Story's talked about quite a bit, where because people can work go faster, You're getting teams where people are going off in different directions. I'm doing this, you're doing and so people are losing their joined up sense of the system as a whole that we're building. We're losing sense of our collective understanding of the pro problem. We're spending less time talking. We're spending less time having conversations. There's a really great paper by Peter now called Programmers Theory Building. And he makes the argument. That our understanding of a program of what a program fundamentally is is basically a mental model shared by the developers. It's a paper written back in like 85. It's a really fascinating piece. He won the Chewing Award for it, I believe. But the shift I would say now is that our understanding of what a program is is actually a mental model in the hand in the minds of the developers and the product owner and the users as well. I think it's a collective mental model. Our use of AI shouldn't fragment that understanding, it should give us more time to focus on building and refining that understanding. AI is supposed to free us from drudgery. It's not supposed to increase our drudgery. It's not supposed to increase the amount of context switching we're doing and the amount of parallel work we're doing. We feel like we've got less time to stop and think. The studies around cognitive debt have shown that people who suffer from it are having a reduction in their ability to do critical thinking. speaker-1:

I feel that this brings us back to the resilience a lot, right? Like there's a all factors that are also super important for resilience in systems. speaker-0: Yeah, and I mean this comes back to you know your graceful extensibility, your sustained adaptability. You need to be able to think critically. Incident analysis needs clear eyed thinking. I think we should be using AI to free ourselves up to be more human. That's another that's a quote from one of my clients in the US. He says, I want my developers to be more human, right? Humans are great at thinking about trade offs. They're great at thinking critically, at having conversations, about sharing ideas, about learning, because our LMs don't learn in the way that we would conventionally think of it. So our use of AI should free us up to think more and to work together more. I saw an announcement from ChainGuard recently that they're stopping human review of diffs as part of PRs because they want to free people up to spend more time working around things like design docs, which kind of implies we're not stopping human code reviews so we can just write more code. We actually want to free people up to think more about the design and architecture and trade-off conversations. That feels like an okay trade-off to me, right? So I think it's always like, what are we getting for this? But we also have to recognize this is early days. Everything's moving very, very fast. The studies that we're seeing in terms of uplift from AI are based on the short period of time. We don't know the long term impacts. At the same time, the models are getting better. So it's really a volatile space. And so I think the most important thing you can do when you've got this hugely volatile space is creating an environment at work where you can think deliberately about what you want to do, that you can experiment, that you can try different things, that you can create the opportunity to hedge, because that's the only way. You're gonna survive. I think being all in on X is an insane thing to do when X may not even exist and X might be superseded by Y or Z within two years. So we speaker-1: Two weeks, you don't know. speaker-0: don't know. I think anyone that tells you they know is lying to themselves or lying to you and they're probably trying to sell you something, right? So for me it's a catch- speaker-1: race my on my own a little bit. This is like it sounds and I get reminded of that a lot. It's like go go slow to go fast. It's a little bit I don't know. I feel that speaker-0: There's something too there. I think it's like the team I work with quite a bit in the US, they're very thoughtful about their use of AI. They have different teams that are using it in different ways. They're talking all the time, they're sharing knowledge, they're adapting, they're triangulating, then they think, that's great, we'll adopt that. But that's not a transactional activity. That's something that's just continually happening as part of how they do work. And their understanding of the capabilities of AI and how they use it in development is now influencing how they use it in their production systems as well, how they embed it and give things to the customers. And I think that's the only thing that makes sense to me is like being free time to be more human. Don't assume the future is set. going back to our those four concepts of resiliency. Yeah I would speaker-1: That makes them resilient, you know? speaker-0: You've got to deal with surprise, right? so I think that's that's for me is is all part of it. Is like we people who work together to build a product together, coming back to the Peter Now stuff, we need to spend time refining those mental models and building those things up. AI should free us up to do that more or help us in doing that. But we can't replace that mental model with what AI thinks because that's not how it works. So I think there's a yeah, I think. The world's uncertain and so you need to find a way for your company to operate and for your to to deal with that uncertainty fundamentally, both at a personal level, at a process level, at an architectural level, at a strategy level. speaker-1: Sam, thank you so much. I think this is a very, very good point to to come to a stop. I really, really enjoyed our conversation. thank you again for being here. And everybody please buy the new book. it's on my list already. speaker-0: I've got like little postcards of the new book, which hopefully you can see. I don't know if you better see that. There we go. It's gonna be reversed, but but it's got a badger on the front. speaker-1: Thank you so much. And yeah, maybe see you around. speaker-0: See you soon. speaker-1: Thank you so much for listening to this episode of Rethink Engineering. Sam, thank you as well. It has been a pleasure to talk with you about resilience. And I really think that AI is going to change a lot in that area too. And yeah, we can read all about that in your book, right? Resilient distributed systems. we can read about that on your blog, Sam Newmanio. And I think we can also See you at some conferences in the Code Talks or the future coding days. Put the links in the description below. And on the next episode, we are going to talk with Patrick Dubois. I'm also really looking forward to that one. it's going to be great. Please subscribe so that you don't miss it. And that was Rethink Engineering by Rethink and Hackers and Wizards. speaker-0: Mm.